nordisk/nornir

Clone / populate event stream — why a member didn't populate, readable remotely

Design + guide + idiot-guide for PLAN #6: make per-member clone/fetch/populate failures first-class, persisted, and readable from a thin viz/CLI client instead of eprintln!-only on the server log. The durable twin of the monitor's eprintln!("…fetch error…"). Reuses the release_events / airgap_events event-table shape and the EPIC N-data run→persist→live pattern.

1. The problem (seen live)

When nornir-server's poll loop (or a Workspaces.Fetch RPC, or nornir workspace add) clones+fetches a monitored workspace's git members, a per-member failure used to surface only as a line on the server's stderr:

nornir-monitor: nordisk::korp fetch error: clone-fetch https://… Couldn't obtain Username

A thin viz/CLI client (nornir … --server, nornir viz against oden) has no access to that log. It just sees the member's data missing and no reason why — was it an auth failure (https→ssh), a vanished/renamed repo, a private repo, a network blip? Invisible. The populate outcome was log-only, not data.

2. The fix — one row per member attempt, persisted to the warehouse

New append-only Iceberg table clone_events (LAW: persist to the warehouse, not /tmp; on the fast warehouse disk), one row per fetched member attempt:

Scalar columns only (no depends_on DAG edges — a fetch has no inter-member ordering), so SQL readers (DuckDB / PyIceberg) query it directly. Schema: src/warehouse/iceberg_schema.rs::clone_events. Writer/reader: src/warehouse/clone_events.rs (mirrors release_events.rs / airgap_events.rs).

Where the monitor writes failures

src/monitor.rs::fetch_workspace now records a MemberOutcome { member, remote, op, status, detail, elapsed_ms } for every fetched member (ok OR error) onto FetchReport::outcomes. Because nornir-server holds the single-writer redb lock, the monitor itself does not open the warehouse — the call sites that own a handle persist the report via clone_events::record_fetch_report(&wh, &report) (best-effort + non-fatal — a logging-row failure never aborts a fetch/republish):

3. Expose it over gRPC — Viz.CloneEvents

proto/nornir.proto: rpc CloneEvents (VizTimelineRequest) returns (JsonResponse) on service Viz. Handler in src/bin/nornir-server.rs (VizSvc::clone_events) resolves the served workspace via AppState::ws(metadata) and returns query_clone_events(&wh, CloneSelector::Workspace(name)) as a JSON array of CloneEventRow — the SAME rows the embedded read returns (mirrors Viz.ReleaseEvents / Viz.TestResults / Viz.BenchTelemetry). Client decoder: src/viz/remote.rs::fetch_clone_events.

4. Show it in the viz — 🧬 nornir root pane

src/viz/nornir_tab.rs gains a 🩺 populate status panel for the active workspace: a per-member grid (member · op · ok/failed · detail) with a green/red summary (✓ all N ok / ✗ K member(s) failed to populate). It loads on a ⟳ button and auto-loads on a workspace switch:

• local mode reads the workspace warehouse with IcebergWarehouse::open_read_only (lock-tolerant — never contends the server's single-writer lock);
• remote mode reads Viz.CloneEvents.

LAW #6 (components emit what they render): state_json()["populate_status"] lists { workspace, total, failed, members:[{member, op, status, detail, remote, ts, elapsed_ms}] }, and the button is discoverable as { id:"populate_status", rpc:"Viz.CloneEvents" } so the headless matrix mechanically covers the surface.

5. CLI parity — nornir workspace events <name>

nornir workspace events nordisk          # human view, newest-first
nornir workspace events nordisk --json    # raw CloneEventRow array
nornir --server oden:7878 workspace events nordisk   # thin path over Viz.CloneEvents

Local path reads <root>/<name>/builds via open_read_only; thin path calls Viz.CloneEvents with the nornir-workspace header set to the target. No viz-only capability — the CLI prints the same event list (LAW: CLI parity).

6. Tests (inject-assert, not "didn't panic")

• warehouse::clone_events::fetch_report_failure_persists_and_reads_back — feed a FetchReport with one OK + one FAILED member → assert exactly two rows persist with the right (member, op, status, detail, remote, elapsed_ms), read back workspace-scoped, the failed member's error chain is intact, a different workspace's scope is empty, the human render names the failure, and JSON round-trips.
• warehouse::clone_events::republish_outcome_records — a workspace-level republish row records (member *, snapshot id in detail).
• viz::nornir_tab::state_json_surfaces_failed_member_populate_status — inject rows → assert state_json["populate_status"] surfaces the failed member, its error detail (RFC3339 ts), the failed-count, and the Viz.CloneEvents button.
• viz::remote::clone_events_wire_roundtrip — the server's serde_json::to_string ↔ the client's from_str round-trip is lossless (failed member + error chain survive).

7. Place in the constellation

This is the EPIC N-data read+write loop for the fetch/populate surface: the producer (the monitor) streams outcomes into the warehouse; the thin client reads them back over Viz.CloneEvents or nornir workspace events. It complements release_events (release op-DAG) and airgap_events (bring-up DAG) — the same event-table shape, the same "see why it stalled" goal, now for the clone step that feeds every other table.