nordisk/nornir

nornir viz — the thin/remote read path (tab → RPC → table)

How a thin (remote) nornir-ui viz fills every tab from a running nornir-server instead of opening a local Iceberg warehouse. This is the companion to viz-modes.md (how the app chooses fat/thin) — this doc is the data plane: which tab reads which RPC reads which warehouse table.

Design — why a remote read path exists at all

The viz is source-agnostic: every tab renders from a plain data model (Timeline, Vec<ReleaseEventRow>, a KnowledgeSummary, …). In embedded (fat) mode the tab opens the workspace's Iceberg warehouse read-only and queries it directly. But a thin client (nornir viz --server …, or bare nornir viz with no local config) has no warehouse and no source checkout — those live on the server, which holds the redb single-writer lock. So a thin tab must read the same data over gRPC.

The contract every remote-backed tab follows:

The server produces exactly the data the embedded path would, serializes it (serde → JSON), and the client deserializes it into the identical model type. The tab's render + state_json code is then byte-identical across modes.

This is why the server handlers literally call the same query_* functions the embedded tab calls, and the row structs derive serde::{Serialize, Deserialize}.

The tab → RPC → table map

Bold rows are the RPCs added in feat/viz-remote-rpcs — before them, Release, Leaderboard, Knowledge, and the Live durable hydrate were empty against a live server (the embedded read had no remote twin).

The three new RPCs (all on service Viz, all return JsonResponse)

• Viz.ReleaseEvents(VizTimelineRequest) → JsonResponse — the server calls query_release_events(&wh, EventSelector::All) and serde_json::to_strings the Vec<ReleaseEventRow>. The client serde_json::from_strs it back. Backs the 🚀 Release tab and the 📡 Live Run pane's durable hydrate (the baseline a historical workspace shows when no run is streaming).
• Viz.BakeoffResults(VizTimelineRequest) → JsonResponse — same shape for query_agent_model_runs(&wh, BakeoffSelector::All) → Vec<AgentModelRunRow>. Backs the 🏆 Leaderboard.
• Viz.Knowledge(VizTimelineRequest) → JsonResponse — the producer case: the knowledge map is a syn parse + git-history walk of each repo's source, which a thin client does not have. So the server runs knowledge::scan_all over each member's materialized clone (<ws_root>/git/<member> = config::repo_dir_resolved(workspace_root, member), the same path the poll loop's republish scans) and returns a KnowledgeSummary { repos, crates } — the per-crate "bubble" rows + per-repo rollups the 🗺 tab renders. The client decodes it into the same KnowledgeSummary and renders crates>0.

All three reuse the existing VizTimelineRequest message (a workspace field); the served workspace is actually selected by the nornir-workspace gRPC header (set by the top-bar picker), so the request body's workspace is unused for these and left empty by the client wrappers.

Guide — how the remote read path works, end to end

1. Mode resolution (viz-modes.md) builds Source::Remote { endpoint, token }. app.rs constructs each tab's state with its remote constructor, threading (endpoint, token, workspace_name) in:

   • ReleaseTabState::remote(endpoint, token, ws)
   • LeaderboardTabState::remote(endpoint, token, ws)
   • LiveRunState::new_remote(endpoint, token, ws)
   • KnowledgeState::new_remote(root, repos, endpoint, token, ws)

2. First draw calls the tab's load() (or ensure_started for Live), whose Src::Remote arm calls the matching remote::* wrapper. Those run a private current-thread tokio runtime (remote::call) so the synchronous egui loop never blocks on async. Knowledge fetches on a background thread (the scan is heavy) and drains the result on a later frame, exactly like its local scan.

3. The interceptor (remote::Auth) attaches authorization: Bearer <token> and, when set, nornir-workspace: <ws> to every call — so the server routes to the right served workspace (AppState::ws).

4. The server handler resolves the workspace, hops to a spawn_blocking thread (warehouse scans + syn parses are blocking), runs the same query_* / scan_all the embedded path runs, serializes, and replies.

5. Workspace switch: the top picker calls app.switch_workspace(ws), which calls set_workspace/set_workspace_name on each remote tab (re-pointing the gRPC header) and reload()s — so the next draw refetches for the new server workspace. No restart.

How to add a NEW remote-backed tab

1. Give the row/summary struct #[derive(serde::Serialize, serde::Deserialize)] so it crosses the wire losslessly.
2. Add an rpc Foo (VizTimelineRequest) returns (JsonResponse); to service Viz in proto/nornir.proto (the crate regenerates via protox on cargo build --features server,viz — no protoc).
3. Implement it on VizSvc in src/bin/nornir-server.rs: resolve wctx, spawn_blocking, call the embedded query_*, serde_json::to_string.
4. Add a remote::fetch_foo(endpoint, token, workspace) wrapper that calls the generated client and serde_json::from_strs the response.
5. In the tab, add an Src::Remote { endpoint, token, workspace } variant + a set_workspace method; call remote::fetch_foo in its Src::Remote load arm.
6. Wire app.rs (the match &source constructor + switch_workspace).
7. Test: a serde wire round-trip unit test in remote.rs (to_string → from_str → assert-equal, real values), and a row in tests/viz_remote.rs's viz_remote_matrix (ephemeral server, empty-Ok) + live_full_matrix (asserts non-empty against the real fleet).

Idiot guide — "why is tab X empty when I point the viz at a server?"

A tab is empty in thin mode for exactly one of these reasons:

1. No remote read path — the embedded tab opens the warehouse directly, but no RPC ships that data, so the remote arm just showed a "… RPC TODO" note. This was the Release / Leaderboard / Knowledge / Live bug. Fix: add the Viz.* RPC + the remote::fetch_* wrapper + wire the tab's Src::Remote arm (the recipe above). The tab's state_json["source"] now reads remote … (Viz.ReleaseEvents) instead of (… RPC TODO).
2. No data on the server — the RPC works but the served workspace genuinely has nothing (no nornir release run yet → empty release_events; no nornir bakeoff → empty Leaderboard; an empty funnel). That's correct emptiness, not a bug — run the producing verb on the server host.
3. Wrong workspace — the top-bar picker selects which served workspace the nornir-workspace header targets. A historical workspace (e.g. holger) has data; a fresh one doesn't. Switch the picker.

The Dep Graph tab is special: it has no RPC of its own — it lays its nodes out from the Timeline's lanes. If Timeline renders (it does, via Viz.Timeline), Dep Graph builds. A dep_graph: null only because you never opened the tab is lazy-load, not breakage.

The Knowledge tab is the odd one: it's a producer (it parses source), not a warehouse read. The thin client has no checkout, so the scan runs on the server (Viz.Knowledge) over the repos the poll loop already cloned to <ws_root>/git/<member>. If it's empty, check the server actually has those checkouts (a path-member or SSH-only workspace may have none).

Tests

• Wire round-trip (src/viz/remote.rs unit tests): release_events_wire_roundtrip, bakeoff_results_wire_roundtrip, knowledge_summary_wire_roundtrip — prove serialize→deserialize is lossless and preserves real field values (LAW: real data in, real rows out).
• Render-from-summary (src/viz/knowledge.rs unit tests): remote_summary_renders_nonempty_rows asserts state_json shows crates>0 from an injected server summary — the inverse of the crates:0 empty-remote bug.
• Ephemeral end-to-end (tests/viz_remote.rs::viz_remote_matrix): spins a real nornir-server over a temp warehouse and calls all three new RPCs — they must return cleanly (empty-Ok on a fresh warehouse, not a transport error).
• Live fleet matrix (tests/viz_remote.rs::live_full_matrix, --ignored): rows 19–21 assert Viz.ReleaseEvents / Viz.BakeoffResults / Viz.Knowledge against the real server; the Knowledge row fails on 0 crates (catches a silent empty remote map).