nordisk/holger

Task for Henrik: the smart flow (HOL-FLOW-SMART)

Owner: Henrik. Holger's coordinator lanes keep their hands off this, so there is no clash. Ask Rickard about scope. The plan is docs/ROADMAP.md, section "The smart flow".

Goal

Today the console, Overview and Flow all show the same drawing: the whole RON in one picture. Make it smart, as three zoom levels of one model.

1. Level 1, Overview: the estate from our server's view.

   • The boxes: console (ui), front, main, twin, each downstream (fetching or one-way), the diode drop area (shown when export.push_to is set), and the upstream sources (pypi, Chainguard, crates…).
   • The lines are detected connections with live state: pairing, sync lag, bytes, last contact.
   • The counters sit on the boxes and lines: artifacts, waiting for approval, quarantined, lag.

2. Level 2, Flow: click a machine and it unfolds in place, showing its part of the RON:

   • main: repositories, the intake chain (incoming → scanned → policy → approved → sealed → served), gates, storage, the promotion ladder;
   • front: doors, hosts, routes;
   • a downstream: its own RON summary;
   • twin: the mirror, and its lag. The RON's inputs and outputs end as ports on the machine border, joining the level-1 lines. Sources enter main's intake port; bundles leave main's export port and enter a downstream's inbox port.

3. Level 3: a "Show the full RON" button unfolds every machine.

The menus then each get one job:

• Overview = level 1;
• Flow = levels 2 and 3;
• Console = the operations pages, with no drawing.

Where the code is today (holger main)

• server/lib/src/flow.rs: the flow model and SVG renderer. It was restored from before the lean cut and draws every RON relation; counters come from the audit aggregates. It is served as AdminService.FlowSvg, GET /-/flow.svg and GET /-/flow.json, all admin-read.
• AdminService.Topology (server/lib/src/topology.rs) gives front, main and twin, the downstreams with their via, lag and the pairing certificate end.
• holger-ui (crate holger-ui-web): the Overview page and the /flow page show the server's SVG as it is sent.
• Downstreams report their sync point upstream (GrowService.Downstreams / DownstreamReport). Extend that report so a downstream also sends a small summary of its own RON (repositories, inbox, trust, boundary OneWay/Fetch), without secrets.

Rules (the same as the rest of Holger)

• The server builds the model and the drawing. The UI stays a thin layer: it only requests and shows. Unfolding is the server re-drawing: /-/flow.svg?open=main,downstream-a (and the same in /-/flow.json), so there is no layout logic in the browser.

• Each RON part knows which box it runs on. Put that in the model, not in the renderer.

• Counters without a source show "not recorded", never 0. A downstream is never drawn or labelled as an upstream.

• Test-first: a RED commit, then GREEN. The tests to write:

  • level 1 draws exactly the boxes and lines Topology plus the RON give;
  • ?open=main shows main's RON part with ports matching the level-1 lines;
  • a one-way downstream and the diode drop box appear when they are configured;
  • level 3 shows everything;
  • console, Overview and Flow serve the right levels.

• facett can later show the same JSON as an optional large view. Keep /-/flow.json the single source.

What you have to draw with

Examples are in docs/tasks/flow-smart/:

• new-flow-holger-rs.json: today's /-/flow.json for holger.rs;
• 3-new-flow-holger-rs.png and 4-new-flow-sources-steps-downstreams.png: today's drawing;
• 1-old-flow-9521bf06-holger-rs.png: the old drawing from before the lean cut.

1. The flow model: /-/flow.json (server/lib/src/flow.rs, schema in the file)

• Top level: schema, title, ladder (the promotion stages), nodes, edges, summary, refresh_secs, max_consumers, audit_recorded.

• A node: id, label, kind, detail, stage, gated, facts (declared, from the RON), rows (the measured counters), band, column, slot, state.

  • Node kinds today: console, front, main, twin, repository, step, door, storage, consumer, no_sources, no_downstream.
  • Bands: estate, chain, flow, storage. The estate band is your level 1; chain, flow and storage are what unfolds inside main at level 2.

• An edge: from, to, kind, cell (the counter text on the line).

  • Edge kinds: fronts, follows (twin), console, serves, stores, checks, promotion.

2. The live estate: AdminService.Topology (proto/holger.proto, TopologyReply)

• boxes (TopologyBox), one per machine:

  • name, role, address, state + why, this_box;
  • last_seen_unix_ms / seen_ago_ms, the disk (disk_total_bytes / disk_used_bytes), headline;
  • twin_state, lag_entries, held;
  • cert_not_after_unix, pairing_not_after_unix;
  • for a downstream: fetched_total, fetched_today, upstream_kind, feeds, via (front or wireguard);
  • sources.

• links (TopologyLink): from, to, kind, state. These are the detected connections for level 1.

• Also: estate, server_unix_ms, silent_after_ms, and TopologyConsumers (principals, peers, anonymous_today).

3. What is missing, and is part of this task (server side, test-first)

• Which box each RON part runs on: add a box field to every flow node, for example main for repositories, steps and storage, front for doors, the downstream's name for its own parts.
• A downstream's own RON summary: extend the downstream's sync report (GrowService.Downstreams / DownstreamReport) with its repositories, inbox, trust and boundary (OneWay or Fetch). No secrets. The upstream then has something to unfold at level 2. The summary carries its own proof (summary_proof, over the summary and the report's text), and the text the report's proof signs stays the 3.0.0 text, so an older upstream still hears a newer downstream and a newer upstream says "no summary" for an older one (HOL-SYNC-SKEW).
• The front's own report (HOL-FLOW-FRONT, done): main did not know the front's doors, hosts or routes, so the front tells it. holger-estate-front front_report sends POST /-/front/report with its doors, every name it serves and where each goes (main, the console, monetize) and the console's address, proven under a key both ends derive from the front's and the box's console keys (holger_downstream_rules::front, its own context, never the bearer). Main keeps the last report (server/lib/src/front_report.rs), and the flow unfolds the front to it at levels 2 and 3 with the same box and port rules. An older main answers 404 and the front carries on; a front that never reported stays one closed box.
• Another Holger's own flow (HOL-FLOW-FRONT, done): a downstream's summary carries its console's address (console_url, proven by its own console_proof), and its box carries a link to that server's /flow (FlowNode.link, a FlowLink over its "its own flow" row). With no known console there is no link.
• The diode drop node: a node and an edge when the RON has export: (push_to: …). A one-way downstream (boundary OneWay) shows its inbox as its port.
• Upstream source nodes: one per sources: entry (pypi, crates, OCI upstreams…), on main's intake port.
• Ports: for each machine, the RON inputs and outputs that cross its border. The model lists them, and level 1's lines attach to them.
• ?open=: /-/flow.svg?open=main,downstream-a and the same parameter on /-/flow.json. The server returns the drawing or model with those boxes unfolded.

4. The target picture (level 1, Overview)

 [pypi] [chainguard] [crates]                         sources
        \      |      /
 ┌ console ┐   ┌── front ──┐   ┌──── main ────┐ ⇄ ┌─ twin ─┐
 └─────────┘   │ doors     │──▶│ ▣ (unfold)    │   │ mirror │
               └───────────┘   └──────┬────────┘   └────────┘
                                      │ signed bundles (export port)
                  ┌───────────────────┼────────────────────┐
           downstream A          downstream B       [diode drop area]
           fetch · WG · lag 0    fetch · front           │ one way
                                                  downstream C (one-way inbox)

Each box and line carries its counters and its live state. Clicking main gives level 2: main's box grows and shows today's chain, flow and storage bands inside its border.

The visual style follows today's flow.rs SVG: the same boxes, fonts and theme tokens. Keep it one SVG renderer for all three levels.

5. What to draw, and where each fact comes from (rules)

• Nothing that is not there is drawn. No twin, no twin box and no "follows" line. No front, no front box. No console, no console box. No downstream, nothing below main (no empty "no downstream" placeholder at level 1). A test must prove it: a plain single-server installation draws main, its sources and nothing else.

• No console, no box. Draw the console only when this installation has one: a TopologyBox with the console role, or a console that has recently signed in or called. A plain cargo install holger without holger-ui draws no console. The same holds for front and twin: a box is drawn only when it exists.

• The diode case is in the RON on both sides:

  • upstream: export: (push_to: Some("<dir>"), every: …) draws a "diode drop area" box with a one-way arrow out of main's export port;
  • downstream: boundary: OneWay plus bundle: (inbox: …) draws the downstream with ONE input port, its inbox, and no line back. A one-way downstream never shows a fetch or receipt line.

• Upstream → downstream without a diode (fetch) is known from three sources. Draw it from all three, and show which ones agree:

  1. Declared in the RON:

     • on the downstream: boundary: Fetch and mirror: with one ron_kind: Holger source (its url), or the sync upstream with its kept invite;
     • on the upstream: sync.downstreams, if it is listed there.

  2. Paired (enrolled): the upstream's enrolment records, from the signed invite (holger-cli downstream invite, the console's Invite button). These are kept in the server's store, not in the RON, and read through GrowService.Downstreams. Each one has the downstream's name, its key and its via (front or WireGuard).

  3. Seen (live): the downstream's sync reports (DownstreamReport: its sync point, lag and fetched counts) and Topology's links, with last_seen.

• The line's look tells the state:

  • solid = paired and seen recently;
  • dashed = declared or paired but not seen within silent_after_ms (the label says "not seen since …");
  • grey with a "?" = seen, but neither declared nor paired (it should not happen; show it, never hide it);
  • one-way arrow = diode or USB, which carries no live state; show the last bundle's sequence and time from the import audit.

• Downstreams of downstreams: draw only one hop from this server. A downstream that is itself an upstream says "upstream of N" on its box; clicking it opens that server's own flow, if you can reach it.

• Never invent a line: a fact without a source is not drawn, and a counter without a source reads "not recorded".

How to work on it

• Repositories: ssh://git@git.gunnar.rs:2222/nordisk/holger.git (the server: flow.rs, the doors, the tests) and ssh://git@git.gunnar.rs:2222/nordisk/holger-ui.git (the console's Overview and Flow pages).

• Base: holger main (this file is at 1f0a94af) and holger-ui main. Merge main into your branch often: several other lanes merge into holger main every day.

• Branch: henrik/flow-smart in both repositories. Push it to git.gunnar.rs, open a pull request into main on gunnar.rs, and tell Rickard. Rickard (or his lanes) merges with --no-ff after the gate is green.

• Build and test:

  • cargo test -p holger-server-lib flow covers the flow tests (the_flow_is_the_ron, flow_door);
  • cargo xtask ci --quick runs before every push;
  • holger-ui: cargo xtask build, plus its browser tests in browser-tests/.

• Files to start from: server/lib/src/flow.rs, server/lib/src/topology.rs, server/lib/tests/flow_door.rs, holger-ui src/pages/estate_board.rs (where the console shows the flow SVG today), tests/the_board_is_mounted.rs and browser-tests/tests/console.rs, and the proto AdminService.FlowSvg.

• Commits: your own name, small steps, a RED test before each GREEN.

Where to try it

test2.holger.rs runs a recent DEV build. Ask Rickard for an admin passkey invite there, or run it locally with the guide "Three products by hand" in the README.