nordisk/knut

knut-popviz — guide

Class: guide · audience: a techie running the population loader or driving run_load / the introspection oracle. The what is in popviz-design.md. This file is the workflow.

Reuse: popviz.uses = [falkordb, graphar, popsim] — see falkordb, graphar.

Run the GUI

# FalkorDB target (default build has no display-less deps beyond egui):
cargo run -p knut-popviz --features gui -- \
    --target falkor --scale 1.0 [--drop] [--autostart]

# Iceberg target (needs the iceberg feature + a warehouse dir):
cargo run -p knut-popviz --features gui,iceberg -- \
    --target iceberg --warehouse /tmp/wh --scale 1.0 [--autostart]

You get a progress bar per label (the 12 register labels A–L), overall rows/s, and a Start button. --autostart (or KNUT_POPVIZ_AUTOSTART) kicks the load off on the first frame — no click — which is exactly what the headless smoke tier uses. --start-db (or KNUT_FALKORDB_AUTOSTART) boots a FalkorDB container first for the falkor target.

The headless run_load API

The whole load logic is in the default build with no egui — call it directly:

use knut_popviz::{run_load, LoadParams, PopvizState, Target};
use std::sync::{Arc, Mutex};

let params = LoadParams {
    url: "redis://127.0.0.1:6379".into(),
    graph: "pop".into(),
    scale: 1.0,
    drop: true,
    target: Target::Falkor,                     // or Target::Iceberg { warehouse }
};
let state = Arc::new(Mutex::new(PopvizState::new(params.scale)));  // lib.rs:90

// run_load is blocking — put it on its own thread; `wake` repaints a UI (no-op here):
let st = state.clone();
std::thread::spawn(move || run_load(params, st, || {}));          // lib.rs:250

// poll the shared state:
loop {
    let s = state.lock().unwrap();
    if s.phase == knut_popviz::Phase::Done { break; }
    if s.phase == knut_popviz::Phase::Failed { panic!("{:?}", s.error); }
    // drop(s); sleep …
}

PopvizState::new(scale) sizes every LabelProgress (total = LABELS[i] × scale, loaded = 0); run_load connects, generates every label's Arrow batches in parallel (rayon par_iter), then loads each label concurrently via insert_vertices_multi_par(…, concurrency, on_progress) (up to available_parallelism() queries in flight), a barrier, the edges via insert_edges_keyed_par, then create_index(label, "_gar_id") per label, ending in Phase::Done (or Phase::Failed with error). Both the sim (gen) and the sink (insert) are multicore — the fix for the old ~1-core serial ingest.

The introspection oracle

PopvizState::state_json() (lib.rs:132) is the external-state oracle — assert on it with no display. The GUI writes it to $KNUT_POPVIZ_STATE every ~300 ms (default <tmp>; overwritten in place). It reports:

{ "phase": "loading vertices", "target": "falkor",
  "total": 11000000, "loaded": 250000, "fraction": 0.022,
  "rows_per_s": 24800.0, "elapsed_s": 10.1,
  "labels": { "Person": {"loaded": 250000, "total": 8000000}, "Address": {…}, … },
  "error": null, "placeholders": [] }

Poll the file (or the shared state) as your post-condition: phase: "done" with every label loaded == total. placeholders carries "not started" while idle — assert it empty once data is expected.

The robot-test matrix (tests/robot.rs)

Every tier asserts on the same state_json() oracle through the shared assert_done_state bar (robot.rs:30), so asserting the JSON is asserting what the GUI shows. Three of the four tiers run unattended in CI — no container:

The no-DB Iceberg tier is the one that exercises run_load end to end without any external service (the synthetic tier pokes state; the live tier needs a DB) — so CI holds the full driver, not just the UI contract, to the done bar.

The live-GUI smoke variant runs the binary with --autostart + KNUT_POPVIZ_STATE set and polls the JSON file — no window needed.

The popsim model as usage (determinism note)

knut-popviz re-exports the shared knut-popsim model (LABELS, geo, EKONOMISKT_BISTAND_RATE). Generate a graph directly when you want the model, not the load:

use knut_popsim::{simulate, schema, SimParams};
let g = simulate(SimParams { n_persons: 1_000, n_kommun: 20, seed: 0xC0FFEE });
// g.nodes / g.edges — a generic PopGraph, no backend types
let s = schema();   // the 12-label schema graph (11 hub + 4 cross edges)

Determinism is a contract: simulate uses a seeded LCG (no rand), so the same SimParams yields a byte-identical graph; geo placement is splitmix64 seeded by node id, so a given person always lands in the same place. That is what lets the loader and the browser demo agree, and what makes the robot matrix reproducible.

Notes

• Throughput is bounded by FalkorDB's online write path, but the client no longer is: run_load now generates batches on all cores (rayon) and keeps up to available_parallelism() UNWIND-CREATE queries in flight (insert_*_par over the multiplexed link) instead of one-at-a-time — lifting the ingest off ~1 core. The graph produced is byte-identical to the serial path. See falkordb-design.md (f3a — the parallel bulk path).
• The viz schema is generic (_gar_id + a couple columns) — it watches throughput, not the faithful per-label schema the population example carries.