knut-popviz — design
Class:
design· Function axis:f0headless-core + thin-GUI-shell ·f1the shared knut-popsim model ·f2population gen to Arrow ·f3introspectable state ·f4the load driver ·f5the egui app. Strict: this file states only the shape we run. The system-wide why lives inreasoning.md; this function ships design + guide.Reuse:
popviz.uses = [falkordb, graphar, popsim]— see falkordb, graphar; it drivesgraphar-falkordb'sFalkorDbLoaderand wraps the sharedknut-popsimmodel (f1below).
f0 — headless core, thin GUI shell (the whole)
knut-popviz is a demonstrator + test harness for the bulk-insert path: it generates the synthetic 12-label Swedish-population graph as Arrow and loads it into FalkorDB through graphar-falkordb's FalkorDbLoader (UNWIND-batched Cypher — no falkordb_bulk_loader), while a desktop egui app shows a progress bar per label and live rows/s. crates/knut-popviz/src.
The whole design is one split:
The lib (PopvizState, vertex_batches, run_load) has no egui types and lives in the default build; the egui binary (feature gui) is a pure rendering of PopvizState. So the load logic and the introspection are unit-tested with no display, and the GUI is a thin shell over the same shared state (Arc<Mutex<PopvizState>>) the driver mutates.
Two laws hold it:
| law | statement | enforced by |
| same introspection contract as the pipeline viewer | PopvizState::state_json() is the external-state oracle; the GUI dumps it to $KNUT_POPVIZ_STATE ~every 300 ms |
state_json (lib.rs:132); pipelines-design.md#f6 shares the contract |
--autostart for robots |
--autostart / KNUT_POPVIZ_AUTOSTART kicks the load off on the first frame — a headless run loads + writes the JSON with no click |
main.rs:57,:102 |
Rule: any new visible field MUST be added to state_json().
f1 — the shared knut-popsim model (knut-popsim)
The single source of truth for the demo population — a pure, wasm-clean, deterministic model, re-exported here (pub use knut_popsim::{geo, LABELS, EKONOMISKT_BISTAND_RATE}, lib.rs:20,:36) and shared with the facett wasm demo. crates/knut-popsim/src.
It has no FalkorDB, Arrow, tonic, reqwest, tokio, or filesystem deps (serde is the only optional one), so it compiles for wasm32-unknown-unknown. Every consumer converts the same PopGraph to its own world.
| item | signature | where |
| Labels | pub const LABELS: [(&str, i64); 12] — the 12 register labels A–L + scale-1.0 counts |
lib.rs:47 |
| Tie rates | PERSON_TIE_PROB: [f64; 12], PERSON_REL: [&str; 12] |
lib.rs:70,:87 |
| Sample | pub fn simulate(SimParams) -> PopGraph |
lib.rs:243 |
| Schema | pub fn schema() -> PopGraph (12 label nodes, 11 hub + 4 cross edges) |
lib.rs:288 |
| Params | pub struct SimParams { n_persons: u32, n_kommun: u16, seed: u64 } |
lib.rs:209 |
| Graph | pub struct PopGraph { nodes: Vec<PopNode>, edges: Vec<PopEdge> }; PopNode { label: u8, kommun: u16 }, PopEdge { src, dst, rel: &'static str } |
lib.rs:193,:114,:135 |
| Geo | pub mod geo (address_for, is_ekonomiskt_bistand, cluster_hotspots, CITIES, SWEDEN_BBOX) |
geo.rs |
Laws: pure + wasm-clean (the build must pass for wasm32-unknown-unknown, default and serde); determinism is the contract (simulate uses a seeded LCG, no rand — same SimParams → byte-identical graph; geo uses splitmix64 seeded by node id, so a person always lands in the same place); generic graph, not backend-specific (no facett/FalkorDB types). serde (optional) derives on the model; PopEdge's &'static str rel interns on deserialize (intern_rel over ALL_RELS, lib.rs:153,:146).
f2 — population gen to Arrow (src/lib.rs)
vertex_batches(label, count, offset) (lib.rs:198) emits one label's vertices as Arrow RecordBatches of ≤CHUNK (25 000) rows. Every batch carries _gar_id (Int64) + idx + name; geo-bearing labels (Person/Address/Property, via label_has_geo, lib.rs:40) additionally carry lon/lat (Float64) + kommun (Utf8) from knut-popsim::geo::address_for, and Person carries the ekonomiskt_bistand (Boolean) flag from geo::is_ekonomiskt_bistand. Those columns flow verbatim through skade→Iceberg and FalkorDB.
f3 — the introspectable state (src/lib.rs)
PopvizState (lib.rs:79) is the whole observable truth, shared Arc<Mutex<…>> between the driver thread and the UI.
| item | shape | where |
PopvizState |
{ phase, labels: Vec<LabelProgress>, elapsed_s, error, target } |
lib.rs:79 |
Phase |
Idle/Vertices/Indexes/Done/Failed (label) |
lib.rs:56,:65 |
LabelProgress |
{ name, loaded, total } per label |
lib.rs:48 |
| derived | total/loaded/fraction/rows_per_s |
lib.rs:108–:122 |
state_json() |
the oracle: phase, target, total, loaded, fraction, rows_per_s, elapsed_s, per-label {loaded,total}, error, placeholders (e.g. "not started") |
lib.rs:132 |
state_json() lives in the default build (no gui), so the data-layer matrix asserts on it with no display. Same external-state contract as the pipeline viewer's $KNUT_VIEWER_STATE.
f4 — the load driver (src/lib.rs)
run_load(params: LoadParams, state: Arc<Mutex<PopvizState>>, wake) (lib.rs:250) is the blocking driver (the caller runs it on its own thread). It builds a multi-thread tokio runtime, computes per-label totals + _gar_id offsets, sets Phase::Vertices, and dispatches on LoadParams.target:
| item | shape | where |
LoadParams |
{ url, graph, scale, drop, target } |
lib.rs:183 |
Target |
Falkor | Iceberg { warehouse } (Iceberg needs feature iceberg) |
lib.rs:164 |
| Falkor path | connect → (--drop DETACH DELETE barrier) → drive falkor_load_phase_order() = [Vertices, Indexes, Edges]: rayon parallel batch-gen (vertex_batches par_iter) → per-label concurrent insert_vertices_multi_par(…, concurrency, on_progress) → Phase::Indexes create_index per label (BEFORE edges) → Phase::Edges parallel edge-gen → insert_edges_keyed_par → Phase::Done |
load_falkor |
| Falkor multicore | concurrency = available_parallelism(); the "sim" (Arrow gen) fans across cores with rayon, the "sink" (UNWIND-CREATE) dispatches up to concurrency queries in flight over the multiplexed link — fixing the ~1-core serial ingest. Same produced graph (_gar_id in the batch column; order-independent). Progress on_progress(rows) folds into buffer_unordered completion |
load_falkor |
| Falkor index-before-edges | the _gar_id index is built after vertices, before edges (falkordb-design f3b): edge endpoint MATCHes then hit an index (O(1)) instead of a full O(V) label scan, so the edge phase is O(edges) not O(edges×vertices) — the server-side half of the ingest speedup the client-side pipelining can't reach. Order guarded DB-free by falkor_indexes_are_built_before_edges |
falkor_load_phase_order, load_falkor |
| Iceberg path | one table per label into the warehouse via skade — no database, no container | load_iceberg (lib.rs:374) |
On the happy path it ends in Phase::Done; any error lands in Phase::Failed with error set. wake repaints the UI after each batch.
The Iceberg path is what lets the driver be robot-tested end to end with no external service: robot_iceberg_load_reaches_done_no_db (tests/robot.rs:147, #[cfg(feature = "iceberg")]) drives the real run_load into a tempfile warehouse and asserts the shared done-state bar — a genuine end-to-end drive, not synthetic state-poking, that still runs unattended in CI. See popviz-guide.md for the full tier matrix.
f5 — the egui app (src/main.rs [feature gui])
main.rs is a pure rendering of PopvizState: a progress bar per label, overall rows/s, a Start button, and the $KNUT_POPVIZ_STATE dump every ~300 ms. It spawns run_load on a worker thread and reads the shared state each frame. --autostart / KNUT_POPVIZ_AUTOSTART (main.rs:57) kicks the load on the first frame (main.rs:102); KNUT_POPVIZ_STATE (main.rs:59) sets the dump path. --start-db / KNUT_FALKORDB_AUTOSTART (main.rs:215) optionally boots a FalkorDB container first for the Falkor target.
Architecture map
| Area | Where |
| Population model | knut-popsim (LABELS, geo, PopGraph, simulate/schema) — re-exported; the canonical wasm-clean model. f1 |
| Population gen (Arrow) | src/lib.rs (vertex_batches — _gar_id + cols, geo from knut-popsim::geo) |
| Introspectable state | src/lib.rs (PopvizState, Phase, LabelProgress, state_json) |
| Load driver | src/lib.rs (run_load — connect → per-label insert_vertices → create_index) |
| egui app | src/main.rs [gui] — progress bars + --autostart + $KNUT_POPVIZ_STATE dump |
Robot + test-matrix coverage (the UI is driven in code)
knut-popviz is a thin egui shell over PopvizState; state_json() is the whole UI contract, so the robot asserts JSON instead of pixels. Coverage (tests/robot.rs, tests/testmatrix_wiring.rs) — all reuse edda's nornir-robotui/nornir-testmatrix, none hand-rolled:
| Tier | Runs | What it proves |
| synthetic | default, container-free, CI | phase progression idle→loading→indexing→done + the failure path |
| iceberg | --features iceberg, container-free, CI |
a REAL run_load into a scratch skade/Iceberg warehouse reaches done (no DB, no Podman) |
| live FalkorDB | #[ignore], needs $KNUT_FALKORDB_URL |
the real bulk-loader end-to-end against FalkorDB |
| matrix-emit (new) | --features "iceberg testmatrix" |
the real run_load Ok arm emits a GREEN knut-popviz/run_load row into the nornir matrix end-to-end — closes the gap between "the wrapper forwards" and "the loader calls it" |
The remote/introspection surface (state_json + the functional_status emit) is exercised in code, unattended — no human, no live DB — so the matrix stays honest on every CI run. See the load-path map: assets/popviz-loadpath.svg.
