.nornir/design.md

knut — design (f0, the whole)

Class: design · Function axis: f0 — how the seven functions wire into one system. Each function has its own *-design.md chapter; this file is the map that joins them. Strict: it states only the shape we run — every why and every rejected alternative lives in reasoning.md.

knut is a knytpunkt (Swedish: hub/junction) over property graphs: it carries graph data the whole way from the columnar world (Arrow, Iceberg, GraphAr files) to a live, queryable graph (FalkorDB) and back out over Arrow Flight SQL — one Arrow pipe, no JVM required. The through-line is Arrow 58, one tree workspace-wide (Cargo.toml [workspace.dependencies]), so every seam is zero-copy or a plain Arc bump.

The seven functions

function	crate	one-line	chapter
graphar	`graphar`	the GraphAr `gar/v1` chunked-file substrate over `object_store`	graphar-design.md
falkordb	`graphar-falkordb`	Arrow `RecordBatch` → FalkorDB via `UNWIND … CREATE/MERGE`	falkordb-design.md
flight	`graphar-flight`	Cypher over Arrow Flight SQL; graph → GraphAr / Iceberg export	flight-design.md
bifrost	`knut-bifrost`	watch lakehouse tables, materialise them as a graph by CDC	bifrost-design.md
thund	`knut-thund`	Arrow-native streaming dataflow engine, pluggable backend	thund-design.md
pipelines	`knut-pipelines`	viewer for Spark Declarative Pipelines + Spark Connect jobs	pipelines-design.md
popviz	`knut-popviz` (+`knut-popsim`)	population bulk-loader + headless-introspectable GUI	popviz-design.md

The reuse graph (how they fit)

The functions are a layered stack, not a flat list. Each declares what it uses; nothing below depends on anything above it:

graphar            (leaf — object_store only)
  ├─ falkordb      uses graphar          (_gar_id identity)
  │   ├─ flight    uses falkordb, graphar (Flight SQL front + export)
  │   ├─ bifrost   uses falkordb, graphar (table→graph CDC fabric)
  │   └─ popviz    uses falkordb, graphar, popsim
  pipelines        (standalone Spark viewer; facett adapter optional)
  └─ thund         uses pipelines         (IR ⊃ DataflowGraph; SparkBackend reuses the connector)

_gar_id is the spine identity. graphar mints a stable internal id per vertex; edges reference endpoints by it (graphar/src/writer, graphar/src/reader). That single key is what makes falkordb's MERGE, flight's typed vertex/edge schemas, and bifröst's MERGE-both-endpoints all line up. See falkordb-design.md.
The lakehouse → graph path is bifröst: TableSource → transform fabric → FalkorDbSink MERGE (knut-bifrost/src/listener.rs:Listener::sync_once). The graph → lakehouse path is flight's export (graphar-flight/src/export.rs, feature skade for Iceberg).
The two engines are siblings: bifröst (table→graph CDC) and Þund (batch+streaming dataflow). Þund lowers its IR either to a native Arrow/DataFusion backend or, via Pipeline::to_sdp, onto the pipelines function's Spark Connect connector — so a Þund run and a Spark run render identically in the same viewer.
Two viewers, one introspection contract. Both the pipeline viewer (knut-pipelines) and the population viewer (knut-popviz) expose their entire UI state as JSON from a state_json() that lives in the default (no-gui) build, written every frame to an env-var-named file ($KNUT_VIEWER_STATE / $KNUT_POPVIZ_STATE). So both GUIs are robot-tested headlessly. Law: any new visible control MUST be added to state_json().

The Arrow-58 invariant (the seams)

seam	where	shape
chunk file ↔ Arrow	`graphar/src/io/mod.rs:read_chunk`/`write_chunk`	five formats, one Arrow schema
Arrow → graph	`graphar-falkordb/src/cypher.rs:ColRenderer`	per-column downcast, one pre-sized buffer
graph → Arrow	`graphar-flight/src/convert.rs:response_to_batch`	FalkorDB RESP → `RecordBatch`
Arrow → Iceberg	`graphar-flight/src/export.rs:bridge_56_to_57`	identity `clone()` — knut + skade share Arrow 58
Iceberg → Arrow	`knut-bifrost/src/source.rs:IcebergSource` (feature `skade`)	reverse read of an Iceberg snapshot
Spark ↔ Arrow	`knut-pipelines/src/client.rs`	Spark Connect transports results as Arrow batches

Because there is one Arrow major across the workspace, the skade/Iceberg boundary (bridge_56_to_57) is a plain Arc bump, not a copy — the export and its round-trip test read the same seam.

Locked laws (system-wide)

arrow 58 / one tree so orc-rust and spark-connect agree; spark-connect carries its own private tonic, gated behind a feature, so the tonic majors never mix.
MERGE, keyed on _gar_id, both endpoints on every edge — tables sync in arbitrary order, so a placeholder endpoint node is enriched later by the vertex MERGE on the same id; this is also what makes crash-replay idempotent.
REFRESH TABLE before every snapshot poll — Spark caches Iceberg metadata per session.
"Redis" means Valkey 9+ — the graph store and the shared SyncState ledger run on Valkey (Redis-API/RESP-conformant); redis:// is only the wire scheme.
Heavy stacks are opt-in features — grpc, gui, skade, native, spark, wasm, flight, metrics-http are all off by default; the lean build is pure model + core.
Every shipping binary embeds version AND git-hash — --version prints <bin> v<pkg-version> (<short-git-sha>). The SHA is stamped by each shipping crate's build.rs (knut-bifrost, knut-pipelines, knut-popviz), which reads the repo's git metadata pure-fs / zero-shell (walk up to .git; loose ref → packed-refs; detached HEAD; linked-worktree gitdir:/commondir), emits cargo:rustc-env=KNUT_GIT_HASH, and the crate lib exposes GIT_HASH + VERSION_LINE (concat! of CARGO_PKG_VERSION + the hash). clap bins wire version = <crate>::VERSION_LINE; knut-popviz (no clap) prints it on --version/-V. Applies to bifrost, knutpipe, knut-spark-server, knut-popviz — NOT the rel2graph-timing bench fixture.

Surfaces (system)

binary / entry	function	verb
`knutpipe`	pipelines	`show` / `watch` / `gui` / `pipeline`
`bifrost`	bifrost	`run` / `gen-sample` (feature `cli`)
`knut-popviz`	popviz	GUI loader (feature `gui`)
Flight SQL server	flight	`run_server` / `run_server_with`
`python/knut`	—	thin Flight SQL client → pandas / NetworkX (networkx-export.md)
`xtask`	—	the nornir bench suite (benching-benchmarks.md)

Which recipe reader this repo uses, and why (2026-09-20: the code is the source)

Reader: nornir-build-thing 0.1.18, REGISTRY, from crates/knut-build-thing.

knut no longer READS a recipe at all. The impl BuildThing in that crate IS the declaration, and .nornir/build-variants.toml is EMITTED from it (cargo run -p knut-build-thing -- emit) — code → toml, one-directional, never read back as authority. Until 2026-09-20 it ran the other way: the toml was include_str!-ed into crates/knut-build and parsed back, so the file was the build thing and the Rust was a reader of it. The crate's NAME changed with the direction, because dwarves' forge finds a recipe as <repo>-build-thing (a cdylib) and knut-build matched neither — knut's code road was closed by name.

The dependency is the REGISTRY and never a ../edda path: a path dep is read at manifest-load time, so on a box without the sibling it aborts resolution for the WHOLE workspace (measured 2026-09-10). 0.1.18 is the first published version that carries the emitter.

Why: in-constellation; recipe is schema 1. The requirement had been the floating "0.1", and knut's graph carried a registry copy at 0.1.8 beside the path copy at 0.1.9 — two readers of the same file, one repo. The registry half closes on the release cascade (the pull comes through published transitives).

The rule itself — [nornir-build-thing] schema = N, a refusal that names the version to upgrade to, and a loader that refuses to DROP a key it cannot read — lives once, in edda .nornir/build-design.md, "THE COMPATIBILITY RULE". The cross-version guard that would have caught the 2026-08-30 skew is edda/nornir-build-thing-skew.