nordisk/knut

knut — reasoning (f0, why this overall shape)

Class: reasoning. This file holds the why of the whole system — why the seven functions are stacked the way they are, and the system-wide decisions + roads not taken. The descriptive shape lives in design.md, which by rule mentions none of this. Per-function why lives in each *-reasoning.md.

Why a knytpunkt at all (the one payoff)

knut exists to carry graph data the whole way from a columnar lake to a live, queryable graph and back, in one Arrow pipe with no JVM required. Every rival covers part of that arc — GraphAr the files, Neo4j the graph engine, Spark+GraphFrames the compute — but none carries a RecordBatch from an Iceberg snapshot into a queryable FalkorDB graph and back out over Flight SQL without a JVM. The columnar seams (lake ↔ Arrow ↔ Flight) carry no serialization tax — they are Arc bumps on one Arrow tree. The graph store itself is the honest exception: FalkorDB speaks RESP over the valkey socket, so rows are fully serialized into Cypher/bulk blobs on the way in and RESP-decoded into Arrow builders on the way out. That single arc is the reason the seven functions exist and the reason they are Arrow-native end to end everywhere the wire format allows it.

Stack versions (newest, verified 2026-07-18)

knut tracks the newest of each core dependency; the analytics vintage matches korp's standardized Tools stack (Spark 4.1.x + Iceberg 1.11.0 containers).

Gap note: the Rust iceberg line (0.11.x, via skade's iceberg-arrow59 forks) is far behind the Java line (1.11.0) — different version series entirely. knut consumes it via skade and reaches Iceberg 1.11.0 table format through the Spark 4.1.2 container path; the pure-Rust skade read/write path uses the iceberg-arrow59 0.11 forks. Spark itself: knut defaults to 4.1.2 GA (the blessed pair with the 4.1 Iceberg runtime, since no iceberg-spark-runtime-4.2 exists yet while 4.2 is GA) and reaches 4.2.0 GA via KNUT_SPARK_IMAGE / the SPARK_TAG build-arg.

Why one Arrow tree (arrow 59 workspace-wide)

The workspace pins one Arrow major (Cargo.toml [workspace.dependencies] arrow 59). It is chosen, not incidental: orc-rust and spark-connect both bind an Arrow major, so a split tree would mean two incompatible RecordBatch types meeting at every seam. With one tree:

• the skade/Iceberg boundary (graphar-flight/src/export.rs:to_skade_batch) is a plain clone() — an Arc bump, not a buffer copy;
• the columnar↔lakehouse seams are zero-copy or near it. The graph seam is not: at the FalkorDB boundary rows are serialized into Cypher/GRAPH.BULK payloads going in and the RESP reply is decoded cell-by-cell into Arrow builders coming out. Zero-copy holds up to the valkey socket, not through it.

spark-connect carries its own private tonic tree behind the grpc/spark feature gates, so the two tonic majors never mix in one link unit — the types never touch.

The road not taken (now history). Before the workspace unified on one Arrow major, skade was on arrow 57 and the rest of knut on 56; they met only through the Arrow C Data Interface — a one-unsafe transmute of the ABI-stable FFI_ArrowArray/FFI_ArrowSchema structs between the two crates' arrow majors, in both directions (the export to_skade_batch out, the ingest from_skade_batches back — then named bridge_56_to_57/bridge_57_to_56). It worked and was ~2× faster than a parquet temp-file bridge, but it was a standing liability (one unsafe, two majors to keep ABI-compatible). Unifying on one Arrow major (58 then, 59 now) deleted the FFI bridge entirely; to_skade_batch/from_skade_batches survive only as named identity-clone seams so the export path and its round-trip test read the same. Some older bench prose still says "arrow-56↔57 bridge" — that is the historical name for what is now a plain clone.

Why this reuse stack (and not a flat set of crates)

The functions form a strict layered stack (graphar at the leaf; flight/bifrost/popviz above falkordb; thund above pipelines). The discipline: nothing below depends on anything above, and the shared spine is a single key.

• _gar_id as the spine identity. graphar mints one stable internal id per vertex and edges reference endpoints by it. Making that one key the identity is what lets falkordb's MERGE, flight's typed vertex/edge schemas, and bifröst's MERGE-both-endpoints all line up without each function inventing its own identity scheme. One key, one join everywhere.
• Two engines, not one. bifröst (table→graph CDC) and Þund (batch+streaming dataflow) are deliberately separate runtimes: bifröst's job is materialise a lake table as a graph, idempotently; Þund's job is author and run the compute that fills the lake. Folding them would couple graph-materialisation to dataflow execution. They share Arrow and the viewer, nothing else.
• thund reuses pipelines, not the reverse. Þund's IR is a strict superset of knut_pipelines::DataflowGraph; Pipeline::to_sdp lowers onto the pipelines function's Spark Connect connector so a native run and a Spark run render in the same viewer. Putting the dependency this way keeps the viewer engine-agnostic and free of Þund.

Why MERGE, keyed on _gar_id, both endpoints

Tables (and file drops) sync in arbitrary order, so an edge can land before the table that owns one of its endpoints. MERGE-ing both endpoints on every edge means the missing endpoint is created as a placeholder and enriched later by the vertex MERGE on the same id key — order stops mattering. The same property makes crash-replay idempotent: re-running a round MERGEs the same rows to the same nodes. This is why the sink is MERGE, not CREATE, and why per-table state is written only after the sink succeeds (state-after-sink) — a crash replays a round that is safe to replay. Full per-function detail: bifrost-reasoning.md.

Why "Redis" means Valkey 9+

The graph store and the shared SyncState ledger run on Valkey 9+, the open-source, Redis-API/RESP-conformant fork, reached via the redis Rust crate. redis:// in a URL is the wire-protocol scheme (which Valkey implements), not a vendor choice — no Redis-lineage server is used or required. The one wrinkle: FalkorDB's module gates on redis-compat ≥ 8.0 but Valkey 9 self-reports redis_version 7.2.x; the containers/falkordb-valkey9 Containerfile lowers the gate to 7.0 at build time so the module loads on Valkey 9.

Why both GUIs share one introspection contract

Both viewers (knut-pipelines, knut-popviz) expose their entire UI state as JSON from a state_json() that lives in the default (no-gui) build, dumped every frame to an env-var-named file. The reason is testing a GUI without a GUI: a headless run can drive the load/stream and assert on the JSON the screen is derived from — no display, no screenshot diff. That is why state_json() must live outside the gui feature and why the law "any new visible control MUST be added to state_json()" is enforced: an un-dumped control is an un-testable one. Mirrors nornir viz's urdr-threads.

Why heavy stacks are opt-in features

The lean default build is pure model + core, and everything expensive is a feature: grpc (tonic + Spark Connect codegen), gui (egui/winit/glow), skade (arrow-58/iceberg tree), native (DataFusion), spark, wasm, flight, metrics-http. The reason is compile time and airgap-ability: a consumer that only wants GraphAr files pays for none of Spark, Iceberg, or a GPU stack, and a CI that only tests the data layer never pulls a JVM. The cost is a longer feature-flag matrix, which the test matrix and the per-function chapters document.

What each function's reasoning covers

The system-wide why is here; each function keeps its own trade-offs and rejected alternatives:

Open work and locked decisions with dates: plan.md.