nordisk/knut

knut — branch & crate state (2026-07-29)

Class: history. A tracked snapshot of knut's branches, sub-crates and what has landed, so the state survives agent context loss. knut's main is API-stable for korp / korp-collectors (path consumers) and a live map agent runs against it.

TL;DR

All named feature branches (feat/spark41-iceberg-falkor, enforce-valkey9, bench_fix) are already merged into origin/main — nothing was stranded. The "forks from yesterday" work (Þund SQL authoring f1, chunked multicore GRAPH.BULK f9, popviz real-count gating) is all on origin/main. Local main had merely drifted 3 commits behind and was fast-forwarded to 26ddb45. No new merges were needed; no korp-consumed API changed.

Branch inventory (vs origin/main @ 26ddb45)

feat/spark41-iceberg-falkor = Spark 4.1 Iceberg→FalkorDB: bifrost watch --warehouse (sip an Iceberg warehouse into a graph, detached), optional S3 (s3a) build-arg, and the chunked multicore GRAPH.BULK encode (f9). enforce-valkey9 = default the FalkorDB image to a Valkey-9 build, never stock redis. Both already in main.

Sub-crate inventory (crates/)

Excluded from the native workspace: crates/knut-bifrost/wasm-guest (wasm32-only).

Landed this session

• Fast-forwarded local main b907a49 → origin/main 26ddb45 (the Þund SQL f1 merge + docs).
• Committed the untracked .nornir/rel2graph-design.md (design capture, 07-26; no code yet — a planned knut-rel2graph engine surfaced as a korp Tools-menu function). Docs-only, zero API impact.
• Refreshed Cargo.lock to match the current edda/nornir-build-thing resolution (knut-build path dep). Metadata only; does not affect korp (separate workspace) or any published API.

Verification

• cargo build --workspace + cargo test --workspace (CI-style, no --locked; CI itself does not use --locked) — see the session report for the green result.

Pending / not done

• ny_dok and salvage/2026-07-13 left on the remote (recommend abandon; not deleted by this agent).
• knut-rel2graph is design-only; build deferred until after the korp demo (per the design doc's Status).

korp API risk

None. No korp-consumed crate (knut-pipelines, knut-popviz, graphar-flight, knut-graphdb, knut-bifrost) had a public signature changed by anything landed here; all feature work was already on origin/main that the live map agent runs against. The only new commits are a docs file and a lockfile refresh.

korp-swarm C/D/E landed (2026-07-29 — feat/korp-swarm-knut-cde, merged to main)

The swarm leg C/D/E, verified green and merged fast-forward (see thund-design.md, thund-reasoning.md, pipelines-design.md, bifrost-design.md for the design capture):

• C — Þund ExecBackend, one IR / pluggable backend. both_backends_consume_the_same_batch_ir (in knut-thund/tests/iceberg_spark_gap.rs) authors ONE Pipeline and runs it on the native DataFusion backend (real SUM(v)=60 asserted) AND lowers the SAME value to a valid SDP graph the Spark backend accepts in check. All 4 authoring surfaces lower to the one IR — typed Rust builder (the IR itself), RON DSL (dsl feature, round-trips), SQL (sql feature, 13 tests, SQL→IR→RON→IR→SQL), facett graph (reuses knut-pipelines PipelineView). spark_4_2() flips CDC/sinks/persistent-views in the capability report.
• D — Iceberg↔Spark streaming GAP, proven honest. SparkBackend::lowering_report → SdpLoweringReport/SdpFlowDrop names, per flow, every knob SDP-over-Connect drops (watermark, window, non-append output mode, CDC on 4.1, SCD Type2 history, expectations). The native backend RUNS what SDP drops: native_runs_what_sdp_drops_watermark_and_window (a late row dropped by the watermark, windows close+evict) and cdc_gap_spark_41_drops_all_42_drops_type2_history (native keeps both SCD2 versions). RED-when-broken both ways.
• E — Iceberg→graph RON codegen seam. knut-bifrost MappingSpec gains RON parity (from_ron/to_ron/load_ron, YAML still accepted) plus from_arrow_schema → a candidate mapping emitted as RON where every column is mapped or SURFACED in unmapped (nested types), never silently dropped; guesses are listed for review. 6 knut-bifrost mapping tests. Crate-split (teach thund an Iceberg source vs route thund→bifrost) is the deferred design decision from the findings doc — left open for Rickard (runa vs knut-local RON dialect), not made implicitly.
• Spark-view backend API (for korp). knut-pipelines::spark_view adds neutral SparkRuntimeInfo/StreamingQueryInfo/StreamingQueryStatus (always compiled, no gRPC), fed by grpc-gated PipelinesClient methods (runtime_info, list_active_streams, stream_status, stream_recent_progress, stop_stream). Read-side full + stop; executor/worker topology is off the Connect wire and honestly not modelled.

API stability: every change is purely additive — new MappingSpec methods + 2 new BifrostError RON variants, a new spark_view module + re-exports, extra knut-thund re-exports (SdpFlowDrop, SdpLoweringReport). No existing korp-consumed signature changed; cargo build -p knut-pipelines -p knut-bifrost -p knut-popviz -p graphar-flight -p knut-graphdb is clean.

Python client to_networkx tested + documented (2026-07-29 — main)

The python/knut client's KnutClient.to_networkx (FalkorDB → Arrow Flight SQL → pandas → NetworkX) was wired with an example but untested and barely documented. Landed:

• Real offline test python/tests/test_networkx.py (9 tests) — exercises the actual to_networkx via a FixtureClient that overrides query to serve canned pyarrow.Tables, so the whole production path (query_df.to_pandas → _build_networkx → nx.from_pandas_edgelist / set_node_attributes) runs for real; only the gRPC/DoGet hop is stubbed. Asserts node/edge counts, node+edge attribute preservation, directedness (reverse edge absent), the undirected option, custom column names, and the Alice→Dave shortest path from social_graph.py. RED-when-broken (proven: breaking directedness fails 3, the fix below fails 2). Full suite 22 passed (9 new + 13 in test_proto.py).
• Bug fixed (Python-side only). to_networkx passed edge_attr=True for a src/dst-only edge frame; nx.from_pandas_edgelist (networkx ≥3) raises NetworkXError when there are no non-endpoint columns. Now edge_attr = [...] or None. The example never hit it (always has a since column). No korp-consumed Rust API touched.
• Testability seam. Extracted the pure KnutClient._build_networkx(vertex_df, edge_df, …) from to_networkx; the public to_networkx signature is unchanged (it now fetches then delegates).
• Docs. New design doc networkx-export.md (routes into docs/manual.pdf via <!-- nornir:class: design -->), linked from design.md and guide.md. PDFs not regenerated this session.