nordisk/knut

knut-bifrost — crate card

Reverse-engineered (from source — verify, don't trust the name)

• One-line pitch: Bifröst — the bridge from lakehouse tables to a graph: watch Iceberg tables (via Spark Connect or skade) and continuously materialise them as a FalkorDB property graph through a declarative-or-custom transform fabric.
• Does: A Listener runs sync rounds pulling from a source (FileSource, feature-gated SparkConnectSource/IcebergSource), applying a MappingSpec (TableMapping, PipelineStep, SyncMode) and a TransformRegistry of Transforms that emit GraphMutations (VertexSet/VertexDelete/EdgeSet), which a sink (FalkorDbSink) applies; SyncState/TableState track incremental progress. CDC support (change rows, deletes, delta scan). Rich optional surfaces: WASM transforms (wasmi), Flight transforms, Airflow AIP-82 asset events, Prometheus metrics (--metrics-file / live /metrics HTTP), a bifrost CLI, and a test-matrix status emitter. Live round-trip is proven end-to-end by --ignored integration tests: integration_spark_falkordb (Spark 4.2+ GA) and, on the blessed GA axis, integration_spark41_streaming (feature spark41: Spark 4.1.2 + iceberg-spark-runtime-4.1_2.13:1.11.0 streaming into FalkorDB), both of which bring their containers up FROM RUST — Spark via knut_pipelines::server::SparkServer, FalkorDB via graphar_falkordb's autostart — rather than re-implementing boot.
• Key deps: graphar + graphar-falkordb (graph write), arrow, redis, tokio, async-trait, tracing, serde_yaml; feature-gated spark-connect (shares the workspace arrow 59 directly — the old arrow-56→59 IPC bridge is retired), skade (Iceberg), wasmi+arrow-ipc, arrow-flight+tonic, reqwest (Airflow), clap+tracing-subscriber (CLI), tiny_http (metrics), nornir-testmatrix.
• Shape: lib + bifrost bin (feature cli); large, most feature-gated. Modules: listener, source, sink, mapping, mutation, transform, registry, state, metrics(+_http), airflow, wasm, flight, testmatrix. Companion wasm-guest/ sub-crate (excluded from workspace).
• Observed role in the workspace: The flagship ingest/materialisation engine. Consumes graphar + graphar-falkordb; benched by xtask; its airflow module is a designed downstream of knut-pipelines run events.

Fill in (one line each — only the human knows these)

• Primary consumer(s):
• Why it exists (vs the obvious alternative):
• Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
• Release cadence & independence (rides a parent? own repo someday?):
• Status (production / prototype / experimental / stubbed):
• Must-NOT-change invariant:
• Biggest known gap or TODO:
• Anything a newcomer would get wrong about it: