nordisk/knut

The rel2graph engine seam — one entrypoint, any data source

Class: design. The single, source-generic entrypoint every DB-free consumer of knut's rel2graph engine calls, and the exact seam korp's iceberg2graph / falkor_graph demo step should route through. It replaces the old shape where the only offline driver lived in the benchmark crate behind a synthetic-database type. Source: module knut_bifrost::rel2graph — TableSource, build_mutation, ArrowTables (crates/knut-bifrost/src/rel2graph.rs).

The one call

use knut_bifrost::rel2graph::{build_mutation, ArrowTables, TableSource};
use knut_bifrost::MappingSpec;

// 1. the reviewed relational→graph mapping (the same spec rel2graph proposes/edits)
let spec = MappingSpec::from_yaml(yaml)?;      // or MappingSpec::load(path)

// 2. whatever produced per-table Arrow batches — an Iceberg scan, a parquet read,
//    a Spark-Connect read — collected behind the TableSource seam
let mut tables = ArrowTables::new();
tables.insert("erp.hr.employees", employee_batches);
tables.insert("erp.sales.orders",  order_batches);
// …

// 3. THE engine call: canonical GraphMutation (VertexSets + EdgeSets)
let mutation = build_mutation(&spec, &tables)?;

build_mutation compiles each table's declarative pipeline once, applies the Transform fabric to that table's batches, and merges every step's output into one GraphMutation. It is exactly the per-round work the live Listener does, minus the source read and the sink write — so the mutation it returns is the graph a live load would MERGE, at the same seam the correctness oracle asserts on.

build_mutation_reported — telling an empty build from a built one

build_mutation skips a spec table the source does not carry. That is the right default: a partial source yields a partial-but-consistent graph. It is also exactly how an empty graph comes back looking like a successful build — point the source at the wrong namespace, every table is "skipped", the mutation is empty, and the call returns Ok.

let (mutation, report) = build_mutation_reported(&spec, &tables)?;
let report = report.require_non_empty()?;   // names every table that gave nothing

BuildReport { tables_mapped, tables_empty, rows_read, vertices, edges } is carried BESIDE the mutation rather than folded into it, because the mutation is what a sink MERGEs and a report is not. build_mutation delegates to build_mutation_reported, so the counts can never describe a different run than the mutation they came with.

Driving the LIVE listener from the same batches

ArrowTables implements source::ArrowBatches, so the identical holder can drive the full round loop — sink, state ledger, asset events — through source::ArrowSource::new("skade", tables). Same Arrow, same mapping, same GraphMutation; one path is one-shot and offline, the other is a poll loop with a ledger. See the bifröst design f1, and note that an Arrow source gives a batch read, not CDC.

Why this is the seam (the sin it fixes)

bifröst is the rel2graph engine: one reviewed MappingSpec becomes the canonical GraphPlan, which fans out to three consistent forms — the PySpark job (GraphPlan::to_pyspark), the thund pipeline (knut_thund::rel2graph::to_thund_pipeline), and the directly-materialised GraphMutation (this call). All three derive from the SAME plan; the two-projections-agree invariant pins that they never diverge.

Before this seam, the only offline path to a GraphMutation lived in knut-rel2graph-bench::build_mutation and took the synthetic GeneratedDb. Every real consumer therefore either pulled in the bench or bypassed the engine. Promoting the driver into knut_bifrost::rel2graph behind [TableSource] means the data behind the engine can be swapped without touching the engine:

What korp calls (for the consumption agent)

korp's demo iceberg2graph / falkor_graph step must NOT re-implement the relational→graph mapping or bypass the engine. It should:

1. read the source tables it already reads from the lakehouse (Spark-Connect / Iceberg) as Arrow RecordBatches — no change to that read;
2. drop them into an ArrowTables keyed by the fully-qualified table name the MappingSpec uses;
3. call knut_bifrost::rel2graph::build_mutation(&spec, &tables) to get the GraphMutation;
4. hand that mutation to the sink it wants — the live FalkorDB load via knut_bifrost::sink::FalkorDbSink::apply(&mutation) (the falkor_graph endpoint), or the read-only Arrow Flight projection via knut_flight::Catalog::from_mutation(&mutation, source_tables) (the viewer path, flight-graph-server-design.md).

The mapping spec is the one artifact korp owns and edits (rel2graph proposes it, the PALANTIR view reviews it); everything downstream of build_mutation is knut's and is identical whether the data came from a bench, a parquet dir, or Iceberg.

The adapters knut ships

• ArrowTables — an in-memory BTreeMap<table, Vec<RecordBatch>> TableSource. insert / with_table to fill it; build_mutation reads it. Tables the spec names but the source lacks are skipped (a partial-but-consistent graph), so a source may carry a superset.
• Catalog::from_parquet_dir(spec, dir) (knut-flight) — the batteries-included live path: reads <dir>/<table>.parquet for each spec table into an ArrowTables, runs build_mutation, and serves the result. Proven end to end in knut-flight/tests/live_data.rs (parquet on disk → engine → Arrow Flight → client pull) and runnable as knut-flight-server --parquet <dir>.
• knut_thund::rel2graph::to_thund_pipeline — the sibling projection to a runnable thund Pipeline; the native graph-sink write is exercised end to end from a real invocation by cargo run -p knut-thund --example rel2graph_native --features native,rel2graph.

Proving the live FalkorDB sink

The offline seam is asserted DB-free against the correctness oracle; the LIVE half — build_mutation → FalkorDbSink::apply → a real FalkorDB — is proven by crates/knut-bifrost/tests/live_falkordb_sink.rs. It is #[ignore]-gated (CI has no DB) and runs on a host with one reachable:

cargo test -p knut-bifrost --test live_falkordb_sink -- --ignored --nocapture
# KNUT_FALKORDB_URL overrides the default redis://127.0.0.1:6379

It drives the engine over a compact Swedish-myndighet case schema (Person / Handlaggare / Arende / Beslut + two junction/handling tables), applies the mutation, then reads the graph back LIVE with GRAPH.QUERY and asserts every pattern landed. It uses a unique throwaway graph name and GRAPH.DELETEs it at the end, so a seeded graph such as korp_demo is never touched. Last run: 11 nodes / 14 edges, all read-back assertions green.

Relational patterns the engine + sink cover (each read back live)

The null-FK "no edge" invariant (a live-path correctness fix)

EdgeTransform projects the endpoint columns verbatim, so a nullable FK (a self-ref root's manager_id, an optional FK) reaches the sink as an edge row with a NULL src/dst. The offline oracle already counts these as null_endpoint_rows and excludes them, but the sink used to render them — MERGE (b {id: null}) — minting a spurious node keyed on null and a dangling edge into it. rows_as_cypher_maps now drops any edge row whose src or dst is NULL (standard SQL FK semantics: a null FK is no edge), so the live graph is byte-consistent with the oracle, and SinkReport::edges_written counts real relationships only. Guarded DB-free by sink::tests::edge_rows_with_null_endpoint_are_dropped and end-to-end by the live test's REPORTS_TO root + unassigned-case assertions.

Known limitation (honest scope): a node whose PRIMARY KEY is composite (more than one column) is not expressible through the declarative vertex step — VertexTransform/FalkorDbSink MERGE on a single id_column, and edges MERGE each endpoint on a single key. Composite keys are covered where a real schema needs them today — junction tables with a composite PK (handled as edges with no vertex, above). A composite-PK entity node would need a multi-key vertex step + composite endpoint MERGE (the graphar-falkordb loader already has the composite-key edge primitive, insert_edges_keyed); it is not yet wired into the bifröst declarative path.

See also

• The engine internals + correctness rules: rel2graph-design.md, rel2graph-codegen-design.md, benching-benchmarks.md.
• The Flight viewer that serves a mutation: flight-graph-server-design.md.
• The networkx bridges over the same mutation: rel2graph-networkx-design.md.