nordisk/knut

rel2graph CODEGEN — one MappingSpec, two backend projections (2026-07-29)

bifröst is the rel2graph engine. One concept, many tools: codegen · pipeline-config authoring · the live CDC listener · schema→mapping inference. This doc folds Rickard's authoritative rel2graph-codegen prompt into the real knut code, framed as bifröst = rel2graph and codegen = one of its tools. It reconciles the spec's proposed RON shape against the code that already exists, and records what was filled additively vs. what is honestly still a worklist.

See also rel2graph-design.md (the analyze→propose→execute engine placement) and thund-design.md / thund-reasoning.md (the execution IR + backends).

0. The core idea (reconciled with the coordinator's steer)

bifröst maps a relational schema to a graph once — a reviewed [MappingSpec], distilled to the canonical GraphPlan (labels, merge keys, edge directions, props, batch). That ONE plan is then projected onto a backend. bifröst is backend-agnostic: the two codegen outputs are both just projections of the one plan:

The pipeline.ron is not a bespoke second runtime and not the spec's flat Pipeline{…} struct — it is a knut_thund::ir::Pipeline (datasets + flows) that thund's pluggable ExecBackend already runs. "bifrost mapping → thund → Spark|Native" is one path; thund picks the tech.

The invariant: both projections derive from the same GraphPlan and must agree exactly on labels, merge keys, edge read columns, edge directions and batch size. Any divergence is a bug — pinned by knut_thund::rel2graph::tests::two_projections_agree (2-table) and, on the full 12-table schema down to endpoint read columns + merge keys, by the bench's twelve_table_projections_agree_down_to_keys (§2.1).

Why the projection lives where it lives (dependency direction)

knut-pipelines already depends on knut-bifrost (its airflow feature), and knut-thund depends on knut-pipelines. So bifröst must not depend on thund (that would be a cycle bifröst → thund → knut-pipelines → bifröst). The acyclic direction is thund → bifröst: therefore the GraphPlan → thund Pipeline projection + the two-projections-agree test live in thund (behind the optional rel2graph feature, default-features = false on bifröst), while the pure PySpark projection + the canonical GraphPlan live in bifröst. korp consumes bifröst with default features — unchanged; the new dep is opt-in.

1. What bifröst's codegen already did vs the spec (gap table)

Before this change (the "RON codegen seam", commit 1add167):

Do-not-remake note: the CDC engine, the FalkorDB/GraphAr sinks, the Spark Connect client, MappingSpec, from_arrow_schema, and the thund IR/backends all already exist and were reused, not rebuilt. Codegen is a thin, additive projection layer over them.

2. RON field-name reconciliation (spec proposed → REAL)

The spec proposes a flat Pipeline{source,sink,mode,batch_rows,concurrency,nodes,edges,indexes}. That struct does not exist in the code, and per the coordinator it should not — the runnable artifact is a thund Pipeline. The spec's fields survive, but as real names spread across the canonical GraphPlan, the thund Pipeline, and each thund Dataset's knut.* properties. Use the real names; every mismatch is flagged here.

2.1 The endpoint-key STANDARD — read column vs merge key (self-ref/renamed-FK correctness)

An edge endpoint has two distinct attributes, and conflating them is a correctness bug for any FK whose column name differs from the referenced PK (self-referential edges; any renamed FK like manager_id → employee_id):

• read column (from_col/to_col = EndpointSpec.column) — the column in the edge table the endpoint VALUE is read from. For Employee-[:REPORTS_TO]->Employee the head value is read from manager_id.
• merge key (from_key/to_key = id_column ?? column) — the property NAME the endpoint node is identified/MERGEd by in the graph. For REPORTS_TO both endpoints MERGE on the Employee node key employee_id.

The transform fabric (EdgeTransform::apply) always had this right — it reads src.column/dst.column and carries node_id_column() as the MATCH property. The codegen projection originally carried a single to_key = id_column ?? column and used it for BOTH roles, so selectExpr emitted employee_id AS dst — reading each employee's own id and collapsing REPORTS_TO into a self-loop of every employee to itself. Both projections shared the field, so they "agreed" with each other while both diverging from the data.

The standard, now enforced end-to-end: EdgePlan (and the knut.edge.* props) carry from_col/to_col (read) AND from_key/to_key (merge) as separate fields. The PySpark selectExpr reads from_col AS src, to_col AS dst; _write_edges(…, from_key, to_key, …) MERGEs on the node keys. The thund projection projects the FK columns (from_col/to_col) in the flow, and the native graph-sink renames those read columns to src/dst while taking the merge keys as EdgeSet.{src,dst}_id_column. They coincide for a plain FK (order_id read AND MERGEd) and correctly diverge for a self-ref / renamed FK. Pinned by the bench's twelve_table_projections_agree_down_to_keys (parity down to read columns + merge keys) and native's edge_sink_self_ref_reads_fk_column_merges_on_node_key.

3. The additive implementation

3.1 knut_bifrost::codegen (pure Rust, always compiled)

• SourceKind (Postgres|Iceberg), SinkKind (FalkorDb|GraphAr), PlanMode (Batch|Stream), Tier (T1..T4), CodegenOptions (spec defaults: 1000 rows, coalesce 1, Iceberg→FalkorDB batch).
• NodePlan / EdgePlan / EdgeProposal / GraphPlan — the canonical carrier.
• GraphPlan::from_mapping(&MappingSpec, &CodegenOptions) -> Result<GraphPlan> — walks tables schema-then-name, vertex→node, edge/fk_edges→edge (direction src→dst), custom→SURFACED in WARNINGS (never dropped). Errors on the spec's blocking gaps: stream on a non-Iceberg source; a node with no merge key.
• GraphPlan::to_pyspark() — projection (a). The Python hard rules: FAIR scheduler; ensure_indexes at startup; nodes-before-edges; sink client built inside the partition fn; coalesce(N) before foreachPartition; idempotent UNWIND $rows AS r MERGE in BATCH_ROWS chunks; streaming wraps the same writer in one foreachBatch with one checkpointLocation.
• GraphPlan::with_proposals(...) — T4 proposals stamped Tier::T4InclusionDependency, emitted commented-out with evidence, never as live code.
• GraphPlan::contract() — the canonical labels/keys/directions/batch digest the invariant compares against.

3.2 knut_thund::rel2graph (feature rel2graph → dep knut-bifrost + dsl)

• to_thund_pipeline(&GraphPlan) -> Result<Pipeline> — projection (b). One Dataset per node (node__<Label>) / edge (edge__<From>__<Rel>__<To>), each OutputType::Sink with the graph target in knut.* properties, fed by a projection Flow over the source table (pruned to key+props for a node / from_col+to_col+props for an edge — the FK VALUE columns, not the merge-key names; see §2.1). Streaming ⇒ Materialization::Incremental. validate()d before return.
• to_pipeline_ron(&GraphPlan) -> Result<String> — the pipeline.ron (thund DSL), round-trips through authoring::dsl::from_ron.
• The prop::* constants name the knut.* property namespace a future graph ExecBackend reads.

3.3 Tests (real, deterministic, no cheating)

• bifröst codegen::tests (5): canonical ordering + byte-identical determinism; the Python hard rules; the single-brace Cypher regression guard (the literal {/} that broke the kube build if emitted through format! — built via raw push_str, asserted present, no {{/}} leak); streaming needs Iceberg + one foreachBatch; custom step surfaced; T4 proposals commented-out never applied.
• thund rel2graph::tests (3): two_projections_agree — node (label,key) parsed out of the PySpark index statements vs. read from the Pipeline knut.* properties; edge (from,rel,to) parsed from _write_edges(...) call sites vs. Pipeline properties; batch from BATCH_ROWS vs. property — all three asserted equal, both non-vacuous. Plus: the Pipeline is real thund IR (validates, nodes-before-edges, round-trips RON); streaming ⇒ incremental datasets.
• Scaled parity — knut-rel2graph-bench's twelve_table_projections_agree_down_to_keys re-runs the two-projections invariant on the full 12-table GraphPlan and extends it to the endpoint read columns AND merge keys (the 7-tuple from,rel,to,from_col,to_col,from_key,to_key, parsed independently from the PySpark selectExpr+_write_edges and the thund props). This is what proves the self-ref/composite cases agree, not just the 2-table toy. The native seam is locked by edge_sink_self_ref_reads_fk_column_merges_on_node_key (REPORTS_TO reads manager_id, MERGEs on employee_id, no self-loop).

4. Determinism (the spec's byte-identical law)

GraphPlan::from_mapping canonicalizes: tables schema-then-name ((namespace, leaf) key), node props ordinal, nodes-before-edges (two vecs), edges sorted (from_label, rel, to_label). No timestamps, no randomness; the one HashMap in the source (rename) never reaches the plan. GraphPlan fields are stable; thund Dataset.properties is a BTreeMap (sorted). plan_is_canonical_and_deterministic pins byte-identical PySpark on repeat; the thund RON is stable via the BTreeMap + declaration order.

5. Honest worklist — spec-but-not-yet-implemented

Everything below is described by the code (types/enums/props exist) but the behavior is a follow-up. Nothing here is faked green.

1. Live catalog reflection (T1/T2/T4 resolution). Tier is recorded, but a plan's tier is currently whatever CodegenOptions.declared_tier says (default T3 = the reviewed mapping). The real passes — T1 Postgres information_schema PK/FK, T2 Iceberg identifier-field-ids, T4 inclusion-dependency LEFT ANTI JOIN orphan test — are not wired. GraphPlan accepts T4 EdgeProposals from outside; the pass that computes them (a live DB query) is TODO.
2. A graph-sink ExecBackend. (Largely LANDED.) The NativeBackend (built native + rel2graph, i.e. the falkordb feature) now turns each falkordb-node/-edge OutputType::Sink dataset into a real UNWIND … MERGE load via knut_bifrost::sink::FalkorDbSink — nodes-before-edges, both endpoints MERGEd, composite/hashed keys, parallel edges, typed projection, and the read-column vs merge-key split of §2.1. The SparkBackend structurally cannot (an SDP flow is a SQL relation, an SDP output a table — nowhere for a Cypher closure), so SparkBackend::check refuses a graph-sink pipeline up front with an actionable message ("use NativeBackend or the PySpark projection"). Remaining: the live FalkorDB round-trip is exercised only under an #[ignore]d container test; GraphAr sink write is still description-only.
3. concurrency has no thund IR home. Carried in GraphPlan + PySpark coalesce(N); mapping it onto a thund Dataset/Flow knob is TODO.
4. Clarification protocol (INTERACTIVE). The spec's ≤6-question QUESTIONS block on a blocking gap is not emitted; today from_mapping errors on the two hard gaps (no merge key; stream/source conflict) and SURFACES the rest in WARNINGS. INTERACTIVE=false "nodes-only + T4-commented" degrade path is TODO.
5. Mapping rule coverage. Junction-table collapse, composite-PK concat_ws('|',…) as _key, and the full type-mapping table (numeric→double precision warning, timestamp→epoch-millis, json→stringified) are documented in the spec and partially present in transform.rs/from_arrow_schema; codegen consumes whatever the reviewed MappingSpec already encodes and does not yet re-derive them.
6. CLI surface. No bifrost codegen subcommand yet; the API is library-only (GraphPlan, to_pyspark, to_thund_pipeline). Both projections + the native graph-sink construction are reachable from a real invocation via cargo run -p knut-thund --example rel2graph_native --features native,rel2graph (not only the bench/tests). The third, directly-materialised form (GraphMutation) is the engine seam every DB-free consumer calls.

6. FIVE ZEROS

Codegen is pure Rust string/IR building: zero rayon, zero shelling, zero cpu-wait (synchronous, no spin), zero copy beyond the owned strings an emitted artifact inherently is, Rust 100%. The one deliberate non-Rust output is the emitted PySpark text — a described Spark artifact, not a shelled process.