nordisk/knut

Bifröst — tables → graph, reasoning

Class: reasoning. This file holds the why, the trade-offs, every design we rejected, and the honest limitations. The descriptive shape we run is in bifrost-design.md, which by rule mentions none of this. If you want to know "why not X?" — X is here. Reuse: bifrost.uses = [falkordb, graphar].

Why the fabric at all

Bifröst exists to make lakehouse tables queryable as a graph continuously, without a batch export step. The whole design serves four goals, and every law below is in service of them:

1. Arrow end to end. Spark Connect and skade both hand back Arrow RecordBatches; they flow through transforms and into the FalkorDB loader untouched. No row-object marshalling, no intermediate serialization.
2. Declarative first, code when needed. ~90% of table→graph mappings are "this table is a label, these two columns are an edge" — those are YAML vertex/edge/fk_edges steps. The remaining 10% get a full Rust Transform with the whole Arrow API. The config stays declarative; the Rust stays out of config files (custom: name + a registry).
3. Exactly-once-ish. Per-table progress is a cursor, data ships once, replays are safe.
4. Source-agnostic. One TableSource trait; Spark, Spark-free Iceberg, and file drops are all just implementors.

Why state-after-sink (exactly-once-ish)

The cursor is written only after the sink round succeeds. A crash between sink and cursor replays the whole round — and because the sink is MERGE keyed on a stable id, a replay is idempotent (the second MERGE is a no-op). This is cheaper and simpler than a two-phase commit and gives at-least-once delivery with idempotent effect = effectively exactly-once. The alternative — advancing the cursor first, or transactionally — either drops data on a crash or couples bifröst to sink-side transactions it does not own.

Why MERGE both endpoints on every edge

Tables sync in arbitrary order, so an edge can land before the table that owns one of its endpoints. If the sink only matched existing nodes, that edge would be dropped. Instead every edge MERGEs both endpoints, creating a placeholder node on the id key; when the owning table later syncs, its vertex MERGE on the same id enriches that placeholder in place. No ordering constraint on the mapping, no lost edges.

Why REFRESH TABLE before each Spark poll

Spark caches Iceberg metadata per session. A long-lived poller that never refreshes would keep seeing its own session's stale snapshot and miss every other writer's commits. The Spark source issues REFRESH TABLE before reading the snapshot list so it sees the true latest commit each round. (The Spark-free IcebergSource reopens the catalog per poll, so it has no equivalent staleness.)

Why backoff only on a no-progress round (idle ≠ failure)

An idle round — the source simply had no news — is the normal steady state of a poller, not a failure. Backing off on idle would slow a healthy quiet system to a crawl. So a round is a failure (RoundReport::is_failure) only when ≥1 table was attempted and every attempt failed; an idle round (nothing synced, nothing failed) keeps base cadence, and a partial round (some tables synced, some failed) counts as progress and resets the schedule. Consecutive failures double the sleep to a cap; the first progressing round resets it.

Why declarative-first, code-when-needed (not one or the other)

A pure-YAML DSL would eventually grow a scripting language to express the 10% hard cases — reinventing Rust badly. A pure-Rust API would push the trivial 90% into code that a data engineer shouldn't have to touch. The split keeps each in its lane: YAML for shape, a custom Rust Transform for logic, joined by name through the registry.

Rejected: a plugin system (why a compile-time registry, for now)

The fabric is a compile-time TransformRegistry: you depend on knut-bifrost and hand it your Arc<dyn Transform> impls before the listener starts. We did not build a plugin loader. Compile-time registration covers the embedding use case fully, with zero dynamic-load surface. When out-of-process transforms were needed, the answer was not a plugin ABI but two implementors of the same Transform trait that keep GraphMutation as the contract: WasmTransform (a sandboxed wasmi guest, Arrow IPC in/out) and FlightTransform (a polyglot Arrow-Flight DoExchange service, any language, out of process). Both are just custom transforms — no new plugin machinery, same currency.

Deferred / future sources

• Iceberg REST catalog (via iceberg-rust) — a third TableSource for shops that run a REST catalog instead of skade's embedded one. Slots behind the same trait; not yet built.
• Debezium CDC — a Debezium topic → batches → the same fabric, as an alternative CDC feed to the Iceberg changelog path. Deferred; the SyncMode::Cdc routing already exists to receive it.

(Both the Spark-free Iceberg source and Airflow AIP-82 emission started life on this list and are now shipped — see bifrost-design.md f1/f6.)

The honest limitation: iceberg-rust has no incremental scan

iceberg-rust 0.9.1 (what skade pins) does NOT expose an incremental / changelog scan. Its TableScanBuilder has snapshot_id(id) (time-travel to one snapshot) but no from_snapshot_id/to_snapshot_id and no create_changelog_view equivalent — that procedure is Spark-only. So the Spark-free delta had to be built from the manifest layer by hand (skade's Table::read_delta): walk the snapshot lineage back to the cursor, classify each Added manifest entry (Data → inserted parquet, EqualityDeletes → a delete file), and read only those files. That manifest diff is the reason a Spark-free delta exists at all, and its two honest edges:

• Equality deletes, not position deletes. A Delete snapshot with equality-delete files (delete by business-key value) is a first-class CDC delete (_change_type = DELETE → VertexDelete → DETACH DELETE), exercised by a real unit + container test. Position deletes (delete by file + row offset) can't be mapped to a graph identity, so they set needs_full_reload and the whole target snapshot is re-read — correctness over speed.
• needs_full_reload is un-exercisable through this stack. iceberg-rust 0.9.1's Transaction only does fast_append, so skade cannot produce an overwrite or a position-delete to stage a test for it. The branch is defensive for a table some external engine (Spark, PyIceberg) overwrote; it is covered by the code path and this doc, not a live test.

Why the spark-connect fork is vendored in-tree (off-box buildability)

The spark source needs spark-connect, but upstream 0.2.2 (the newest published release) still pins arrow 56 — which dragged a whole second arrow-56 tree into knut just for the Spark path, forcing an IPC round-trip bridge to hand its batches to the arrow-58 workspace. The fix was a fork that bumps only the arrow version (56 → 58, no source changes — the crate's Arrow-IPC surface is API-compatible across the bump), so the entire dependency graph stays arrow-58-only.

We vendored that fork into vendor/spark-connect-arrow59/ rather than pointing the patch at an external sibling checkout. The reason is off-box buildability: a [patch.crates-io] at a ../sibling path means a bare git clone knut cannot build the spark feature at all. In-tree, with every proto committed and a vendored protoc in build.rs, the Spark path builds from a clone alone — no sibling repo, no system protoc, no network. It reverts to the crates.io crate the moment upstream ships an arrow-58 release (see the header of vendor/spark-connect-arrow59/README.md).

This closed the spark-connect leg of off-box build. The remaining bare-clone blockers — the root [patch.crates-io] on skade, plus sibling path deps on draupnir, nornir, znippy-zoomies, facett-pop and facett-pipeline — are now gone too: every one of those crates had published a version carrying the API knut needs, so each is a plain crates.io dependency. skade's equality-delete surface (DeltaPlan::delete_files, read_equality_deletes, Table::delete_equality) shipped in skade 0.5.0, which retired the patch entirely. A bare git clone knut && cargo build --workspace now resolves.

Two properties of cargo make this worth stating plainly, because both were mis-assumed in this tree at one point:

• A path that escapes the repo breaks resolution for the whole workspace, not just the crate declaring it — manifests are read before features are considered.
• Neither optional = true nor an accompanying version = "…" protects you. The first blocker a bare clone actually hit here was draupnir, which was both optional and versioned.

crates/knut-tracing/tests/no_sibling_path_deps.rs walks every manifest in the repo and fails if any path resolves outside the repo root, so this cannot silently come back. In-repo relative paths and vendor/ are what it permits — vendoring in-tree, as spark-connect is, remains the sanctioned escape hatch for a crate that genuinely cannot be published.

Known gaps

Composite / multi-label MERGE keys and edge-only changelog deletes remain open. TLS/auth on the /metrics endpoint is out of scope — front it with a reverse proxy if you expose it beyond a private scrape network.