nordisk/knut

CDC — the research trail and the roads not taken

Sister to cdc-design.md, which names only the chosen design. This file holds the sources the design rests on and the alternatives that were rejected.

Sources (checked 2026-07-22, not recalled)

Spark 4.2 native CDC

Where the docs are wrong-by-omission: the 4.2.0 SQL reference does not index a CHANGES page. docs/latest/sql-ref-syntax.html's Data Retrieval list has no CDC entry and docs/latest/sql-ref-syntax-qry-select-changes.html returns 404. Anyone checking the docs site alone would conclude the feature does not exist. It does; the page is missing. This is why the grammar was read from the source tag.

Iceberg's side

knut's own tree had already recorded the runtime half of this: containers/spark-iceberg/Containerfile ("no 4.2 runtime yet") and crates/knut-pipelines/src/server.rs (DEFAULT_IMAGE_41, with 4.2.0 GA dated 2026-07-14 as a named alternate). The connector half — that Iceberg implements no Changelog at all, so the clause would not resolve even with a runtime — is new here, and it is the fact that forced detection to be a probe rather than a version comparison.

Alternatives rejected

Choose the backend from config. A cdc_backend: spark_native | iceberg key in the mapping YAML. Rejected: it makes the operator responsible for a fact only the server knows, and it fails in the worst way — a mapping that says spark_native against a 4.1 server errors every round, and one that says iceberg against a future CDC-capable connector silently keeps the slow path forever. Detection is free (one version request + one zero-row probe, cached per table).

Choose from the version string alone. >= 4.2 → native. Rejected by the Iceberg finding above: the engine having CDC and this table having CDC are different questions, and today the answers differ. A version-only check would have shipped a path that errors on every real table.

Trust the probe's absence of an error as "it works". The probe is … CHANGES FROM VERSION 0 LIMIT 0. It proves the clause resolves — the analyzer found a Changelog on the connector — which is exactly the question. It does not prove the data is correct; the round-trip test does that.

One _change_type vocabulary, imposed. Normalise everything to Iceberg's UPPER-CASE spelling at the source and delete the Spark one. Rejected: the wide batches flow to the listener untouched (that is what makes the pushdown a pure narrowing), so the spellings genuinely coexist in the pipeline. Making ChangeOp::parse accept both, and canonicalising only at the renderable edge, keeps one vocabulary without rewriting anyone's data.

A Vec<ChangeRow> struct for the UI. Rejected: the whole pipeline is Arrow, FalkorDB's loader takes RecordBatch, and a row-struct would force a transpose per batch and lose zero-copy. The Struct columns give the UI the same per-row ergonomics with none of that.

futures::Stream for the feed. Rejected: futures is an optional dep (feature flight) and the core is deliberately dependency-light. tokio is already a hard dep, so tokio::sync::mpsc gives bounded, backpressuring delivery for free. If a Stream impl is ever wanted it is a wrapper, not a rewrite.

A second incremental reader in knut. Rejected outright (L5): skade owns read_delta / read_changelog / project_columns and knut consumes them. Where skade's surface is missing something (a projection on the delta scan), that is logged as skade-side work rather than worked around with a private copy.

A second row-filter implementation for CDC. Nearly shipped, then rejected. 89d7e31 had just landed mapping::apply_row_filter, and the CDC path needed one behavioural difference: a row whose filter column is NULL must be kept (that is a widened DELETE), whereas the scan-planner semantics every other mode must match is drop. Twinning the kernel lowering to get that would have been two predicates that could drift. Instead FilterNulls { Drop, Keep } was added to the one implementation. Same code, one explicit policy argument, and the difference is now a named concept instead of an accident.

The bug this work uncovered

mapping::apply_row_filter ends in arrow::compute::filter_record_batch, which treats a NULL mask entry as false — the row is dropped. Since a CDC DELETE row carries nulls in every non-identity column, a mapping with any filter: in mode: cdc silently dropped every delete. It was live on main and no test covered it. See cdc-design.md §f4 and the RED-when-broken test cdc::post_filter_keeps_unjudgeable_rows_where_the_scan_planner_would_drop_them.

That is the concrete argument for why "never lose a delete" is the invariant the CDC module is organised around, rather than a slogan: the failure is one default argument away, it produces no error, and the graph just quietly keeps a node that no longer exists.