nordisk/knut

CDC for beginners — "what changed?", explained slowly

What problem is this solving

You have a table. Yesterday it had a million rows. Today it has a million and five, one row got edited, and two got deleted. You want a screen that shows those eight events — not the million rows.

That list of events is a change feed, and producing it is called CDC (change data capture).

The naive way, and why it is bad

Read the whole table today, read the whole table yesterday, subtract. That works and it is terrible: you pay for a million rows to learn about eight, and you cannot tell an edit from a delete-plus-insert.

Table formats like Apache Iceberg keep every commit as a numbered snapshot, so you can ask "what happened between snapshot 41 and snapshot 42" and get back only the changes. That is what knut does.

Three ways to ask, and knut picks one

Annoyingly, there are three different mechanisms depending on what is on the other end:

1. Spark 4.2's own CDC. New in Spark 4.2 (July 2026). You write SQL:

SELECT * FROM my_table CHANGES FROM VERSION 41 TO VERSION 42

Spark hands back the changed rows plus three extra columns saying what kind of change each was.

1. Iceberg's CDC. Older, and what Spark 4.1 and earlier have. You call an Iceberg stored procedure that builds a temporary view of the changes, then SELECT from that view.

2. No Spark at all. knut can read Iceberg's files directly with the skade library — pure Rust, no Java, no cluster. It works out the changes from Iceberg's own bookkeeping files.

You do not choose. knut asks the server which it is, tries the fast path, and uses whatever actually works.

The plot twist

Spark 4.2's native CDC is real and it works — but only for tables whose connector has been taught to speak it. Iceberg has not been taught yet. Iceberg does not even publish a Spark 4.2 build at the time of writing.

So if you point knut at a Spark 4.2 server holding Iceberg tables, it will:

1. see "4.2 — good, native CDC exists";
2. actually try it (a zero-row test query);
3. get an error back;
4. say so, loudly, and use Iceberg's procedure instead.

That last step is the important one. A tool that quietly used the fallback and said nothing would leave you believing you were on a code path you were not on. knut writes fell_back: true plus the engine's own error message into its log and into the state file it keeps on disk.

What comes out

One row per change event, with five columns:

So a delete has a before and no after. An insert has an after and no before. An edit has both — that is your diff, ready to paint red/green.

They arrive in batches, as they are produced, so a change pane can start showing rows immediately instead of freezing until the whole scan is done.

Why "don't read columns you don't need" matters here

If your screen only shows a person's id and name, there is no reason to read their address, their eight payload blobs, and everything else. Telling knut which columns you need is called a projection, and telling it which rows you care about (country = 'SE') is a predicate.

knut already did this for whole-table reads. It did not do it for change feeds — every CDC read pulled every column, always. That is now fixed, and knut.cdc_pushdown_read measures the difference.

One caveat the honest version has to include: on the no-Spark path, knut trims the columns after reading the file, not before. So it saves memory and downstream work, not disk reads. Making it skip the disk reads too is a change inside skade, not inside knut.

The one rule that must never break

A change feed must never silently lose a delete.

Everything else is a performance question. If a delete goes missing, your graph keeps a node that no longer exists and nobody finds out. So:

• an unrecognised change-type is an error, not a skipped row;
• a null change-type is an error;
• when a filter cannot judge a row (because a deleted row's columns are blank — which is normal), knut keeps it rather than guessing;
• there is a test that inserts, edits and deletes rows in a real Iceberg table and fails if the delete does not come out the other end.