nordisk/skade

skade incremental read (read_delta) — the snapshot diff

Table::read_delta(from, to) reads only the rows appended between two snapshots — not the whole table. skade/src/read.rs.

Diagram: lineage walk → per-snapshot op guard → manifest-list classify (Data→inserts / EqualityDeletes→identity-deletes / PositionDeletes→full-reload) → DeltaPlan → consumers polling commit_seq() then read_delta(cursor, latest).

Why it exists

iceberg-rust 0.9.1 (what skade pins) has no incremental / changelog scan. TableScanBuilder only offers snapshot_id(id) (time-travel to one snapshot) — there is no from_snapshot_id/to_snapshot_id and no changelog procedure (that is Spark-only). So an Iceberg consumer that wants a streaming delta (read each new snapshot's new rows, not re-scan everything) has nothing to call. read_delta fills that gap, built from the manifest layer, pure Rust.

The first consumer is knut-bifrost's Iceberg → FalkorDB streaming ingest: poll the table, read_delta(cursor, latest), inject only the new rows into the graph, advance the cursor. The second is modgunn (Móðguðr), whose Engine B world-delta watcher polls commit_seq() then read_delta(cursor, latest) over the shared warehouse to track every change to the dependency world — modgunn is a leaf consumer of skade (never of nornir), so this is its only change-sensing path. (modgunn/.nornir/modgunn-master-plan.md §1 CDC seam.)

How it works (manifest diff)

1. Lineage walk — from to, follow parent_snapshot_id() back to from (exclusive), collecting the new snapshot ids oldest-first.

2. Operation guard — if any new snapshot's summary().operation is not Append (overwrite/delete/replace), set needs_full_reload and return no batches; an append-only file delta would leave stale rows behind, so the caller re-reads to in full.

3. Operation set — Append (inserts) and Delete (identity/equality deletes) are both expressible as an additive delta. Overwrite/Replace rewrite rows in place — those set needs_full_reload.

4. Manifest-list scan — load to's manifest list. Each ManifestFile has added_snapshot_id + content. For manifests added by a snapshot in the new set, parse the avro and classify each Added entry by its data-file content type:

   • Data (in a Data manifest) → added_files (rows to insert).
   • EqualityDeletes → delete_files (one EqualityDeleteFile { path, equality_ids }; the identity columns that define the deleted rows).
   • PositionDeletes → needs_full_reload (delete-by-offset can't be mapped to a graph identity, so re-read).

5. Read — added_files → insert RecordBatches. delete_files are read on demand by read_equality_deletes(table, plan) → (equality_ids, batches) per file (the deleted-row identities). Both through the table's FileIO.

Returns (Vec<RecordBatch>, DeltaPlan); the DeltaPlan { snapshots, added_files, delete_files, needs_full_reload } lets a caller see/assert what the delta resolved to — inserts and deletes.

CDC deletes — both sides, pure Rust

iceberg-rust 0.9.1's Transaction only implements fast_append, which rejects non-Data content (validate_added_data_files), so it cannot write delete files. skade now writes them at the metadata layer instead:

Table::delete_equality(keys, equality_columns) (skade/src/delete.rs) writes an equality-delete parquet (the identity columns of the removed rows) + a v2 Deletes manifest + a manifest list that carries the existing manifests forward, then commits an AddSnapshot (operation = delete) + SetSnapshotRef through RedbCatalog::commit_table. The result is a real Iceberg equality-delete snapshot, read back by read_delta as a delete_file — a complete CDC delete with no Spark.

Limitations (honest)

• Identity (equality) deletes, not position deletes. delete_equality writes equality-delete files (delete by business-key value), which a graph upsert keyed on that id can resolve. Position deletes (delete by file+row offset) and Overwrite/Replace set needs_full_reload.

Tests

skade/tests/delta.rs (no container, real local warehouse):

• read_delta_reads_only_appended_rows — A(3)+B(2): the A→B delta reads only {4,5}, one file, cursor=B; a caught-up cursor reads nothing.
• read_delta_spans_multiple_snapshots — A→B→C: the A→C delta reads {3,4,5,6} (both new snapshots), two files, oldest-first lineage.
• read_delta_surfaces_equality_deletes — A(3) then a real equality-delete of id=2: the A→B delta has no inserts, one delete_file, no full reload; read_equality_deletes reads back the deleted identity {2}. The delete is produced by Table::delete_equality — the inject side is pure Rust too.

cargo test --test delta            # in skade/skade

SnapshotUtil parity in the vendored fork (vendor/iceberg-arrow58)

read_delta above resolves its snapshot lineage by delegating to the vendored iceberg-arrow58 fork's TableMetadata::snapshots_added_between (swarm #38, 2026-07-29) — it no longer reimplements the ancestor/parent-chain walk inline. The fork carries the general snapshot-range resolution as first-class TableMetadata methods — the metadata-layer primitives Java's SnapshotUtil exposes and that stock iceberg-rust does not ship in a released form:

Tested in vendor/iceberg-arrow58/src/spec/table_metadata.rs (snapshot_range_resolves_append_lineage, overwrite_in_range_is_not_append_only) — an append lineage resolves, an overwrite spliced into the range flips the append-only check to false, and both guards (unknown to, a from off the lineage) are asserted.

Wiring (swarm #38 — dedup collapsed). read_delta now calls metadata.snapshots_added_between(from, to) for the (from, to] chain instead of walking parent_snapshot_id itself; this gives the primitive its first production caller and removes the duplicate lineage walk (one codepath). read_delta's needs_full_reload gate stays a local explicit loop over the resolved chain — it is Delete-tolerant (Append | Delete are both expressible as an additive/CDC delta; only Overwrite/Replace force a full reload) and is therefore deliberately wider than the fork's is_append_only_between (Append-ONLY). Wiring that stricter primitive into the gate would wrongly full-reload a pure-Delete window and shift observable behaviour for the knut-bifrost + modgunn CDC consumers, so it is intentionally not used there. The public signature of read_delta, the DeltaPlan shape, and read_changelog are unchanged; the delta suite proves the observable result is identical (read_delta_delegates_to_snapshots_added_between asserts plan.snapshots == snapshots_added_between(...) across an append range and an overwrite range).

Behaviour-identity note / two edge-case refinements. For every input a real CDC consumer passes (from = a prior snapshot that IS an ancestor of to on the branch, no expired ancestor mid-window) the delegation is byte-identical to the old inline walk. It refines exactly two unreachable-in-practice edges: a from that is not an ancestor of to now errors (the old walk silently returned the whole lineage — the #2337 footgun, latent re-stream-everything bug), and a dangling (expired) parent now truncates the walk rather than erroring. Both match the fork's Java-parity semantics and are not exercised by any consumer or test. Flagged for Rickard as a (strictly-more-correct) design decision, not a forced behaviour change.

Consumer verification (swarm #38). modgunn builds green against the changed skade (native path = "../skade/skade" dep) and its delta suite passes (cdc_delta_scans_only_newly_added_components et al. drive table_delta → read_delta). knut-bifrost consumes the published skade by deliberate policy (the removed [patch.crates-io] skade note in knut/Cargo.toml); a local path-patch to verify against the changed skade is blocked by an unrelated dual-iceberg-arrow58 version collision (knut pins published 0.9.8, the local fork vendors 0.9.1) — not a semantic shift. Since read_delta's public API/behaviour is unchanged, knut-bifrost is unaffected.

Why the fork carries this — the upstream iceberg-rust assessment (2026-07-29)

Checked apache/iceberg-rust before implementing (the "check upstream first" rule):

• #1469 "support incremental scan between 2 snapshots" and #1470 (its PR): both closed as not-planned / stale. The original request that named SnapshotUtil, IncrementalScan, BaseIncrementalAppendScan — abandoned.
• #2241 "Add helpers to scan snapshot ancestors" — closed, done via #2285: upstream main did land ancestor helpers. But they are not in the 0.9.1/arrow-58 line the fork pins, so the fork provides its own (the arrow-58 pin is load-bearing for RecordBatch interop across the skade stack, §Cargo).
• #2337 "feat: Incremental append scan" — OPEN, unmerged, awaiting review. Its own description documents a footgun: if from_snapshot is not on the lineage it silently returns all ancestors of to. The fork's snapshots_added_between errors in exactly that case — a deliberate divergence, not an oversight.
• #2152 "incremental reads between snapshot-ids" and #1636 "(incremental) changelog scan for CDC" — both open, unimplemented upstream.

Fork plan / what's genuinely missing (reported, not yet in the fork):

1. File-planning IncrementalAppendScan — turning the resolved id range into a planned set of added data files (the scan layer above these metadata primitives). skade already has an equivalent at its own layer in read_delta's manifest walk; upstreaming/adopting #2337's scan once it merges is the consolidation path.
2. Changelog / net-changes scan (#1636) — CDC over the changelog view (INSERT/UPDATE_BEFORE/UPDATE_AFTER/DELETE). skade covers the append+equality- delete slice today via read_delta + delete_equality; the full changelog procedure is Spark-only upstream and remains a frontier for the fork.

The must-have for the Iceberg↔Spark streaming gap's native incremental path — resolve a correct, lineage-guarded snapshot range and know whether it is append-only — is the piece implemented here.