nordisk/skade

Table compaction — rebuild-and-swap — design

Class: design · Function: compaction · Uses: commit, write, catalog. Only the shape that runs; the observed pathology it fixes (unbounded snapshot/metadata growth under continuous fast_append ingest → OOM) is the motivation, not an alternative — there is no second design here.

f0 — why a rebuild, and the shape it takes

iceberg-rust 0.9.1's only writer is fast_append: each commit adds one snapshot and one manifest and carries the whole manifest list forward. A long-lived, continuously-appended table therefore grows its snapshot log, manifest list, and tiny-file count without bound — thousands of snapshots, GBs of metadata, and eventually process OOM (observed on the garmr dogfood warehouse: 13 GB / 5869 snapshot files, RSS peak 7.2 GB).

Compaction fixes this by rebuilding the table rather than editing its history: stream the current rows into a fresh table written as a handful of large zstd files in exactly one commit, then atomically swap that table into the live name. The compacted table has one snapshot regardless of size, so the growth arithmetic converges — no separate expire_snapshots pass is needed. Peak memory is one Parquet row group; a large table is never materialised whole.

f1 — Warehouse::compact_table (the orchestration)

warehouse.rs · compact_table(name, prune) -> CompactReport

1. Load the live table; reject partitioned tables (a stream can span many partition keys, and skade appends are single-partition).
2. Create a uniquely-named scratch table <ns>.<table>__c<nanos> with the same Arrow schema.
3. rebuild_into: scan the old snapshot to Arrow and feed the stream to append_stream_props (§f2) with ZSTD + a 128Ki row-group target → one commit, one snapshot on the scratch table. An optional prune closure transforms each batch en route (e.g. drop rows already sealed to cold storage — the one safe moment to shrink, since the writer is single and the rewrite is total).
4. On any failure in step 3: drop the scratch table and remove_dir_all its partial data dir, then return the error — nothing leaks to the next restart and the live table is untouched.
5. catalog.swap_table (§f3) atomically repoints name at the scratch metadata.
6. Return CompactReport { snapshots_before, rows, rows_pruned, old_data_dir }.

Caller obligations after return (documented on the fn): reload any held Table handle for name (the old handle points at retired metadata), and delete CompactReport::old_data_dir only after a grace window longer than the longest in-flight query (readers resolved just before the swap keep streaming the old files).

Concurrency contract: no concurrent writer to name (garmr calls this on its single append actor); a racing append copies the entry snapshot and would be silently lost. Concurrent readers are fine — the swap is gap-free.

f2 — append_stream_props (the one-commit stream writer)

write.rs · append_stream_props(catalog, table, batches, map, props) -> (IceTable, rows_in, rows_out)

Rolls the incoming batch stream into large Parquet files via a RollingFileWriterBuilder + DataFileWriterBuilder, applies the optional map per batch, recasts to the table's Arrow target, and commits all the rolled data files in a single fast_append transaction. Unlike per-group append (one snapshot + manifest each), the result gains exactly one snapshot. Empty input (or a map that drops everything) commits nothing and leaves the table at its current state. Peak memory is one row group. Unpartitioned tables only.

f3 — RedbCatalog::swap_table (the atomic pointer swap)

catalog.rs · swap_table(live, temp, new_snapshot) -> Option<String> — one redb write transaction, Durability::Immediate:

1. Read temp's metadata_location (error if absent); read live's prior location.
2. Remove temp's pointer; insert temp's location under live — no instant where live resolves to nothing (a two-step rename would let a crash leave live absent, which table_or_create would silently re-create as an empty table → data loss).
3. Prune the commit log for both keys over the full [id 0 .. id -1] suffix range (retired entries point at metadata about to be GC'd, and an unpruned log — plus the Ragnar index rebuilt from it — is the same slow-OOM class compaction exists to fix), then re-seed new_snapshot under the live key so log and pointer stay consistent.
4. commit() (one fsync), then L1 pointer-mirror write-through under the lock.

Pointer-only, like rename_table: no data files are moved or deleted — the caller owns physical reclaim (CompactReport::old_data_dir).

f4 — the reader race (datafusion providers)

A table listed a moment ago can be removed by a concurrent swap (the scratch table) or a drop before a session loads it. Both eager providers — the sql module's per-session registration (warehouse.rs) and the vendored IcebergSchemaProvider (vendor/iceberg-datafusion-arrow58/src/schema.rs) — now skip a TableNotFound on load instead of failing the whole session: an unrelated SELECT … FROM events must not 404 on a transiently-absent scratch table.

f5 — bounded handle cache

table_cache.rs · DEFAULT_TABLE_HANDLE_CACHE_CAPACITY dropped 16_384 → 512. Each cached handle pins its parsed TableMetadata (O(snapshots)) plus a scan-populated ObjectCache, and under continuous ingest every commit inserts a new metadata_location — so a generous cap steadily pins gigabytes (observed on garmr's 8 GB SOC host). 512 ≈ tens of MB worst case while still covering every hot table.

Observability

compact_preserves_content_and_collapses_snapshots (RED-when-broken, skade/tests/compaction.rs): 20 tiny appends → 20 snapshots; after compact_table the reloaded table's snapshot log is exactly 1, the many tiny files are coalesced, and the row content is byte-identical (id → score-bits + name multiset) before vs after — and again after a warehouse reopen (durable, gap-free). compact_with_prune_drops_selected_rows_only asserts the prune hook drops exactly its selected rows and keeps the rest byte-identical, with CompactReport.rows / rows_pruned accounting for the split. Reverting the swap turns both red.