nordisk/skade

Table compaction — guide

Class: guide · Function: compaction. The what/why is in compaction-design.md.

When to compact

A table that is appended to continuously (many small append commits) grows one snapshot + one tiny file per commit, unbounded. When the snapshot count or tiny-file count climbs (a few thousand snapshots, or metadata in the hundreds of MB), compact it: the rebuild collapses the whole history to one snapshot in a handful of large zstd files. There is no separate expire_snapshots step — the rebuild is the expiry.

Compaction requires a quiescent writer for that table (call it from your single append actor). Concurrent readers are fine.

Compact a table

let report = warehouse.compact_table("events", None).await?;
println!(
    "compacted {} rows; was {} snapshots",
    report.rows, report.snapshots_before
);

// REQUIRED: reload any Table handle you were holding — the old one points at the
// retired metadata, and its next append would write into the dir you're about to
// GC.
let events = warehouse.table("events").await?;

// Delete the retired data dir only after a grace window longer than your longest
// in-flight query (readers resolved just before the swap still stream old files).
if let Some(old) = report.old_data_dir {
    // e.g. schedule after N minutes:
    tokio::fs::remove_dir_all(old).await.ok();
}

Compact and prune in one pass

The full rewrite is the one safe moment to drop rows (single writer, total rewrite). Pass a prune closure that transforms each batch — e.g. keep only rows newer than a cold-storage watermark:

let cutoff: i64 = 1_000;
let prune = move |b: RecordBatch| -> skade::Result<RecordBatch> {
    // return a batch with the unwanted rows removed (same schema)
    filter_rows(b, |id| id >= cutoff)
};
let prune_ref: &(dyn Fn(RecordBatch) -> skade::Result<RecordBatch> + Send + Sync) = &prune;

let report = warehouse.compact_table("events", Some(prune_ref)).await?;
assert_eq!(report.rows_pruned, /* rows the closure dropped */ n_dropped);

CompactReport.rows counts what survived, rows_pruned what the closure dropped (0 without a hook).

Guarantees

• Byte-identical data. Compaction rewrites files, never values; the surviving rows read back identically (verified in tests/compaction.rs).
• One snapshot out. The compacted table has exactly one snapshot regardless of prior history, so the growth trigger converges.
• Atomic + gap-free swap. name never resolves to nothing across the swap; the compacted data survives a crash/reopen. A concurrent SELECT never 404s on the transient scratch table.

Limits

• Unpartitioned tables only — compact_table errors on a partitioned table.
• No concurrent writer to the same table during compaction (a racing append is silently lost — it appended to the snapshot being retired).