.nornir/iceberg-guide.md · nordisk/znippy

Iceberg store & seal — guide

Class: guide · audience: a techie growing an archive live, then sealing it. The what is in iceberg-design.md; siblings plugins-guide.md · signing-guide.md.

f0 — grow live, then freeze (the whole workflow)

Two phases. First compress with metadata in an Iceberg warehouse (dynamic, resumable, queryable); later seal it into a static content-addressed .znippy. Both need the iceberg feature:

cargo build --features iceberg

f1 — dynamic compress into a warehouse

znippy compress -i <dir> -o sidecar.znippy \
    --meta-format iceberg --warehouse <dir>

sidecar.znippy holds pure blobs; all metadata goes into the warehouse (catalog.redb + Iceberg tables).
The store is durable: kill the process and re-run compress with the same --warehouse to resume — new commits land against the existing rows.
It is queryable live while you write — point DuckDB or Polars at the Iceberg tables.

f2 — seal to a static artifact

znippy seal -i sidecar.znippy --warehouse <dir> [--namespace NAME] -o sealed.znippy

sealed.znippy is content-addressed and blob-preserving: blobs copied verbatim (no recompress), only the metadata tail rewritten. The result is byte-identical to a one-shot znippy compress of the same content, and opens with the ordinary reader:

let archive = znippy::ZnippyArchive::open("sealed.znippy")?;   // warehouse not needed

Entry point: znippy-cli/src/lib.rs Commands::Seal (:479).

f3 — write → restart → resume → seal, no warehouse

Against a native .znippy directly (via ArrowIpcSinkAppend, znippy-common/src/meta_sink_append.rs):

open_existing(path) reopens the archive, finds blob_end (lowest index_offset), and recovers existing rows from the sorted lookup.
append_files() codec-compresses new files, appends their blobs past blob_end, and re-seals the merged old+new rows — the tail (lookup/trie/manifest/ZNPYMIDX footer) is rewritten, blob bytes untouched.

Appending a relative_path the archive already contains REPLACES it (last writer wins): the pre-existing rows for that path are dropped from the re-sealed index and its old blob bytes become unreferenced dead payload. AppendReport::rows_replaced counts them. Without that, the archive would carry two live row-sets for one path and a read would silently return the stale copy — replace semantics is what makes znippy append safe to re-run over a changed directory.

The full cycle is byte-exact — see znippy-iceberg/tests/lifecycle_seal.rs::iceberg_write_restart_resume_seal_static.