Iceberg store & seal — design
Class:
design· siblings:iceberg-guide.md·plugins-design.md·signing-design.md. Strict: this file states only the shape we run — the dynamic store, the seal, the reboot-capable writer. No alternatives, no why.
f0 — dynamic store ⇆ static seal (the whole)
A znippy archive normally writes its metadata inline: an Arrow-IPC index tail ending in [ manifest Arrow IPC ][ 8-byte MAGIC "ZNPYMIDX" ][ u64 manifest_offset ]. The iceberg subsystem splits that in two so an archive can be grown live and then frozen:
| mode | metadata lives in | blobs live in | property |
| dynamic store | skade/Iceberg tables in a catalog.redb warehouse |
.znippy sidecar (pure blobs) |
survives restart, accepts new commits, queryable live |
| sealed export | rewritten native arrow-ipc tail, back inside the .znippy |
same blobs, verbatim | static, immutable, content-addressed; warehouse no longer needed |
Seal law: the seal is content-addressed and blob-preserving — blobs are reused verbatim (no recompress); only the metadata tail is rewritten, and the result is byte-identical to what a one-shot compress would have produced.
Reuse map: tokio and the iceberg crate are confined to znippy-iceberg; every other workspace crate is tokio-free. The whole workspace agrees on one Arrow major (Arrow 58). The sealed artifact opens with the ordinary ZnippyArchive reader.
f1 — the dynamic store (IcebergSink)
znippy-iceberg/src/lib.rs:77 IcebergSink is an ArchiveMetaSink implementation. It keeps the archive's sub-index rows in skade/Iceberg tables backed by a durable catalog.redb (RedbCatalog, skade being the default catalog). The .znippy sidecar holds pure blobs only; all metadata lives in the warehouse.
Because the metadata is a live Iceberg store, it:
- survives a process restart (durable redb catalog),
- accepts new commits — resume writing into the same archive,
- is queryable live — DuckDB/Polars over the Iceberg tables.
Read side: znippy-iceberg/src/lib.rs:194 IcebergZnippyReader reads blobs from the sidecar against metadata from the warehouse.
f2 — the seal (seal)
znippy-iceberg/src/seal.rs:69:
fn seal(sidecar, warehouse, namespace, out) -> SealReportSnapshots the dynamic store into a static, immutable, content-addressed native arrow-ipc .znippy. Mechanics:
- Blobs: reused verbatim from the sidecar — no re-read, no recompress.
- Metadata tail: rewritten — sorted lookup + fst trie + manifest + the
ZNPYMIDXfooter — byte-identical to a freshcompressof the same content. - Output opens with the ordinary
ZnippyArchivereader; the warehouse is dropped.
f3 — the reboot-capable native writer (ArrowIpcSinkAppend)
znippy-common/src/meta_sink_append.rs:73 ArrowIpcSinkAppend is a clone of the native arrow-ipc writer that supports write → restart → resume → seal directly against a native .znippy (no warehouse):
| fn | does |
open_existing(path) |
reads the full manifest incl. reserved sections; finds blob_end = the lowest index_offset; recovers existing data rows from the sorted lookup; positions the cursor so new bytes overwrite the rebuildable tail |
append_files() |
codec-compresses new files → appends blobs past blob_end → re-seals the merged old+new rows (a path already in the archive is REPLACED, not duplicated — see AppendReport::rows_replaced) |
The tail (lookup/trie/manifest/footer) is treated as rebuildable: it is overwritten on every resume, so blob bytes are never disturbed. Byte-exactness of write→restart→resume→seal is proven by znippy-iceberg/tests/lifecycle_seal.rs::iceberg_write_restart_resume_seal_static.