znippy-iceberg — crate card
Reverse-engineered (from source — verify, don't trust the name)
- One-line pitch: Optional Apache Iceberg backend for znippy's metadata layer — writes the archive sub-indexes as Iceberg tables while blobs stay in the
.znippysidecar. - Does:
IcebergSinkimplementsArchiveMetaSink(materializes each sub-index as an Iceberg table instead of inline Arrow-IPC);IcebergZnippyReaderimplementsZnippyReader(reads tables back, serves file bytes from the sidecar).seal.rsexposesseal/SealReport. Confines all async/tokio + Iceberg + Arrow to this crate, bridging async→sync via a private multi-thread runtime +block_on. Handles the Arrow-58↔57 version bridge (IPC round-trip) and Iceberg unsigned-int widening/unwidening. - Key deps:
znippy-common(traits/schema/codec),skade/skade-katalog(embedded redb-backed Iceberg catalog + file:// warehouse; sibling repo, version-only dep),tokio+futures(confined here),arrow-array/arrow-schema/arrow-ipc/arrow-cast58. - Shape: lib, ~800 LOC:
lib.rs(536),seal.rs(264); testsround_trip.rs,lifecycle_seal.rs. - Observed role in the workspace: Pulled in only by
znippy-cli --features iceberg(and xtask's iceberg bench). Selected at runtime bycompress --meta-format iceberg --warehouse …. Isolated so the sync, tokio-free core crates stay unaffected.
Fill in (one line each — only the human knows these)
- Primary consumer(s):
- Why it exists (vs the obvious alternative):
- Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
- Release cadence & independence (rides a parent? own repo someday?):
- Status (production / prototype / experimental / stubbed):
- Must-NOT-change invariant:
- Biggest known gap or TODO:
- Anything a newcomer would get wrong about it: