nordisk/znippy

znippy-compress — crate card

Reverse-engineered (from source — verify, don't trust the name)

• One-line pitch: The write path — turns a directory (or a stream of in-memory entries) into a .znippy archive with parallel chunked compression.
• Does: compress_dir (in slot_packer.rs) does two-pass directory compression: pass 1 handles BIG files (> 200 MB slice) via sequential chunked reads, pass 2 reads SMALL files into slots and extracts metadata in-memory, writing the Arrow-IPC index incrementally per pass. stream_packer.rs provides compress_stream/StreamCompressor built on the no-barrier Gatling engine for already-in-RAM entries. Uses io_uring batched opens+reads for the small-file path.
• Key deps: znippy-common (format, sinks, codec, slot pool), znippy-zoomies (Gatling ordered-sink engine), io-uring+libc (batched small-file reads), walkdir, blake3, crossbeam-channel, arrow 58.
• Shape: lib, ~1.4k LOC across slot_packer.rs (974), stream_packer.rs (443); a bench_stream example.
• Observed role in the workspace: Called by the CLI (compress_dir) and by znippy-iceberg's roundtrip test; the top-level znippy re-exports it via the CLI. Feeds compressed blobs + metadata batches into whichever ArchiveMetaSink is selected.

Fill in (one line each — only the human knows these)

• Primary consumer(s):
• Why it exists (vs the obvious alternative):
• Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
• Release cadence & independence (rides a parent? own repo someday?):
• Status (production / prototype / experimental / stubbed):
• Must-NOT-change invariant:
• Biggest known gap or TODO:
• Anything a newcomer would get wrong about it: