nordisk/znippy

The compress engine — design

Class: design · Function axis: f0 (the Gatling engine) + f1 reader · f2 slot pool · f3 workers · f4 writer/finalizer · f5 codec · f6 metadata sink. Strict: this file states only the shape we run. Every why, every alternative, lives in compress-reasoning.md; the how-to is in compress-guide.md.

f0 — the Gatling engine (the whole)

znippy is a content-addressed, seekable compressor: an Arrow-IPC index over zstd/lz4 chunks, with blob dedup. Compress is the "Gatling no-barrier" engine underneath it. Every input is split into slices — a slice is the single unit of both blake3 integrity and parallel work. There is one global slice queue across all in-flight slots; there is no per-core barrier, so a slow slice never blocks another slot's slices.

Two entry points feed one engine — a directory path and an in-memory stream — and both run the identical reader → pool → workers → writer/finalizer shape:

The non-io_uring fallback reader path is byte-identical to the io_uring path — asserted by fallback_byte_identical_to_io_uring (znippy-compress/src/slot_packer.rs:817).

Four laws hold the engine together:

f1 — the reader (1-end, read only)

The io_uring reader submits 128 files per batch and fills ChunkRevolver coalescing slots; it does no CPU work — file split is a cheap O(n_workers) candidate-offset division, not compression. Big files (> one slice) are sliced across slots; small files pack into a single slot. file_split_block_size = 10 MB: a file larger than that spans several index rows, one per chunk, all sharing relative_path and ordered by chunk_seq.

f2 — the slot pool (the Gatling magazine)

The pool itself is upstream: gatling::revolver in coalescing mode, reached as znippy_zoomies::chunk_revolver. znippy owned a fork of it (slotpool.rs::Magazine) until 2026-08-25 and no longer does — the canonical implementation beat the replica in 5 of 5 interleaved runs by 5-15 % with 7.9x fewer allocations, so the collapse cost nothing and removed a second writer.

What stays znippy's is the cut policy, PoolPlan (znippy-common/src/poolplan.rs): slice_size is the big/small partition threshold and the big-file cut length, so it lands in chunk_seq, fdata_offset and every chunk checksum. slot_size and num_slots are only how much buffer is in flight and are planned against the cgroup budget, which is why a memory-shrunk run is byte-identical to a fat one.

Two behaviours differ from the fork and both are safer. A Filler dropped un-published returns its slot (the fork leaked it), and claim_slot blocks for the pool's whole life instead of reporting shutdown by sender-drop. The first one bites a drain loop that does not hold what it reclaims — see slot_packer.rs, where the mechanical translation of that loop was a use-after-free that every unit test passed through.

f3 — the workers (N-end, carry a slice through ALL heavy stages)

N workers self-dispatch from the one global slice queue. Per slice a worker: blake3s the ORIGINAL bytes (integrity is over the source, not the frame), then compress_into a reused buffer (or stores raw), then pwrites its own payload at an AtomicU64 cursor. No barrier: a worker that finishes a slice grabs the next, regardless of which slot it belongs to.

f4 — the writer / finalizer (1-end, stitch/drain in order)

Workers write payload themselves; the one finalizer builds the Arrow index from ChunkMeta only — there is no payload channel to the finalizer (G-4). It orders rows by chunk_seq (G-3) and hands the metadata tail to the sink (f6).

f5 — the codec (store-if-incompressible)

Codec::compress_into(&mut self, input, out) (znippy-common/src/codec.rs:42) runs OpenZL zstd/lz4 with per-worker buffer reuse and a long-lived ctx; decompress_into (:66) mirrors it. Two store-raw rules set the index compressed flag to false:

1. skip path — resolved by znippy-common/src/precompressed.rs (a re-export of znippy-zoomies/ldeflate/src/precompressed.rs, which owns the probe; only the extension leg is znippy's) in a fixed order: an explicit caller ContentHint, then the extension table (.jar / .gz / .crate / .png / .parquet / .pack / .idx), then a magic-byte probe over the first 16 bytes (zlib, PACK, gzip, zstd, xz, bzip2, zip, png, jpeg …). Stored raw without a compress attempt. The probe is the only tier that can reach an entry with no extension, which is what a git loose object is.
2. any slice whose produced frame is ≥ its input is stored raw.

no_skip forces compression of every slice (disables rule 1); it is exactly SkipPolicy::compress_everything(). compress_dir_with_policy / compress_stream_with_policy take the full SkipPolicy, so a caller that already knows its input is compressed says so once for the batch rather than being sniffed.

Rule 1 is a CPU optimisation, not a size one — rule 2 already guaranteed the bytes. Measured 2026-08-03 on oden, 5388 real git loose objects (35.8 MB) and a real 22.4 MB packfile: compress CPU 3.70 s → 0.30 s and 4.06 s → 0.21 s (92% / 95% saved). The archive grows 0.17% / 1.2%, because rule 2 had been recovering that much from already-deflated data at full codec cost; --no-skip buys it back.

f6 — the metadata sink

trait ArchiveMetaSink (znippy-common/src/meta_sink.rs:43); default impl ArrowIpcSink (:73) writes sub-indexes + a lookup + an fst trie + a manifest + the ZNPYMIDX footer. The sink is chosen by a MetaSinkFactory so the container backend (inline Arrow IPC, or an Iceberg warehouse) is swappable without znippy-compress taking the heavy async dependency.

f7 — the tree's shape (__znippy_fsmeta__)

The data index is a table of chunks, and a chunk belongs to a regular file. Two facts of a real tree therefore had no row shape at all and were dropped:

• an empty directory owns no file, so it owns no chunk. Extract only ran create_dir_all on a file's parent, so a directory containing nothing did not come back.
• a unix mode. Nothing recorded metadata().mode() and nothing restored it, so an extracted file got 0666 & ~umask — a 0755 script came back 0664.

Both were silent: verify green, every file byte-exact, tree still wrong.

Fixed with one reserved section, ZNIPPY_FSMETA_MODULE (znippy-common/src/fsmeta.rs), holding Arrow IPC (relative_path, kind, mode). Reserved means is_reserved_module already makes every data reader skip it, so it is additive both ways — no ZNIPPY_FORMAT_VERSION bump, no new column for the iceberg/append/gunnar writers, and an older znippy opens the archive unchanged. A mode column on the base schema would have been all three of those; a zero-byte data row per directory would make list/get/DuckDB see a file where a directory is.

Rows are one per directory (empty or not), plus one archive-scoped FileDefaultMode row carrying the tree's modal file mode and one row per file that deviates. The first cut compared against a hardcoded 0o644 and its own test caught it: that is 0o666 & ~umask for umask 022, and on a umask 002 box every plain file "deviated". A row per file is exact but pays a second copy of every path the lookup sub-index already holds — ~4 MB on a 50 000-blob mirror to say one thing 50 000 times. The modal row is exact, umask-independent, and O(distinct modes).

Restore runs in three phases and each ordering is forced: directories first (before extraction — nothing else will ever create an empty one), file modes after extraction (a 0444 file must be written before it is sealed), directory modes last, deepest first (a 0555 directory cannot have children created inside it). Applied only on a full extract; a filtered extract restores a subset and would be misrepresented by the whole tree's directory set.

f8 — a container-registry mirror, end to end

The use case the shape section and --format generic were added for: pack what talosctl image cache-create writes into ONE archive and restore it byte-exactly on a fresh host.

<root>/blob/<sha256-hex>                          1 KB … 500 MB, ALREADY gzip/zstd, NO extension
<root>/manifests/<host>/<repo>/reference/<tag>    small JSON
<root>/manifests/<host>/<repo>/digest/<sha256>    small JSON

znippy compress --format generic -i /var/cache/mirror -o mirror
znippy decompress -i mirror.znippy -o /var/cache/mirror   # byte-exact, dirs and modes included

--format generic (aliases none/raw/plain/blob, type_id 26) is the handler that claims nothing: no extensions, no columns, matches_path always false. The default rust also extracts nothing from such a tree — matches_path is ends_with(".crate") — but it records pkg_type = 1, so --type rust and every RustView consumer would report a registry mirror as crates.io content. A pkg_type is a promise; the honest one here is none.

The blob filenames are bare hex, so the extension table (f5 rule 1) cannot reach one and the magic-byte probe is the only leg that can. MEASURED 2026-08-24, 3975WX, 1.2 GB tree (12 × 100 MB gzip blobs + 2 000 manifests, output to NVMe):

80× the wall clock and 750× the CPU for a byte-identical archive. Blobs are content-addressed, so this is also the path where a single wrong byte breaks an image pull with a digest mismatch and no other symptom — diff -r over the restored 1.2 GB tree reports nothing, empty directories included.