nordisk/znippy

The compress engine — reasoning

Class: reasoning. This file holds the why, the trade-offs, and every design we rejected. The descriptive shape we actually run is in compress-design.md, which by rule mentions none of this. If you want to know "why not X?" — X is here.

The Gatling flow

Why "Gatling" at all — the 1→N→1 rule

The whole engine exists to keep NVMe saturated while N cores compress. The shape that does it is 1 → N → 1:

• 1 reader, read only. It does no CPU work: splitting an input into slices is a cheap O(n_workers) candidate-offset division, not a compression pass. The instant the reader starts doing heavy work, it becomes the serial ceiling and every worker starves behind it.
• N workers carry one unit through ALL heavy stages. A single slice goes through blake3 and compress on the same worker, with no barrier, and workers self-dispatch the next slice.
• 1 writer, stitch/drain in order. The finalizer only orders and drains; it does no compression.

The law behind it: heavy work on either 1-end becomes the serial ceiling. Both 1-ends must stay trivial; all weight lives on the N-end. This is the single rule to remember.

The "blame Gatling" trap

There was a moment when a Gatling path ran slower than plain rayon, and the tempting conclusion was "Gatling is the wrong model, rayon is faster." That was wrong. The real cause was heavy work sitting on a 1-end — a stage that belonged on the N workers had crept onto the reader/writer, so the serial ceiling dropped. Moving the weight back to the N-end fixed it. The lesson is the 1→N→1 rule, not "rayon is faster." Never blame the model for a misplaced stage.

Rejected: rayon fork-join with a per-slot-completion BARRIER (the "lbzip2-v1 bug")

The obvious parallel shape is rayon: fan a batch of slices out, join, then release the batch. We rejected it because the join is a barrier: the slot cannot recycle until its slowest slice finishes, so one slow slice stalls the whole batch and the other cores sit idle waiting. This is the classic lbzip2-v1 failure. The fix is the no-barrier slot ring: one global slice queue, workers self-dispatch, and a slot returns (Returner) only after its own last slice — a slow slice in slot A never blocks slot B's slices. The barrier is exactly the thing the ring removes.

Rejected: copying Vec<u8> payloads across channels

The tidy dataflow is "worker compresses, sends the Vec<u8> payload down a channel to a single writer." We rejected it: on ~7 GB/s NVMe, that extra payload copy halves throughput — the memcpy bandwidth becomes the bottleneck before the disk does. Instead the channel carries slot indices only (law G-1), and each worker pwrites its own payload at an AtomicU64 cursor. Zero payload copies between compress and disk. The finalizer likewise gets ChunkMeta only, never bytes.

Rejected: per-file open/read syscalls

Reading each input file with its own open/read pair means a syscall storm on a tree of many small files — the reader (a 1-end) spends all its time in the kernel and starves the workers. We rejected it for the io_uring 128-files-per-batch submission: one batched submission amortizes the per-file syscall overhead away, keeping the reader trivial as the 1→N→1 rule demands. The non-io_uring fallback exists for portability and is held byte-identical by fallback_byte_identical_to_io_uring — the reader implementation must never change the bytes.

Why blake3 over the ORIGINAL bytes, ordered by chunk_seq

Integrity is hashed over the source bytes, not the compressed frame: a checksum must verify "is this the file the user gave us," independent of which codec or level produced the frame. And checksums are bound to chunk_seq order, not thread identity (law G-3) — so a 4-core run and a 64-core run of the same input produce the same checksums and the same index. If the hash depended on which worker happened to grab a slice, archives would be non-reproducible across machines. Determinism is the price of admission for content-addressing and dedup.

Why store-if-incompressible, and why no_skip exists

Compressing an already-compressed input (.jar/.gz/.crate/.png/.parquet) burns CPU to produce a frame that is larger than the input. So the skip path stores those raw, and — as a backstop for anything the extension list misses — any slice whose frame ends up ≥ its input is stored raw too, with the index compressed flag set false. no_skip exists for the cases where you want a guaranteed-uniform pipeline (benchmarks, forcing the codec, verifying the store path is never taken): it disables the extension skip and pushes every slice through the codec.