nordisk/znippy

znippy — reasoning (f0, why this overall shape)

Class: reasoning. This file holds the why, the trade-offs, and every design we rejected at the whole-system level. The descriptive shape we actually run is in design.md, which by rule mentions none of this. Per-function reasoning lives in compress-reasoning.md, index-reasoning.md, and zoomies-reasoning.md; the deep engine narrative in the classified chapters (gatling, inflate, benching).

Why znippy exists at all

Every existing archiver forces a choice znippy refuses to make. tar+zstd compresses all-core but decompresses serial and can't pull one file without streaming the whole thing. zip gives random single-file reads but is single-threaded on both ends. 7z solid archives trade random access for ratio. parquet is queryable and columnar but isn't a file archive. znippy's whole reason to exist is to be all-core on both ends, random-access per file, AND directly queryable — without pretending to beat zstd -19 on raw ratio. Ratio was deliberately not the goal; every design call below protects the three properties that were.

Why the index is Arrow IPC (and nothing bespoke)

The "query it with no znippy library" property is only real if the index is a format outside tools already speak. Rejected: a bespoke on-disk B-tree / custom index format — it would be faster to seek but would make DuckDB/Polars/DataFusion useless against the archive, killing the third property. Arrow IPC is the one layout that is simultaneously a valid seekable index and a table an analytics engine reads directly. That single decision is why the lookup sub-index is a sorted Arrow stream rather than a private structure — see index-reasoning.md for why we also carry an fst trie beside it.

Why the four laws

The laws are not style; each buys a measured property:

• L1 zero-copy channels. On NVMe at ~7 GB/s, copying a Vec<u8> payload between the reader and a compressor thread halves throughput. Rejected: sending decoded payloads over channels — we send slot indices into a pre-allocated ring instead, so bytes never move. The engine that embodies this (Gatling) has its own rejected-alternatives record in compress-reasoning.md and gatling.md.
• L2 no hot-path allocation. A fresh Vec per chunk is allocator churn that shows up as variance in the wall time (the candidate-offset output model measured 7.12 s → 6.70 s once churn was removed — inflate.md). Workers reuse buffers; the codec context is long-lived.
• L3 checksums bound to chunk_seq, not thread identity. Rejected: hashing in completion order — it makes the archive's checksums depend on how many cores ran, so no other machine could reproduce them. Binding to chunk_seq makes integrity a property of the data, not the run, which is also what lets f7 sign a stable merkle root.
• L4 concurrent writer. Rejected: compressors that block on the index writer — the seal is serial by nature, so any wait on it caps throughput at the writer's speed. The finalizer runs on its own thread from ChunkMeta only.

Why one Gatling engine for both entry points

Compress-from-directory and compress-from-memory could each have a bespoke pipeline. Rejected: two engines — they would drift, and the no-barrier invariants (the hard part) would have to be proven twice. slot_packer and stream_packer share the same worker/writer/finalizer so the saturation guarantees are written once. The same engine also powers osm-katana's xml→pbf path, which is what made it worth extracting into znippy-zoomies.

Why plugins instead of a fixed schema

An airgap mirror wants the ecosystem view (maven coordinates, python wheel tags) next to the bytes. Rejected: baking every ecosystem's columns into one schema — it would bloat every archive and couple the format to a fixed set of package types. Instead a plugin contributes only its own Arrow columns and the writer unions them; the CLI autodiscovers via meta() so there is no hand-maintained name table. Rejected: a central handler registry for the same reason.

Why "store if incompressible" is unconditional

Already-compressed inputs (.jar, .gz, .crate, .png, .parquet) are stored verbatim on an extension skip-list — obvious. Less obvious: any slice where the produced frame is ≥ the input is also stored raw, compressed = false. Rejected: trusting the extension list alone — it would still pay codec time (and lose bytes) on incompressible content with an innocent extension. The unconditional size check makes "never bigger than the input, per slice" a guarantee.

The size half of that objection was always covered; the CPU half was not, and on git data it was nearly the whole cost. A caller ContentHint and a magic-byte probe now close it (znippy-common/src/precompressed.rs) — 92–95% of compress CPU on real loose objects and a real packfile, measured. Rejected: an entropy estimate as a general fallback — a tunable threshold that costs a scan and misjudges in both directions, whose wrong answers show up as neither a failure nor a red test. Every probe is an exact pattern at a fixed offset instead.

Why a dynamic store and a static seal (f6)

A live mirror needs to accept new commits and survive restarts; a distributed artifact needs to be one immutable, content-addressed file with no warehouse dependency. Rejected: only a static format (can't resume across a reboot) and rejected: only a dynamic Iceberg store (every reader needs the warehouse). Carrying both — IcebergSink for the dynamic phase, seal() to freeze it — lets one workflow be resumable while it's being built and immutable once shipped. Seal reuses the blobs verbatim and rewrites only the metadata tail, byte-identical to a fresh compress, so a sealed archive is indistinguishable from a one-shot one.

Why signing is a merkle-fold, not a re-hash

Provenance (f7) had to be free on top of the integrity we already compute. Rejected: re-reading content to sign it — every chunk already carries a blake3 hash, so the archive root is a merkle-fold of hashes already in hand, no content re-read. It rides reserved manifest sections, so an unsigned archive is byte-unchanged and the blob/compress path is untouched — signing is strictly additive, never a second write path.

Why nornir bench, not criterion

The whole benchmark story is a deliberate fork away from cargo bench/criterion — single-shot real-workload numbers historized by git SHA and gated for >10% regression, not statistical micro-sampling. The full argument (dev-deps can't live in a bin; criterion's output isn't machine-historizable the way the release gate needs) is its own chapter: benching.md.