nordisk/znippy-zoomies

Decompressors — design

Class: design · Function: decompressors (the parallel decode family). Strict: chosen shape only. The bake-off numbers are in benchmarks.md; how to run them in decompressors-guide.md; the "why not keep four hand-rolled pipelines" road-not-taken in reasoning.md and chunk_revolver-unsorted.md.

Per-decompressor deep-dives (exact dataflow + skade-style SVG each)

Each decoder is documented on its own, with a hand-authored flow diagram of its exact dataflow (input → boundary scan → self-dispatch workers → ordered/positional sink → output):

• lgz — lgz-design.md · guide · reasoning
• lbunzip2 (CLI) — lbunzip2-design.md · guide · reasoning
• lbzip2 library readers — lbzip2-readers-design.md · guide · reasoning
• ljar — ljar-design.md · guide · reasoning
• lzip-parallel — lzip-parallel-design.md · guide · reasoning
• shared gatling engines — gatling-design.md; scanners (json/xml/vtd) — scanners-design.md; osm-katana convert — osm-katana-design.md.

What it is

Five workspace member crates (Cargo.toml:17), each a pure-Rust parallel decompressor for a stock archive format — the format every existing tool already reads, decoded across every core where the legacy serial tools leave the box idle. These are the subjects of the bake-off.

The shared decode shape

Each format is a set of independently decodable segments (gzip members, bzip2 blocks, ZIP/JAR entries). The decode is the same three-move gatling pipeline:

1. Scan the compressed bytes for segment boundaries (deflate_scan.rs / block_scan.rs / central_dir.rs) — cheaply, up front.
2. Fan the segments across N workers, each decoding one segment to completion (the linflate fast loop for DEFLATE formats; the BWT/MTF/Huffman path for bzip2).
3. Collect the outputs in order and write them out.

linflate is the reused DEFLATE core underneath lgz, ljar, and lzip-parallel: inflate_segment / inflate_segment_with_prefix / inflate_to_vec (linflate/src/lib.rs) decode one segment given its (optional) back-reference prefix window.

Public library surfaces

Each crate ships a library twin of its CLI (so a decode is callable, not just a binary):

• lgz::decompress_tar_gz_filter(data, needle) / _capped(…) — decode + filter members.
• ljar::decompress_jar(data) / decompress_jar_filter(data, needle) / decompress_jar_stream(reader) → Vec<JarEntry>.
• lzip-parallel::decompress_zip(data) and the _filter / _filter_set / _filter_suffixes / _stream family → Vec<ZipEntry>.
• linflate::inflate_into / inflate_segment* / inflate_to_vec — the raw DEFLATE core.

All five also expose the crate-local functional_status(component, check, ok, detail) introspection shim (mirrors lib.rs:35), so nornir test --features testmatrix sees each decode.

Parallelism substrate (the no-rayon law, workspace-wide)

All five crates are now rayon-free and fan out through the one shared engine:

• lgz, lbzip2, ljar, lzip-parallel route their per-segment decode across gatling::gatling_forkjoin::gatling_for_each (e.g. ljar/src/reader.rs:122, lzip-parallel/src/reader.rs:172); linflate is a pure single-buffer decode library with no fan-out of its own.
• ljar and lzip-parallel were ported off their old OnceLock<rayon::ThreadPool> singletons (rayon deps dropped); lgz/lbzip2 folded their private scoped-thread clones onto the shared engine. tests/rayon_free_law.rs now walks every workspace member and bans private pools, so no crate can quietly reintroduce one. The history of how the codecs were rayon-from-birth and how the law got fixed is in reasoning.md and gatling-reasoning.md.

The writer tail (landed). Each format's decode (the middle "Fan" move) is already parallel; what left the DEFLATE codecs writer-bound (lgz ~3.1, ljar ~3.5, lzip ~4.0 of 12 cores, vs lbz2 ~10.4) was the single ordered collector + writer at the tail — not slot starvation (widening ring_slots is refuted). The fix, now on master, is the gatling::parwrite layer (gatling/src/parwrite.rs): each worker preads and writes its own output — independent files (extract_entries_unordered, ZIP/JAR) or offset-pwrited regions of one pre-sized stream (write_segments_positional / write_slices_positional). Measured cores_busy: ljar/lzip → ~11.5/12, lgz → 7.8/12 (offset-pwrite on -o files; stdout can't be offset-written), lbz2 held ~10.4. See gatling-design.md and gatling-reasoning.md.