nordisk/znippy-zoomies

lgz — design

Class: design · Function: lgz (multi-member gzip decode). Strict: the shape we ship, every claim carrying a real file:symbol. How to run it in lgz-guide.md; why gzip is only partly saturable in lgz-reasoning.md. The family view is decompressors-design.md; the engine is gatling-design.md.

What it is

A pure-Rust parallel decoder for the stock .gz format — anything gzip/pigz wrote, decoded across every core. The CLI is lgz/src/bin/lgz.rs; the library twin is lgz::decompress_tar_gz_filter (lgz/src/lib.rs:63) / _capped (lgz/src/lib.rs:76) and the in-memory decompress_gz (lgz/src/parallel.rs:108).

gzip has two kinds of parallelism, and lgz picks the one the input actually offers, per invocation:

Path 1 — concatenated members (the fast, fully-parallel path)

try_multimember_fastpath (lgz/src/bin/lgz.rs:200). The common .gz from log rotation, cat *.gz, pigz --independent, or a member-per-file .tar.gz is a run of fully independent gzip members. lgz mmaps the file and scans for every member start with lgz::speculative::find_gzip_members_parallel (lgz/src/speculative.rs:471) — itself a gatling_for_each fan-out over the box (header-shape + inflate probe per candidate). < 2 members ⇒ not this path.

Each member's uncompressed length is its own 4-byte ISIZE trailer, so the exact output byte-layout is known without decoding. From there the two output shapes diverge:

• -o file → pre-size the output (set_len(total)), then gatling::parwrite::write_segments_positional (gatling/src/parwrite.rs:191) fans the members across the pool: each worker decodes one member into a reused scratch buffer and immediately pwrites it at its prefix-sum offset. No collector, no serial concat — in-flight memory ≈ workers × member_size, never O(output). The decode validates its length against ISIZE and fails the segment on mismatch.
• stdout / pipe (cannot seek) → gatling::gatling::ordered::run_ordered_sink (gatling/src/gatling_ordered.rs:113) streams members to a worker pool; the ordered sink writes each the instant it is next-in-order while the other workers keep decoding. A Mutex<Vec<Vec<u8>>> recycled buffer pool (lgz.rs:350) hands each drained, warm, ISIZE-presized buffer back to the next member — zero steady-state allocation and no per-member page-fault storm (the old Vec::with_capacity+drop churned 130 K+ minor faults on the kernel mmap_lock, capping saturation at ~8.5/12).

Path 2 — single member with flush boundaries (streaming)

main's streaming pipeline (lgz/src/bin/lgz.rs:420). A single-member stream can still be split iff the encoder inserted Z_FULL_FLUSH points (pigz does). A single dispatch thread is the producer: it reads 200 MB chunks (CHUNK_SIZE), prepends the undecoded carry into headroom, strips the gzip header (first chunk) and 8-byte trailer (last chunk), and calls split_chunk (lgz/src/chunk.rs:29) to cut at flush boundaries. It distinguishes the two flush kinds by the bytes after the boundary (lgz.rs:562):

• full-flush (00 00 00 FF FF — empty stored block resets the LZ77 window) ⇒ the segments are independent, queued one at a time to run_ordered_sink; N map workers run decode_segment_into_hint (lgz/src/chunk.rs:79); the WriteSink (lgz.rs:62) writes in strict stream order.
• sync-flush (back-references cross the boundary) ⇒ segments are not independent, so the whole chunk decodes as one segment (still overlapping I/O with the producer).

Path 3 — no boundary at all (serial fallback, honest)

A plain single-member gzip (gzip foo) has no interior split point — one continuous LZ77 window. When the producer finds no boundary in the whole file it sets fallback_flag and main (lgz.rs:666) tries, in order: decode_concatenated_members → speculative_decode (DEFLATE block-boundary guess, trailer-verified before a byte is emitted, lgz.rs:161) → single-threaded flate2. This path is serial-by-format, not a saturation bug (see lgz-reasoning.md).

Correctness

Every emitted byte is checksum-verified: the multi-member -o path checks each member's decoded length against ISIZE; the in-memory decompress_gz re-checks the whole-output CRC32 + ISIZE (lgz/src/parallel.rs:131); the speculative path is discarded unless gzip_trailer_ok passes.

Whereabouts

Saturation state (just-landed)

Multi-member -o decode reaches ~7.8/12 cores on the quiet box (stdout can't be offset-written, so it uses the ordered-sink path); single-member gzip is serial-by-format and exempt. Honest bench numbers land in the README.md A/B/C mashup on the next heavy-bench wave — this doc records no fabricated cores_busy.