nordisk/znippy-zoomies

lgz — reasoning

Class: reasoning · Function: lgz. Why the engine is shaped this way and where gzip is honestly not saturable. The implemented shape is in lgz-design.md.

Why gzip is only partly a parallel problem

DEFLATE is a single LZ77 back-reference window that flows the whole stream. A byte in the middle can reference any of the previous 32 KiB, which itself references earlier bytes, transitively back to byte zero. So a plain single-member gzip (gzip foo) has no interior point you can start decoding from without first decoding everything before it. That is a property of the format, not a defect in lgz. The honest verdict: single-member gzip decode is serial-by-format and lgz's Path 3 fallback (lgz/src/bin/lgz.rs:666) accepts it — one core, flate2, correct output.

Two escape hatches exist, and lgz takes both when the input offers them:

1. Concatenated members. cat a.gz b.gz, log rotation, pigz --independent, and member-per-file .tar.gz produce independent members — each its own window, each its own CRC/ISIZE trailer. That is genuinely embarrassingly parallel, and it is the common shape in practice, so it is Path 1 (the fast path).
2. Full-flush points. pigz inserts Z_FULL_FLUSH markers that emit an empty stored block, resetting the window. Segments between them are independent. lgz detects full-flush vs sync-flush by the post-boundary bytes (lgz.rs:562) and only splits on full-flush — splitting a sync-flush stream would decode garbage because the back-references cross the boundary.

Why the writer had to change (the ~8.5 → ~11.5 story)

The middle "decode" move was already parallel; what pinned lgz below its rivals was the tail. The old multi-member path built a Vec<Vec<u8>> of every decoded member (O(output) resident — a peak-RSS regression) and drained it through one serial writer. Two fixes landed:

• -o files get write_segments_positional (gatling/src/parwrite.rs:191): each worker pwrites its own member at a prefix-sum offset into a pre-sized file. No collector on the critical path. This is why -o beats stdout.
• stdout can't be offset-written, so it uses run_ordered_sink plus a recycled warm-buffer pool. The subtle win was not the ordering — it was killing the per-member Vec::with_capacity→drop→munmap churn that serialised every worker on the kernel mmap_lock and capped saturation at ~8.5/12 regardless of thread count (lgz.rs:340). Recycling ISIZE-presized buffers gives a warm, already-faulted-in buffer per member, so after warm-up there are no output allocations in the hot loop.

Result: -o multi-member ~7.8/12 (offset-pwrite); stdout held via the ordered path. The remaining gap to a pure pwrite path is stdout's inability to seek — an OS constraint, not a scheduling one.

Why not a bespoke thread pool

Earlier lgz carried its own scoped-thread clone. The constellation's no-rayon law (tests/rayon_free_law.rs) folds every codec onto the one shared gatling engine, so the ordering, backpressure, and permit-bounded memory are proven once and reused. See gatling-reasoning.md.

Honest numbers

No cores_busy is hard-coded here. The real figures land in the README.md A/B/C mashup on the quiet-box heavy-bench wave.