lzip-parallel — reasoning
Class:
reasoning· Function:lzip-parallel. Why the engine is shaped this way and where the real saturation risk lives. The implemented shape is inlzip-parallel-design.md. Reads almost identically toljar-reasoning.md— JAR ⊂ ZIP, one pipeline.
Why ZIP is a genuinely parallel problem
A ZIP is a set of independent members: each entry is its own DEFLATE (or STORE) stream with its own CRC-32, indexed by a Central Directory at the file tail. There is no cross-entry back-reference to decode through — unlike gzip's single continuous LZ77 window. So after one cheap serial step (parse the Central Directory) every entry decodes on its own core. That is embarrassingly parallel, and it is the common shape: a Python wheel or a fat .jar is hundreds or thousands of small entries.
Why the balanced fan-out, not a plain one
The trap is skew. Real archives are lopsided — one giant compiled artifact or vendored blob among a long tail of tiny source files. A plain fan-out hands each worker the next index; whichever worker grabs the monster runs alone long after its siblings have drained the tail — one busy core, the rest idle, exactly the core-saturation bug we refuse to ship. Weighting each unit by uncompressed_size makes the gatling schedule heaviest-first (LPT): the monster is claimed first and the cheap tail backfills the workers that finish early. It is a scheduling change only — streaming_skewed_roundtrip_byte_identical (lzip-parallel/src/reader.rs) proves the decoded bytes and the output ordering are untouched.
Why streaming sorts by offset, batches, and skips the zero-fill
The streaming reader (lzip-parallel/src/reader.rs:38) sorts entries by local_header_offset so the compressed reads walk the file forward — sequential I/O, not a seek storm — and decodes in batches of 64. Batching bounds memory: one batch of output is resident at a time, so peak RSS is independent of ZIP size. The compressed read buffer is uninitialised (Vec::with_capacity + set_len, reader.rs:154-160): read_exact fully overwrites it before any byte is read, so the vec![0u8; len] zero-fill was pure waste on the hot path — a small but honest win on the sequential read loop.
Why not a bespoke thread pool
Earlier lzip-parallel carried its own rayon ThreadPool (LZIP_THREADS); it is gone. The constellation's no-rayon law (tests/rayon_free_law.rs) folds every codec onto the one shared gatling engine, so ordering, self-dispatch, and index-ordered collection are proven once and reused (gatling/src/gatling_forkjoin.rs:209). See gatling-reasoning.md. (The parallel.rs module docstring still says "rayon thread pool" — that comment is stale; the code calls gatling.)
Honest numbers
No cores_busy is hard-coded here. The real figures land in the README.md A/B/C mashup on the quiet-box heavy-bench wave.