nordisk/znippy-zoomies

lzip-parallel — design

Class: design · Function: lzip-parallel (parallel ZIP decode). Strict: the shape we ship, every claim carrying a real file:symbol. How to run it in lzip-parallel-guide.md; why the balanced fan-out earns its keep in lzip-parallel-reasoning.md. The family view is decompressors-design.md; the engine is gatling-design.md. This is the near-verbatim twin of ljar-design.md — JAR ⊂ ZIP, so ljar is the same pipeline over the subset container.

What it is

A pure-Rust parallel decoder for the stock .zip container (wheels, .jar, plain .zip), decoded across every core. The CLI is lzip-parallel/src/bin/lzip.rs; the library twin is lzip_parallel::decompress_zip (lzip-parallel/src/lib.rs:58) with the _filter (:76), _filter_set (:94), _filter_suffixes (:111) and _stream (:126) family, all returning Vec<ZipEntry>.

Every ZIP entry is its own DEFLATE (or STORE) stream with its own CRC-32, indexed by a Central Directory at the file tail — so the parallelism is per-entry, and the only serial cost is the one-time Central Directory parse.

Path 1 — in-memory small file

parallel::decompress_parallel / _filter (lzip-parallel/src/parallel.rs:17 / :25), plus the _filter_set (:126) and _filter_suffixes (:157) variants for wheels. Each reads the Central Directory once (central_dir::read_central_directory), filters out directories and non-matching names before any inflate, then fans the matched entries through gatling::gatling_forkjoin::gatling_for_each_balanced (call sites lzip-parallel/src/parallel.rs:32, :138, :167) — weighted by uncompressed_size. Results come back in Central Directory order. decompress_zip picks this path below SMALL_FILE_THRESHOLD = 10 MB (lzip-parallel/src/lib.rs:41).

Path 2 — streaming large file

reader::StreamingZipRead (lzip-parallel/src/reader.rs:38) with new / with_filter / with_filter_set / with_filter_suffixes. Each seeks to the tail, parses the Central Directory, filters, then sorts entries by local_header_offset (lzip-parallel/src/reader.rs:55) for sequential disk reads. Decoding runs in batches of BATCH_SIZE = 64 (:20): fill_batch (lzip-parallel/src/reader.rs:122) reads each entry's compressed bytes sequentially (seek to local header, skip name + extra, read compressed_size bytes), then parallel-decodes via gatling_for_each_balanced (call site :175) weighted by uncompressed_size. It implements Iterator<Item = Result<ZipEntry, ZipError>>, yielding in file-offset order. Only one batch of decompressed output is live at a time → bounded peak memory; the file is never fully loaded. One micro-opt: the compressed read buffer is uninitialised (Vec::with_capacity + set_len, lzip-parallel/src/reader.rs:154-160) because read_exact fully overwrites it — the zero-fill was wasted work on the hot path.

The saturation fix — LPT balanced fan-out

Weighting each entry by uncompressed_size makes the schedule heaviest-first (LPT): a skewed archive — one fat entry among many tiny ones — can't tail-choke a single core, because the monster is claimed first and the cheap tail backfills the idle workers. The rebalance changes only which worker claims which entry, never the bytes or the ordering. Proven by streaming_skewed_roundtrip_byte_identical (lzip-parallel/src/reader.rs).

Correctness

Decode is entry::decompress_entry_raw(compressed, method, expected_size) (lzip-parallel/src/entry.rs:93): method 8 → linflate DEFLATE, method 0 → STORE copied verbatim — CRC-32 verified. A mismatch fails that entry; its siblings keep running.

CLI extract path

lzip-parallel/src/bin/lzip.rs extracts each entry to its own output file via gatling::parwrite::extract_entries_unordered (gatling/src/parwrite.rs:85): each worker preads its entry, decodes with linflate, writes its own file — no ordered collector, since independent output files need no global order.

Whereabouts

Saturation state

Every parallel hot path fans per-entry DEFLATE across all cores through the one shared engine; the LPT weight keeps a skewed archive from stranding cores on the tail. Honest cores_busy numbers land in the README.md A/B/C mashup on the next heavy-bench wave — this doc records no fabricated figures.