nordisk/znippy-zoomies

ljar — design

Class: design · Function: ljar (parallel JAR decode). Strict: the shape we ship, every claim carrying a real file:symbol. How to run it in ljar-guide.md; why the balanced fan-out earns its keep in ljar-reasoning.md. The family view is decompressors-design.md; the engine is gatling-design.md. ljar's ZIP superset twin is lzip-parallel-design.md — near-verbatim (JAR ⊂ ZIP).

What it is

A pure-Rust parallel decoder for the stock .jar (a ZIP subset), decoded across every core. The CLI is ljar/src/bin/ljar.rs; the library twin is ljar::decompress_jar (ljar/src/lib.rs:65) with the _filter (ljar/src/lib.rs:83) and _stream (ljar/src/lib.rs:98) siblings, all returning Vec<JarEntry>.

A JAR has no interior back-reference chain across entries: every entry is its own DEFLATE (or STORE) stream with its own CRC-32. So the parallelism is per-entry, and the only serial cost is the one-time Central Directory parse at the file tail.

Path 1 — in-memory small file

parallel::decompress_parallel / _filter (ljar/src/parallel.rs:17 / :25). Reads the Central Directory once (central_dir::read_central_directory), filters out directories and (optionally) non-needle entries before any inflate, then fans the matched entries through gatling::gatling_forkjoin::gatling_for_each_balanced (call site ljar/src/parallel.rs:32) — args (matched.len(), 0, 1, weight=|i| matched[i].uncompressed_size, f=decode). Results come back in Central Directory order. decompress_jar picks this path below SMALL_FILE_THRESHOLD = 10 MB (ljar/src/lib.rs:48).

Path 2 — streaming large file

reader::StreamingJarRead (ljar/src/reader.rs:38). new / with_filter seek to the file tail, parse the Central Directory, filter, then sort entries by local_header_offset (ljar/src/reader.rs:55) so disk reads run sequentially. Decoding proceeds in batches of BATCH_SIZE = 64 (ljar/src/reader.rs:20): fill_batch (ljar/src/reader.rs:84) reads each entry's compressed bytes sequentially (seek to local header, skip name + extra fields, read compressed_size bytes), then parallel-decodes the batch via gatling_for_each_balanced (call site ljar/src/reader.rs:126) weighted by raw[i].3 (the uncompressed size). It implements Iterator<Item = Result<JarEntry, JarError>>, yielding in file-offset order. Only one batch of decompressed output is live at a time → peak memory is bounded regardless of JAR size; the file is never fully loaded.

The saturation fix — LPT balanced fan-out

Weighting each entry by uncompressed_size makes the schedule heaviest-first (LPT): a skewed archive — one fat entry among many tiny ones — can't tail-choke a single core, because the monster is claimed first and the cheap tail backfills the idle workers. The rebalance only changes which worker claims which entry, never the decoded bytes or the output ordering. Proven by streaming_skewed_roundtrip_byte_identical (ljar/src/reader.rs), which asserts byte-identical output for a deliberately skewed JAR.

Correctness

Decode is entry::decompress_entry_raw(compressed, method, expected_size, expected_crc) (ljar/src/entry.rs:111): method 8 → linflate DEFLATE, method 0 → STORE copied verbatim, anything else → error — then every entry's output is CRC-32 verified. A mismatch fails that entry; its siblings keep running.

CLI extract path

ljar/src/bin/ljar.rs extracts each entry to its own output file via gatling::parwrite::extract_entries_unordered (gatling/src/parwrite.rs:85): each worker preads its entry, decodes with linflate, and writes its own file — no ordered collector on the critical path, since independent files need no global order.

Whereabouts

Saturation state

Every parallel hot path fans per-entry DEFLATE across all cores through the one shared engine; the LPT weight keeps a skewed archive from stranding cores on the tail. Honest cores_busy numbers land in the README.md A/B/C mashup on the next heavy-bench wave — this doc records no fabricated figures.