nordisk/znippy-zoomies

lbzip2 library readers — design

Class: design · Function: lbzip2-readers (the bzip2 library twins of the lbunzip2 CLI). Strict: shipped shape, every claim a real file:symbol. Run them via lbzip2-readers-guide.md; the whole-slice-vs-stream tradeoff in lbzip2-readers-reasoning.md. The CLI is lbunzip2-design.md.

What they are

Three callable (not-a-binary) bzip2 decoders in the lbzip2 crate, each fanning independent BWT blocks across every core through gatling_forkjoin::gatling_for_each. They differ only in how much input they hold and how they hand output back. All three front their decode with the same parallel block-magic scan — block_scan::find_all_blocks_parallel (lbzip2/src/block_scan.rs:143) — which splits the buffer into N chunks (each scanned with an 8-byte overlap so a boundary-straddling magic is not missed), fans the scan across cores via gatling_for_each, then merges + sorts + dedups by bit_offset to produce the identical boundary set the serial find_all_blocks (block_scan.rs:129) would — proven by parallel_scan_matches_serial_multiblock (block_scan.rs:353). The point: the whole-stream magic walk no longer runs as a 1-core serial prefix.

The three readers

1. parallel::decompress_parallel(data) -> Vec<u8> (lbzip2/src/parallel.rs:15)

Whole-slice, whole-output. Parses the BZh header, find_all_blocks_parallel, then groups blocks into n_segments = (n_threads * 4).min(n_blocks) contiguous segments (parallel.rs:61) — ~4 segments/thread so gatling_for_each's self-dispatch can balance uneven block sizes while amortizing per-block allocation. Each segment worker runs chunk::decode_segment_checked (lbzip2/src/chunk.rs:117), appending every block in its range into one growing buffer via decode_block_into (no per-block Vec). Output is assembled by concatenating only n_segments buffers (parallel.rs:114). Integrity: every block's stored CRC-32 is folded (rotate-left-1-then-XOR) into a combined stream CRC and checked against the stream trailer via block_scan::find_stream_crcs (block_scan.rs:82) for a single stream; concatenated (pbzip2) inputs rely on the per-block CRCs (parallel.rs:109).

2. reader::StreamingBz2Read<R: Read> (lbzip2/src/reader.rs:30)

Read-trait streaming for a Read source. Reads ~100 MB compressed chunks (CHUNK_BYTES, reader.rs:18), runs find_all_blocks_parallel per chunk (reader.rs:111), skips the last block when not at EOF (it may span into the next chunk), decodes the complete blocks across cores with gatling_for_each (reader.rs:143) calling block::decode_block, then drains the decoded region from comp_buf while keeping the undecoded tail. Reading the next chunk overlaps consuming the current decoded output; only one chunk's decoded output is live at a time.

3. reader::ParallelBz2Read<'a> (lbzip2/src/reader.rs:205)

Read-trait over an in-memory &[u8] (e.g. an mmap'd file). Scans all boundaries up front (find_all_blocks_parallel, reader.rs:237), then decodes in BATCH_SIZE = 64-block parallel batches (fill_batch, reader.rs:253; gatling_for_each at reader.rs:267), serving each batch through read().

Whereabouts

Saturation state

These readers are the natural consumers behind e.g. osm-katana's bz2 path. The scan-then-fan shape keeps cores fed; the honest ceiling matches lbunzip2 (bzip2's block independence is the whole win — see lbunzip2-reasoning.md). No cores_busy is fabricated here; real numbers land in the README.md mashup on the heavy-bench wave.