lbzip2 — crate card
Reverse-engineered (from source — verify, don't trust the name)
- One-line pitch: A pure-Rust parallel bzip2 decompressor that scans block boundaries and decodes Burrows-Wheeler blocks across many cores.
- Does: One reader thread fills a ChunkRevolver ring buffer (~100 MB slots) with raw compressed bytes; a cheap bit-scan locates bzip2 block magic (0x314159265359 at arbitrary bit offsets) and dispatches sub-ranges to a worker pool; each worker independently decodes its BWT blocks and output is reassembled in order and streamed as
Read. Modules cover the full bzip2 pipeline:bitreader,block/block_scan,bwt,huffman,mtf,chunk,reader,stream, pluspar/parallel(rayon-free scoped-thread work-stealing map). ExposesBLOCK_MAGIC/FINAL_MAGICconstants. - Key deps:
memchronly (runtime). Deliberately rayon-free — parallelism is its ownpar::par_map, worker count from physical cores orLBZIP2_THREADS. Dev:bzip2(reference/round-trip). - Shape: lib + bin
lbunzip2(471 LoC); ~2.9k LoC. Featuretiming. Benches under benches/ are registered as[[test]]--ignoredprofilers (autobenches off). - Observed role in the workspace: Bzip2 decode backend for
osm-katana(handles bzip2-compressed OSM XML). Version codename "Sleipner".
Fill in (one line each — only the human knows these)
- Primary consumer(s):
- Why it exists (vs the obvious alternative):
- Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
- Release cadence & independence (rides a parent? own repo someday?):
- Status (production / prototype / experimental / stubbed):
- Must-NOT-change invariant:
- Biggest known gap or TODO:
- Anything a newcomer would get wrong about it: