nordisk/znippy-zoomies

lgz — Parallel Gzip Decompressor

"Allfadern ser varje bit"

Production-ready parallel gzip decompressor in Rust. Uses linflate (shared SIMD inflate engine) for all DEFLATE decoding.

Sister projects: lbzip2-rs · ljar-rs · linflate-rs

Performance (32 cores, 1 GB input)

Usage

lgz input.gz -o output       # decompress to file
lgz input.gz                  # decompress to stdout
lgz input.tar.gz -o dir/     # extract tar.gz

Parallel Strategies

Three strategies, tried in priority order:

1. Flush-boundary split — SIMD scan for 00 00 FF FF markers (pigz/BGZF), split at boundaries, decode segments in parallel.
2. Concatenated gzip members — Scan for 1f 8b 08 headers, decode fully independent members in parallel.
3. Pugz-style speculative — Two-pass: scan for stored-block LEN/NLEN patterns, probe-decode to confirm, split with 32 KB window prefix.

Falls back to single-threaded linflate decode if no split points found (with ISIZE hint for optimal allocation).

Architecture

Ring-slot pipeline (shared with lbzip2-rs/ljar-rs):

Reader → Main (carry + split) → N Workers (decode_segment) → Collector → Writer
         6 slots × 232 MB        zero-copy raw-ptr access

• AVX-512 / AVX2 / scalar SIMD flush scanner (runtime-selected)
• Sync-flush vs full-flush auto-detection (5-byte pattern match, zero-cost)
• --features timing instrumentation (zero overhead when disabled)

Dependencies

Acknowledgements

Two-pass speculative decode inspired by pugz (Kerbiriou & Chikhi, 2019, MIT).

License

MIT OR Apache-2.0