nordisk/znippy-zoomies

why cant # ljar-rs Backlog

Working notes that survive token-limit truncation. Cross-check 2026-05-19.

Cross-check: lbzip2-rs vs ljar-rs (2026-05-19)

DESIGN.md and src/bin/ljar.rs are now aligned. The binary implements the full 5-thread pipeline mirroring lbzip2-rs/src/bin/lbunzip2.rs:

Bug fixed (2026-05-19)

decode_segment_into in chunk.rs was passing out_pos = 0 to miniz_oxide::decompress on every iteration. With TINFL_FLAG_USING_NON_WRAPPING_OUTPUT_BUF, the decompressor needs to see all prior output for LZ77 back-references. Fixed to pass the cumulative output length as out_pos.

Backlog: full pipeline rewrite

B1 — ✅ DONE: Full 5-thread pipeline

src/bin/ljar.rs now mirrors lbzip2-rs/src/bin/lbunzip2.rs exactly: Reader → Main(carry+split) → N Workers → Collector → Writer, with 6 cycling ring slots and Arc<Mutex<Receiver<WorkItem>>>.

Small entries (< 4 MB) bypass ring slots as ad-hoc single-segment chunks. Entries without Z_FULL_FLUSH boundaries accumulate into a fallback buffer and decompress as one segment (single-core).

B2 — ✅ DONE: Entry-boundary handling

Implemented via FilledSlot with is_first_of_entry / is_last_of_entry flags. Reader signals boundaries; main thread resets carry at entry transitions.

B3 — ✅ DONE: Slot-pool sizing

6 slots = up to 1.2 GB in flight. Entries larger than that still work: reader blocks on free_rx.recv() until a slot is recycled, then fills the next chunk. Pipeline depth is bounded but throughput is the same. Documented in constants.

B4 — ✅ DONE: Fallback for entries without Z_FULL_FLUSH

Entries without flush boundaries (standard jar tool output) now decompress correctly. Small entries (< 4 MB) are single-segment chunks through the normal pipeline. Multi-chunk entries without flush boundaries accumulate into a fallback buffer and decompress as a single segment when the entry ends.

B5 — Benchmark harness

Once B1 lands, benchmark against:

• unzip (single-core C reference)
• bsdtar -xf (uses libarchive)
• python -m zipfile -e (single-core)
• ljar-rs v0 (current serial binary)
• ljar-rs v1 (full pipeline)

Test JARs:

• Synthetic: 1 × 1 GB DEFLATE entry with Z_FULL_FLUSH every 32 MB
• Real: large Spring Boot fat-jar (~100 MB, many small entries)
• Real: any large Maven artifact downloaded into /home/rickard/work

B6 — Zero-copy decompressed output (DESIGN Rule 5.2 + 6.3)

Currently decode_segment_into allocates a Vec<u8> per segment. The lbzip2 pattern reserve + set_len + decompress_into is in place but the segment Vec itself is freshly allocated each call. To match Rule 6.3 (zero-copy offset variant):

• Pre-allocate one Vec<u8> of uncompressed_size per entry
• Workers write directly into output[segment_offset..]
• Writer thread receives (entry_idx, written_len) only — no copy

This is the production zero-copy path. Convenience API (decompress_jar) keeps the current allocating behaviour.

B7 — Custom DEFLATE decompressor (src/inflate/)

Cherry-picks the best from libdeflate, zlib-ng, zlib-rs, and lbzip2-rs:

Architecture: libdeflate's full-buffer model (no circular window — ljar already has complete segments in memory). Eliminates all window arithmetic, wrap-around masking, and "copy from window" paths.

Key design choices:

• 11-bit litlen first-level Huffman table (libdeflate) — eliminates subtable lookups for virtually all real streams. Packed u32 entries with length base + extra bits baked in — one expression per match decode.
• Branchless 64-bit bit refill (zlib-ng XOR trick) — 4 instructions, no branch.
• 3-literal unroll per iteration (zlib-ng) — best for literal-heavy class files.
• Preload-before-copy pattern (libdeflate) — next Huffman entry loaded BEFORE match copy runs, hiding L1 load latency behind copy instructions.
• Const-generic <N> SIMD match copy (zlib-rs) — LLVM emits vmovdqu ymm for N=32 (AVX2), movdqu xmm for N=16 (SSE2/NEON). Clean Rust, no intrinsics.
• AVX2 _mm256_shuffle_epi8 tiling for short back-refs dist 2..7 (zlib-ng) — single instruction tiles pattern across 32-byte register.
• Thread-local table pool (lbzip2-rs pattern) — zero heap allocation per block.
• inflate_into(&mut [u8]) zero-copy API — caller owns output buffer.
• #[cold] #[inline(never)] slow paths (lbzip2-rs) — keep fast path hot.

Cache budget: ~13 KB total hot working set (litlen 9.4 KB + dist 1.6 KB + precode 0.5 KB + BitReader 32 B + scratch 1.3 KB) — fits entirely in 32 KB L1-D.

Implementation order:

1. bitreader.rs — branchless refill, peek/consume, BMI2 variant
2. tables.rs — packed u32 entry format, 11-bit litlen, table build
3. fixed.rs — pre-built fixed Huffman tables (static)
4. copy.rs — SIMD match copy with const-generic N, AVX2 short-distance
5. fastloop.rs — the hot loop with preload-before-copy
6. mod.rs — public API, block parsing, dynamic header decode
7. Integration tests: verify against miniz_oxide on all 20 Maven JARs
8. Benchmarks: compare against miniz_oxide, measure throughput per core

Full design in src/inflate/DESIGN.md.

Where things live

Reference impl:

• /run/media/rickard/T9/git/lbzip2-rs/src/bin/lbunzip2.rs — the model for B1
• /run/media/rickard/T9/git/lbzip2-rs/src/chunk.rs — split_chunk pattern

Useful test scratch

Generate test JARs in /home/rickard/work/ (per user preference):

python3 -c "
import zipfile, io
buf = io.BytesIO()
with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf:
    zf.writestr('Hello.class', b'cafebabe' * 1000)
open('/home/rickard/work/test.jar','wb').write(buf.getvalue())
"

Run:

cargo run --bin ljar -- /home/rickard/work/test.jar             # list
cargo run --bin ljar -- /home/rickard/work/test.jar /home/rickard/work/out  # extract