nordisk/znippy

znippy perf — batch query + ordered bulk decompress (future)

Status: IDEAS / future perf work, not scheduled. Two API-shape wins on top of the existing engine (already among the fastest); both are measurable with the holger bench harness before any commit. Grounded in the v0.7 multi-index + content-addressed blobs + parallel zstd + Arrow-IPC index structure.

A. Batch query — get_many(&[coord]) -> Vec<Option<Package>>

One-at-a-time extract_file / view.get structurally forgoes four wins a batch unlocks (it's not just call-overhead amortization):

1. Random I/O → sequential. Resolve all N coords → FileLocs, sort by blob_offset, issue coalesced/sequential reads (or readv) instead of N scattered preads. Dominant cost on cold archives (USB/optical in airgap) — random→sequential is the big multiplier there, before decompress.
2. Decompress-once on shared blobs. Content-addressed ⇒ identical bytes = one blob. A dep tree repeating a transitive crate, or pom+jar sharing content, today decompresses the shared blob once per caller; a batch dedups by blob id and decompresses once, fanning out.
3. Cross-file parallelism. Within-file chunk decompress is already parallel; across files the per-call API serializes (caller awaits each). A batch feeds the whole work-list to rayon → saturates cores even for many-small-files.
4. One index pass. Project coord columns once for the batch, not N borrows.

Use case: dependency-tree resolution / holger serving a client's whole resolution set (N-at-once). API: get_many on ZnippyReader + each typed view, keeping the lazy split (batch-resolve handles cheap; batch-materialize bytes = the win). Keep single get — batch is the cold/many path; N=1 or page-cached is a wash.

B. Ordered bulk decompress (export / seal / full extract)

Single-file is already optimal; this is for whole-archive decompression.

1. Columnar SIMD index scan. Index is Arrow IPC → build the work-list (total output size, offset sort, pkg_type filter) as a vectorized pass over contiguous offset/size columns, not row-by-row. (See znippy-arrowipc-analysis.md.)
2. Read in blob_offset order; partition contiguously per core. Avoid striping work across cores (→ random I/O per core). Give each core a contiguous offset range → sequential reads + per-core L2/L3 stays warm. Sequential-per-core beats striped at the same core count.
3. Warm zstd window + buffer reuse. Decompress consecutive blobs (and a file's chunks in chunk_seq order) back-to-back → reuse output buffers, keep the decode working set hot; interleaving files thrashes.

Rule: let the Arrow columns drive a sorted, contiguously-partitioned, sequential plan — not directory/coord order.

De-risk before committing

Both measurable with the existing bench harness (holger/bench-scenarios + xtask/examples/nornir-bench.rs). Add two scenarios on a huge-scale archive:

• get_many batch-read vs N× single, cold + warm (drop page cache between).
• full-extract ordered vs naive mode. First check what archive.rs:154's loop already does (within-file parallelism likely exists; cross-file batching + bulk-ordering likely don't). Numbers first.