.nornir/decompress-design.md · nordisk/znippy

znippy decompress — design

Class: design · Function axis: f0 (the restore) + f1 worker engine · f2 single-file read · f3 safety + CLI. Strict: this file states only the shape we run. Every why, every alternative, lives in decompress-reasoning.md; the how-to is in decompress-guide.md. For the index this reads against, see index-design.md.

f0 — the restore (the whole)

A znippy archive is a content-addressed, seekable blob: every chunk's byte position (blob_offset, blob_size), its output position (fdata_offset), its codec flag (compressed) and its blake3 checksum already live in the index. Restore therefore is not a stream — it is pure random access. There is no reader thread, no channel, no slot pool: N workers share one atomic row cursor and each pulls its own bytes.

Full restore: znippy-common/src/decompress.rs:65 decompress_archive(...); selective restore (by pkg_type / repo sub-index) :76 decompress_archive_filtered(...). Both re-exported at the top level as znippy-decompress/src/unpacker.rs:5 decompress_archive / :11 decompress_archive_filtered.

Three laws govern the restore; a run is conformant iff all hold:

law	statement	enforced by
L1 — random-access input	input is never streamed; each chunk is `pread` from its `blob_offset`. Parallel `pread` gives the NVMe real queue depth	positioned `read_exact_at` (`decompress.rs:225`)
L2 — no hot-path alloc	every worker reuses one read buffer + one output buffer across all its rows	per-worker `read_buf` / `out_buf` reuse (`decompress.rs`)
L3 — deterministic, core-count-independent	every chunk's checksum is bound to its `chunk_seq`; verification never depends on worker count or completion order	`blake3::hash(out)` vs per-chunk checksum (`decompress.rs:250`)

L3 is the load-bearing one: because a checksum is bound to chunk_seq (not to a worker or an arrival slot), the result is byte-identical whether you decompress on 1 core or 128, and a corrupt archive is detected rather than silently rewritten.

Reuse map: decompress depends only on the index reader (index.rs), the codec (codec.rs), and blake3. It is used by the znippy CLI (decompress / get / verify) and by any caller that links znippy-decompress.

f1 — the worker engine

decompress_archive_filtered (decompress.rs:76) opens the archive once as a shared Arc<File>, builds the sorted work list from the index, and spawns N workers over one Arc<AtomicUsize> row cursor (decompress.rs:162). Each worker loops:

row = cursor.fetch_add(1, Relaxed)          // decompress.rs:194  claim next row, lock-free
if row >= total { break }
blob_offset, blob_size, fdata_offset,
compressed, uncompressed_size, checksum  ←  index columns for `row`
pread(blob_offset, blob_size) → read_buf     // decompress.rs:225  positioned, no shared seek
out = compressed ? decompress_into(read_buf, out_buf)   // codec.rs:66  OpenZL, buffer reuse
              : read_buf                                //          raw copy when not compressed
assert blake3::hash(out) == checksum         // decompress.rs:250  L3
pwrite(out, fdata_offset)                    // decompress.rs:269  positioned write, write_all_at

Rows that share a relative_path share one Arc<File> output handle; pwrite is positioned, so ordering between workers is irrelevant. No worker ever waits on another — the only shared mutable state is the atomic cursor. Codec: znippy-common/src/codec.rs:66 decompress_into decompresses into a caller-owned Vec (buffer reuse, no per-chunk allocation).

f2 — single-file random read

To pull one file without touching the rest of the archive: znippy-common/src/decompress.rs:318 get_file(archive_path, target). It resolves the target's contiguous chunk run through znippy-common/src/index.rs:736 locate_file (see index-design.md f2 for the three-tier resolver), then for each ChunkLoc: pread(blob_offset) → decompress_into (or raw copy when compressed is false) → blake3-verify against the per-chunk checksum → place the bytes at fdata_offset. Returns the reassembled file bytes. Same L1/L3 discipline as f1, single-threaded.

f3 — safety and the CLI surface

Every output path is passed through safe_output_path(out_dir, rel_path) (decompress.rs:27) before any write: it rejects zip-slip (../) and absolute paths, so a hostile index can never escape out_dir. Guaranteed by safe_output_path_rejects_zip_slip_and_absolute_paths and decompress_corrupt_archive_errors_not_panics (a corrupt archive errors, never panics).

command	entry	does
`znippy decompress -i <archive> -o <dir> [--type maven\|rust\|python] [--repo NAME]`	`znippy-cli/src/lib.rs:379 Commands::Decompress`	full or filtered restore (f0/f1)
`znippy get -i <archive> -p <relative/path> [-o out]` (omit `-o` ⇒ stdout)	`:420 Commands::Get`	single-file read (f2)
`znippy verify -i <archive>`	`:434 Commands::Verify`	decompress-and-verify every chunk, write nothing

--type / --repo narrow the restore to a data sub-index; an empty filter equals full restore. verify is exactly f1 with the pwrite suppressed — it proves every checksum without producing files.