nordisk/znippy

znippy — guide (f0, the whole workflow)

Class: guide · audience: a techie packing, seeking, or querying a znippy archive. The what is in design.md; the why in reasoning.md. Per-function how-tos: compress · decompress · index/seek · plugins · zoomies · iceberg/seal · signing.

f0 — one end-to-end session

The whole tool in five commands. Build once, then pack, inspect, and pull:

cargo xtask build --suite ORIGINAL --atom znippy   # produces target/release/znippy

# f1 — pack a directory across all cores (already-compressed inputs stored verbatim)
znippy compress -i ./mydir -o mydir.znippy

# f3 — inspect: list every file, then seek ONE back without unpacking the rest
znippy list mydir.znippy
znippy get -i mydir.znippy -p path/inside/archive.txt -o out.txt   # omit -o ⇒ stdout

# f2 — restore everything (or a slice) and verify per-chunk checksums
znippy decompress -i mydir.znippy -o ./restored
znippy verify   -i mydir.znippy

compress writes blobs + the Arrow-IPC index + the fst trie + the ZNPYMIDX footer in one pass; every read verb below rides that same footer.

f1 — compress

znippy compress -i <dir> -o <out.znippy> \
  [--no-skip]                      # force-compress even already-compressed inputs
  [--format rust|python|maven]     # which package plugin fills the typed columns (default rust)
  [--meta-format arrow-ipc|iceberg] [--warehouse DIR]   # inline footer (default) vs dynamic store
  [--sign KEY --sign-cert CERT --sign-alg p256|ed25519] # detached provenance (feature sign)

One archive carries one package type. See compress-guide.md for the engine knobs and the in-memory compress_stream path (HTTP / holger).

f2 — decompress & verify

znippy decompress -i <archive> -o <dir> [--type maven|rust|python] [--repo NAME]  # selective
znippy verify     -i <archive> [--signed --root ca.der]                           # +provenance

Restore is deterministic and core-count-independent (checksums bound to chunk_seq). Detail: decompress-guide.md.

f3 — seek one file, three ways

// (1) hold the archive open, pull many files — O(1) after an O(n) open
let a = znippy_common::archive::ZnippyArchive::open(path)?;
let bytes = a.extract_file("com/google/guava/guava/33.2.0/guava-33.2.0.jar")?;

// (2) one file, no full-index load — trie O(key) → binary search → legacy scan
let bytes = znippy_common::get_file(path, "pom.xml")?;

// browse metadata only (no bytes): whole tree, or one subtree via the trie
let all = znippy_common::get_all_files_meta(path)?;
let sub = znippy_common::get_files_meta_with_prefix(path, "com/google/guava/")?;

-- (3) no znippy at all: the lookup sub-index is a sorted Arrow IPC stream
SELECT blob_offset, blob_size, uncompressed_size, compressed, chunk_seq, checksum
FROM   '<lookup-slice>.arrow'
WHERE  relative_path = 'com/google/guava/guava/33.2.0/guava-33.2.0.jar'
ORDER  BY chunk_seq;

Detail: index-guide.md.

f4 — plugins

znippy handlers                          # list the compiled-in register
znippy compress -i ./crates -o c.znippy --format rust
znippy run rust coords foo.crate         # dispatch to a handler's run_command

Write one by implementing the six ZnippyPlugin methods and adding it to builtin_handlers(). Detail: plugins-guide.md.

f5 — zoomies

The parallel decoders and the gatling engine ship in the sibling znippy-zoomies repo. Drive one directly, e.g. ljar to decode a JAR across cores, or plug a Codec/Sink into gatling::run. Detail + worked codec: zoomies-guide.md; measured throughput: benchmarks.md.

f6 — dynamic store, then seal

# build a dynamic, resumable, live-queryable Iceberg-backed archive (feature iceberg)
znippy compress -i ./mirror -o mirror.znippy --meta-format iceberg --warehouse ./wh
# …later, freeze it into a static, content-addressed native .znippy (blobs reused verbatim)
znippy seal -i mirror.znippy --warehouse ./wh -o mirror-sealed.znippy

Detail: iceberg-guide.md.

f7 — sign & verify provenance

znippy compress -i ./art -o art.znippy --sign key.p8.der --sign-cert cert.der --sign-alg p256
znippy verify   -i art.znippy --signed --root ca.der

Detail: signing-guide.md.

Benchmarks

One suite, run via cargo run -p xtask --release --example nornir-bench (nornir's Bencher, not criterion). Numbers land in the warehouse and render the README tables. What the decompressors do against the legacy C tools, and the write-tail costs: benchmarks.md.