nordisk/znippy

znippy — complete overview

The real README: the full pitch, quickstart, what's-new, and every measured benchmark table. The repo-root README.md is the slim face (title · mashup · pointers); this chapter is the heavy detail, and it is also folded into the manual docs/manual.pdf by nornir docs book.

Why znippy — the only archive that's all-core on both ends, random-access per file, and directly queryable, without pretending to beat zstd/7z on raw ratio:

Full per-backend throughput tables are below — generated by nornir bench run znippy and historized in the warehouse (machine + core count in each header).

What it does

• Pack a directory at all-core speed — a no-barrier Gatling pipeline across every core; already-compressed inputs (.jar, .gz, .crate, .png, .parquet, …) are stored verbatim, never re-coded.
• Pull any single file back without unpacking the rest — O(log n)/O(key) via the sorted lookup sub-index + FST trie + ZNPYMIDX footer (znippy get).
• Query it with no znippy library — the lookup sub-index is a valid Arrow IPC stream, so DuckDB / Polars / DataFusion read it directly (see .nornir/random-access.md).
• Package-aware metadata via plugins (maven, python, …) — the ecosystem view for airgap mirrors.
• Dynamic ⇆ sealed — an Iceberg-backed store that resumes across restarts, then seal()s to a static, content-addressed native .znippy (blobs reused byte-for-byte).

Quickstart

# pack a directory across all cores, list it, then pull ONE file back
znippy compress ./mydir mydir.znippy
znippy list mydir.znippy
znippy get -i mydir.znippy -p path/inside/archive.txt -o out.txt   # omit -o ⇒ stdout

# the index is plain Arrow IPC — query it with zero znippy code, e.g.:
#   SELECT relative_path, blob_offset, blob_size FROM '<lookup>.arrow' WHERE relative_path = '…'

What's new

The latest release cascade (znippy 0.9.2, znippy-common 0.9.4, znippy-compress 0.9.3, znippy-decompress 0.9.2, znippy-cli 0.9.2, znippy-iceberg 0.1.3, znippy-zoomies 0.1.11, osm-katana 0.2.0):

• Iceberg with skade as the default catalog. znippy-iceberg's IcebergSink / IcebergZnippyReader keep the archive's sub-index rows in skade/Iceberg tables backed by a durable catalog.redb (RedbCatalog) — the dynamic archive store: it survives a process restart, accepts new commits (resume), and is queryable live (DuckDB/Polars over the Iceberg tables). The .znippy sidecar holds pure blobs; metadata lives in the warehouse.
• Sealed native export — seal(). znippy seal (and znippy_iceberg::seal()) snapshots that dynamic Iceberg store into a static, immutable, content-addressed native arrow-ipc .znippy. Blobs are reused verbatim (no recompress); only the metadata tail (sorted lookup + fst trie + manifest + ZNPYMIDX footer) is rewritten, byte-identical to a fresh compress. The result opens with the ordinary ZnippyArchive reader with no warehouse needed. See .nornir/iceberg-seal-export.md.
• Reboot-capable ArrowIpcSinkAppend. A clone of the native arrow-ipc writer that supports the full write → restart → resume → seal lifecycle: append more entries across a process boundary, resume, then seal to a static file — proven byte-exact by znippy-iceberg/tests/lifecycle_seal.rs, with A/B parity against the one-shot writer (znippy-common 0.9.4 meta_sink_append).
• osm-katana region-clip (--clip / --poly). OSM → GeoParquet/PBF now carves a region during conversion: --clip takes a bbox or preset (europe / nordics), --poly clips to a region's real shape from an Osmosis .poly or GeoJSON (Polygon/MultiPolygon) — e.g. the real Sweden polygon — via a point-in-polygon test on the node stream. See .nornir/osm-katana-gatling-vs-rayon.md.
• Five arrow-ipc baseline benches. The single-threaded metadata-seal tail and cold random-access read path are now isolated, stable metrics (sub-index write, footer seal, lookup re-materialise, fst trie, locate_file), baselined before any write-path optimisation lands (bench-first gate) and audited A/B against the append-clone. See .nornir/arrow-ipc-benches.md.

v0.9.6 · oden · 32 cores · 2026-07-22

Benchmarks

All numbers are generated by nornir bench run znippy (no hand-editing) and stored in the warehouse keyed by git SHA. The header shows the machine + cores. _st rows are single-core — the same decode pinned to one CPU (taskset -c 0, so available_parallelism()==1); the rest use all cores. The faster cell of each ours-vs-legacy pair is bold; speedup_x is ours ÷ legacy.

znippy-zoomies decompressors vs legacy

Parallel pure-Rust decompressors against the legacy C tools, decoding the same real Maven Central artifacts (the _st row is the honest per-core comparison).

Raw SIMD-DEFLATE core (linflate) vs miniz_oxide — single stream, single core, decoding real class-file data (the shared inflate engine under ljar/lgz/lzip-parallel; see .nornir/inflate.md).

v0.9.6 · oden · 32 cores · 2026-07-22

Parallel JAR decode vs unzip -p

v0.9.6 · oden · 32 cores · 2026-07-22

Parallel ZIP decode vs unzip -p

v0.9.6 · oden · 32 cores · 2026-07-22

Parallel bzip2 decode vs C lbzip2 (1995-lineage bzip2)

v0.9.6 · oden · 32 cores · 2026-07-22

Parallel gzip decode vs pigz -dc

v0.9.6 · oden · 32 cores · 2026-07-22

Tests

Every Rust test in the workspace, grouped by the standard categories. Run with cargo test --workspace (or cargo nextest run --workspace). Vendored upstream-zstd C tests (see Vendored C tests below) run through the C toolchain, not Cargo.

Unit tests (111)

Integration tests (49)

Doc tests (1)

Criterion benchmarks

11 criterion targets across 4 bench files. Run with cargo bench -p <crate> and view HTML reports under target/criterion/.

znippy-zoomies/ljar

znippy-zoomies/lzip-parallel

Vendored C tests (znippy-zoomies/zstd-sys-rs/zstd)

Upstream Facebook/zstd test suite carried in-tree for reproducible builds of the zstd-sys-rs bindings.

Dependency graph