nordisk/znippy

znippy — design (f0, the whole system)

Class: design · Function axis: f0 (the system) + f1 compress · f2 decompress · f3 index/seek · f4 plugins · f5 zoomies · f6 iceberg/seal · f7 signing. Strict: this file states only the shape we run. Every why and every rejected alternative lives in reasoning.md; the per-function detail lives in each function's own *-design.md (linked below). How to use it: guide.md.

f0 — one archive, three properties

znippy is a content-addressed, seekable compressor: it packs a directory across every core, lets you pull one file back without touching the rest, and keeps its index as plain Arrow IPC so an outside SQL engine reads it with no znippy code. One container carries all three:

[ blob_0 … blob_N ]                      compressed or stored-raw chunk bytes
[ data sub-index_0 … _M ]                one Arrow IPC stream per (pkg_type, repo)
[ lookup sub-index ]                     base columns, sorted by (relative_path, chunk_seq)
[ trie blob ]                            fst::Map<relative_path → first lookup row>
[ manifest Arrow IPC ]                   one row per section
[ 8-byte MAGIC "ZNPYMIDX" ][ u64 manifest_offset ]

Base index columns (every archive): relative_path · chunk_seq · fdata_offset · compressed · uncompressed_size · blob_offset · blob_size · checksum (32-byte blake3). Schema: znippy-common/src/index.rs. Each sub-index is a valid Arrow IPC file — DuckDB / Polars / DataFusion read it directly.

Four laws hold everywhere below:

How the functions wire together (the reuse graph)

The f5 gatling engine is the shared substrate: f1 packs through it, and osm-katana's xml→pbf path runs on the same 1 reader → N workers → 1 writer shape. f3 is the spine every reader touches — f2 restores through it, f6 seals into it, f7 signs the roots it computes. Everything outside f6 is tokio-free.

f1 — compress (the write path)

One no-barrier engine, two entry points. slot_packer.rs::compress_dir (io_uring, 128 files/batch into ChunkRevolver coalescing slots) and stream_packer.rs::compress_stream (in-memory Arc<Vec<u8>> rounds) share worker/writer/finalizer: every input is split into slices (the unit of both blake3 integrity and parallel work) feeding one global slice queue; N workers blake3 the original bytes, compress_into a reused buffer (or store raw when the frame ≥ input), and pwrite at an AtomicU64 cursor while the finalizer builds the Arrow index from ChunkMeta only. Detail: compress-design.md.

f2 — decompress (the read path)

decompress.rs: N workers share one atomic row cursor; each fetch_add(row) → pread(blob_offset) → decompress_into(reused buf) → blake3 verify → pwrite(fdata_offset). No reader thread, no channel — the index has every offset, so parallel pread gives NVMe its real queue depth. Detail: decompress-design.md.

f3 — index & random-access seek

The lookup sub-index (base columns re-sorted by (relative_path, chunk_seq)) + an fst::Map trie give three ways to get a row: eager full-index ZnippyArchive::open, lean get_file → locate_file (trie O(key) → binary-search O(log n) → legacy scan), and external Arrow-IPC SQL. Both live as reserved manifest sections (pkg_type = i8::MIN), so the 16-byte footer is unchanged and older archives still open. Detail: index-design.md.

f4 — plugins (package-aware typed index)

A plugin implements ZnippyPlugin (plugin.rs): matches_path, schema_fields, extract_metadata / extract_batch, meta, run_command. The writer composes the schema as base columns + the union of registered plugins' fields; built-ins register once in handlers.rs::builtin_handlers() and the CLI autodiscovers via meta(). Detail: plugins-design.md.

f5 — zoomies (fast primitives)

Pure-Rust parallel decoders — ljar, lgz, lbzip2, lzip-parallel — on one shared SIMD DEFLATE core (linflate), plus the generic gatling streaming engine, the stree static search tree, and osm-katana. Each decoder borrows a &[u8] and never copies its input. Detail: zoomies-design.md; measured throughput: benchmarks.md.

f6 — iceberg ⇆ seal

IcebergSink keeps the sub-index rows in skade/Iceberg tables (durable catalog.redb) — a dynamic store that survives restarts, resumes, and is queryable live; the .znippy sidecar holds pure blobs. seal() snapshots it into a static native container, reusing blobs verbatim and rewriting only the metadata tail byte-identically to a fresh compress. Detail: iceberg-design.md.

f7 — provenance signing

Optional detached CMS on top of the per-chunk blake3 integrity: each signature is a merkle-fold of the chunk hashes already in hand (no content re-read), written as reserved manifest sections via ArrowIpcSink::with_signer. Off by default (feature sign); unsigned archives are byte-unchanged. Detail: signing-design.md.

CLI surface

znippy compress | decompress | list | get | verify (format-level, type-agnostic), znippy seal (feature iceberg), znippy handlers (list plugins), znippy run <handler> <cmd>…. Entry: znippy-cli/src/lib.rs.

Dependency graph

LAW 6 — the canonical branch, and the guard that makes the trap impossible

Ask the repository which branch is canonical. Never assume, never hardcode, never memorise a table. Several repositories in this constellation carry BOTH a main and a master on the remote and they do not agree on which is canonical — one repository's trunk is the next one's abandoned branch. Enumerate them; do not learn them:

for r in ~/git/*/; do
  both=$(git -C "$r" ls-remote --heads origin main master | wc -l)
  [ "$both" = 2 ] && printf '%-20s canonical=%s\n' "$(basename "$r")" \
      "$(git -C "$r" symbolic-ref --short refs/remotes/origin/HEAD)"
done

On 2026-08-30 that listed exactly three, and the answer was not the same for all three:

Two lanes pushed to a non-canonical trunk in one week. The gunnar one was harmless only by luck — that tip happened to be an ancestor of master, so nothing was stranded; the znippy one was not.

What the hook refuses: a push whose destination is a trunk name (main, master, trunk, develop) that is not this repository's canonical trunk. A deliberate push to a feature branch is untouched — roadmap/obruten-kedja, lane/whatever, tags: all pass. The trap is the second default-looking branch, not branches in general.

How it knows: git symbolic-ref refs/remotes/<remote>/HEAD — the remote's own answer — falling back to asking the server with git ls-remote --symref. The canonical name is never written in the hook; the same bytes therefore give opposite answers in znippy and znippy-zoomies, which is the whole design. A fourth place for the repositories to disagree would be the bug, not the fix. If neither source answers, the hook says so and allows: a hook that blocked every push on a repository it could not resolve would be deleted within the day, and then nothing would guard anything.

The refusal names the branch that is right, the command that pushes there (git push <remote> HEAD:<canonical>), the enumeration above, and the deliberate override (GIT_ALLOW_NONCANONICAL_TRUNK_PUSH=1) for the one honest case — maintaining or retiring the non-canonical trunk on purpose.

Install (hooks live outside the working tree, so a clone does not get them):

cp .githooks/pre-push .git/hooks/pre-push && chmod +x .git/hooks/pre-push

The hook chains to git lfs pre-push itself when .gitattributes tracks LFS paths (it buffers stdin and replays it), so it replaces the LFS hook rather than fighting it.

Proven both ways in all three repositories (LAW 2 — never accept a green you have not seen red), 2026-08-30:

and end to end: git push --dry-run origin HEAD:main is refused in gunnar and in znippy, while git push --dry-run origin master / origin main still runs the LFS half.

Which recipe reader this repo uses, and why (lane AJ, 2026-08-30)

Reader: nornir-build-thing 0.1.12, PATH (tests/, xtask/).

Why: in-constellation; recipe is schema 1. znippy's graph ALSO held a registry copy at 0.1.3 — eight generations behind the path copy, in one graph — pulled in by published modgunn, whose own floating "0.1" let it sit there. Raising modgunn's floor to 0.1.11 moved it to 0.1.11. znippy keeps NO [patch.crates-io] on purpose (a patch fork is a structural publish blocker), so the last generation of gap closes on the release cascade.

The rule itself — [nornir-build-thing] schema = N, a refusal that names the version to upgrade to, and a loader that refuses to DROP a key it cannot read — lives once, in edda .nornir/build-design.md, "THE COMPATIBILITY RULE". The cross-version guard that would have caught the 2026-08-30 skew is edda/nornir-build-thing-skew.