CLAUDE.md

RULE: Never add Co-Authored-By or any AI attribution to commits. Ever.

znippy

Arrow IPC archive format with parallel compress/decompress, package-aware metadata extraction, and random-access reads. Repository: https://git.gunnar.rs/nordisk/znippy

Workspace layout

znippy-common/        trait definitions, plugin API, shared types, index schema
znippy-compress/      write path: read → plugin extract → chunk → compress → write
znippy-decompress/    read path: decompress → reconstruct files
znippy/               THE binary `znippy` — a 3-line wrapper over znippy_cli::run()
znippy-cli/           LIBRARY holding all the CLI code (handlers.rs = handler register).
                      Its duplicate `znippy-cli` binary was removed 2026-08-03: it was a
                      byte-identical copy of znippy/src/main.rs. Versions move in lockstep
                      with znippy/ (tests/tests/build_variants.rs enforces it).
plugins/               EVERY handler crate, split by the target it is built for.
                       A plugin crate lives in one of these two directories and
                       nowhere else; see plugins/README.md.
plugins/native/znippy-plugin-python/ wheel/sdist metadata — rlib+cdylib     [type_id 2]
plugins/native/znippy-plugin-maven/  JAR/pom.xml metadata — rlib+cdylib     [type_id 3]
plugins/native/znippy-plugin-media/  image/audio/video metadata — rlib     [type_id 25]
plugins/native/znippy-plugin-skidbladnir/    airgap deploy-bundle members  [type_id 40]
plugins/native/znippy-plugin-rust-toolchain/ airgap rust-dev bundle members[type_id 41]
plugins/native/znippy-plugin-git/    git objects keyed by oid hex — rlib   [type_id 42]
                       carries the reserved __gunnar_oid__ / __gunnar_graph__ /
                       __gunnar_reach__ / __gunnar_refs__ / __gunnar_secrets__ sections
plugins/wasm/znippy-plugin-maven-wasm/  THE wasm example: the maven handler
                       compiled to wasm32-unknown-unknown and loaded through
                       wasm_loader.rs. Holds no maven logic — it depends on
                       plugins/native/znippy-plugin-maven and is only the ABI shim.
znippy-iceberg/       Apache Iceberg integration
znippy-zoomies/       fast primitives: OSM index, gatling engine, psort
znippy-zoomies/{linflate,lgz,lbzip2,ljar,lzip-parallel,zstd-sys-rs}
znippy-zoomies/osm-katana   OSM → GeoParquet/PBF (CLI: convert/xml2pbf/pbf2geo/geo2arrow/verify)
tests/                integration + benchmark suite
xtask/                cargo xtask helpers

See WORKSPACE.md for full dependency graph and release gate protocol.

Design

File format

[ blob_0 ][ blob_1 ] ... [ blob_N ]
[ Arrow IPC sub-index ]    ← rows: path, pkg_type, chunk_seq, offset, len, checksum, metadata cols
[ Arrow IPC sub-index ]    ← one sub-index per plugin type (DenseUnion layout)
[ global footer ]          ← offsets to each sub-index, version, flags

Blobs are zstd or lz4 compressed chunks (or stored as-is for already-compressed files)
Each sub-index is a valid Arrow IPC file — readable by DuckDB/Polars/DataFusion directly
Already-compressed input is stored raw at full I/O speed — it never touches the codec. znippy-common/src/precompressed.rs resolves that in a fixed order: an explicit caller ContentHint (free and exact — SkipPolicy::already_compressed() covers a whole batch), then the extension table (.jar, .gz, .crate, .png, .parquet, .pack, .idx …), then a magic-byte probe over the first 16 bytes (zlib, PACK, gzip, zstd, xz, …), then compress. The probe is what reaches an entry carrying no extension — a git loose object is named for its oid. No entropy estimate: every probe is an exact pattern at a fixed offset, with no threshold to tune. The probe itself lives in znippy-zoomies/ldeflate/src/precompressed.rs and znippy-common re-exports it; only the extension leg (skip_by_path) is znippy's, because ldeflate is handed &[u8] and never a Path. The arrow cannot point the other way — znippy-common already depends on znippy-zoomies, so it would be a cycle.
The tree's shape — empty directories and unix modes — rides in the reserved __znippy_fsmeta__ section (znippy-common/src/fsmeta.rs), Arrow IPC (relative_path, kind, mode). The data index is a table of chunks, so a directory owning no file owned no row: an empty directory did not survive a round trip and a 0755 script came back 0664, both silently, with verify green. Reserved means every data reader skips it, so it is additive both ways — no format-version bump, and an older znippy opens the archive unchanged. One row per directory + ONE archive-scoped FileDefaultMode row (the tree's modal mode) + one row per deviating file, so the section is O(distinct modes) rather than a second copy of every path. Restore order is load-bearing: dirs before extraction, file modes after, directory modes last and deepest-first. See .nornir/compress-design.md §f7-f8.

Pipeline (compress path)

WalkDir
  │
  ├── BIG files (> 200 MB slice)
  │   └── sequential chunked reads → Gatling pipeline → compress → write
  │
  └── SMALL files (≤ 200 MB)
      └── read into slot (ChunkRevolver, coalescing) → batch plugin extract → chunk → compress → write
              │
              └── Arrow IPC index written incrementally per pass (no accumulation)

Design laws

Law 1 — Zero-copy channel communication. All inter-thread messages are slot indices (usize) into a pre-allocated ring buffer (ChunkRevolver). No Vec<u8> clones cross thread boundaries. On 32 cores saturating NVMe (~7 GB/s), copying payloads would halve throughput.

Law 2 — No allocation in hot path. Compressor threads reuse buffers across chunks. OpenZL context is long-lived. blake3 Hasher is .update()'d incrementally.

Law 3 — Deterministic checksums independent of core count. Checksums are bound to chunk_seq order, not thread identity. Any machine reproduces the same checksums by sorting decompressed chunks by chunk_seq before hashing.

Law 4 — Writer is concurrent, compressors never wait for index. The Arrow IPC writer runs on its own thread. Compressors send completed chunks and continue immediately. Index rows are buffered and flushed when a sub-index batch is full.

Gatling pipeline

No-barrier streaming engine in znippy-zoomies. Replaces the original rayon-based pipeline:

Fixed-size slot pool — producer fills slots, sends slot index, consumer drains
No barrier between read and compress stages
Backpressure from pool exhaustion (producer blocks when all slots are in-flight)
Used in both compress and osm-katana xml→pbf paths

io_uring (slot packer)

znippy-compress/src/slot_packer.rs uses io_uring batched opens+reads (128 files/batch) for the small-file path. Zero per-file syscall overhead — the kernel processes the entire batch asynchronously.

Plugin system

A plugin (package handler) targets one packaging ecosystem. It implements ArchiveTypePlugin (znippy-common/src/plugin.rs): ONE static HandlerSpec (name, type_id, aliases, ecosystem, extensions, columns, extra subcommands — each stated once) plus extract_metadata.

Method	Purpose
`spec()`	the static `HandlerSpec`
`extract_metadata(path, &data)`	per-file metadata (the only required logic)
`run_command(cmd, args)`	optional; default is the shared `coords` command
`matches_path(path)`	derived: ends with a spec extension (override for magic/shape matching)
`schema_fields()`	derived from `spec.fields`
`name()` / `type_id()` / `meta()`	derived from the spec (`meta()` is the owned discovery record)

Discovery: znippy-cli/src/handlers.rs::builtin_handlers() is the single registration point. The CLI autodiscovers handlers via meta() — no central name table maintained by hand.

znippy handlers                    # list register
znippy compress --format rust …    # select handler by name/alias
znippy run rust coords foo.crate   # dispatch to handler's run_command()

Where a handler's source lives — rule P-5 (.nornir/plugins-design.md §f3):

tier	where	condition
A	`znippy-common/src/plugins/native/*.rs`	needs only `std`, `arrow` and `znippy-common`
B	its own crate under `plugins/native/`	needs a third-party parser, a decompressor, a network fetch, its own crate-type or its own release cadence
W	its own crate under `plugins/wasm/`	is an ABI shim over a tier-B crate, built for `wasm32-unknown-unknown` and loaded by `wasm_loader.rs`

znippy-common sits under every other crate, so a dependency added there is paid by all of them on every target including wasm32 — that is the whole reason for the A/B split. Neither tier is a staging area for the other, and a tier-B crate with its modules at src/ root is the normal Rust layout, not a misplacement. Tier A currently holds cargo/npm/rpm/deb/gem/conda plus the skeletons.rs stubs; tier B holds the six crates under plugins/native/; tier W holds znippy-plugin-maven-wasm.

Tier A does not move, and the reason is a consumer, not a preference. The six in-common handlers are reached as znippy_common::plugins::<x>_native::<X>Plugin from outside this workspace — holger/traits/src/lib.rs names six of them in non-test code, and four holger repository crates name them in tests. Lifting them into plugins/native/znippy-plugin-* crates would need those crates to depend on znippy-common (for ArchiveTypePlugin) while znippy-common re-exported them back — a package cycle cargo rejects — so the public path would simply disappear from a published crate. Measured 2026-08-05: the six cost gunnar's cargo xtask guard zero crates (they import only std + arrow, and gunnar-coldtier is excluded from gunnar's workspace, so znippy is not in that graph at all), so there is no dependency-weight argument for paying that price.

Tier W is where the wasm ABI lives, and it is a separate crate on purpose. plugins/wasm/znippy-plugin-maven-wasm holds no maven logic: it depends on plugins/native/znippy-plugin-maven and exports only alloc / result_len / extract / plugin_extensions / plugin_schema. Both surfaces read the same row.rs — one MAVEN_SCHEMA, one field writer — and plugins/wasm/znippy-plugin-maven-wasm/tests/maven_native_wasm_agree.rs loads the real .wasm and asserts the two produce the same ExtensionRow (it fails, never skips, when the module is missing or stale).

cargo build --target wasm32-unknown-unknown --release -p znippy-plugin-maven-wasm
cargo test -p znippy-plugin-maven-wasm      # the agreement guard; needs the build above

Compile targets are per crate, not uniform: znippy-plugin-maven and znippy-plugin-python still declare cdylib + rlib, but nothing loads that host .so — the WASM artifact comes from tier W. media, skidbladnir, rust-toolchain and git are rlib-only compiled-in host handlers.

CLI surface

znippy compress   <dir>         # compress directory → .znippy archive
znippy compress --format generic <dir>   # a tree that is NOT a package registry
                                #   (aliases none/raw/plain/blob, type_id 26): no
                                #   metadata, no pkg_type claimed. The default
                                #   `rust` extracts nothing from such a tree either,
                                #   but records pkg_type = 1 — so `--type rust` and
                                #   every RustView consumer would report a
                                #   container-registry mirror as crates.io content.
znippy append -i <archive> -a <dir>  # append a dir into a sealed archive (reuses old blobs, re-seals)
znippy decompress <archive>     # reconstruct files
znippy list       <archive>     # list contents (Arrow query)
znippy verify     <archive>     # verify checksums
znippy handlers                 # list registered plugins
znippy run <handler> <cmd> …    # handler-specific subcommand

Build & test

# THE build door — reads .nornir/build-variants.toml, the one declaration.
cargo xtask build --suite ORIGINAL --list          # what the delivery contains
cargo xtask build --suite ORIGINAL                 # both atoms
cargo xtask build --suite ORIGINAL --atom znippy   # the lean archiver
cargo xtask build --suite ORIGINAL --atom znippy-full  # + iceberg, sign

cargo build --workspace
cargo test --workspace
cargo test -p znippy-tests --test build_variants   # the build-matrix guard
cargo test -p tests              # integration tests
cargo bench -p tests             # full benchmark suite
cargo bench -p <crate>           # per-crate bench

A `git worktree` of znippy cannot resolve the full workspace

Measured 2026-08-03, cargo 1.97.1. In any checkout that is not /home/rickard/git/znippy, every cargo command dies before it builds anything:

error: package collision in the lockfile: packages znippy-common v0.9.12
  (<the worktree>/znippy-common) and znippy-common v0.9.12
  (/home/rickard/git/znippy/znippy-common) are different, but only one can be
  written to lockfile unambiguously

The cause is one edge: xtask dev-depends on nornir at ../../nornir, and nornir path-depends on ../znippy/znippy-common — an absolute destination, so it re-enters the canonical checkout while the workspace member resolves inside the worktree. Two different directories, same name and version.

It is a resolve-time failure, so it is workspace-wide: -p <crate> and --exclude xtask do not avoid it. Three plausible dodges were tried and all fail:

attempt	result
sibling symlink farm so nornir's `../znippy` points back at the worktree	WORKS — see below. This row was wrong.
`paths = [...]` override in `.cargo/config.toml`	path overrides only replace registry sources, never a path dependency
bump `znippy-common`'s version so the two copies differ	works for that one crate, then collides on `znippy-compress`, and on down the list

The sibling symlink farm DOES work (re-measured 2026-08-30 on oden, lane AU, cargo 1.96). The row above said cargo canonicalises package roots so the symlink resolves back to the real checkout. It does not: cargo joins a path dependency against the directory of the manifest as it was opened, so a nornir opened through a symlink resolves ../znippy inside the farm.

The shape that works, exactly:

~/git/au-lane/znippy          ← the worktree, named `znippy`  (git worktree add)
~/git/au-lane/nornir  -> ~/git/nornir       (symlink)
~/git/au-lane/znippy-zoomies -> ~/git/znippy-zoomies
~/git/au-lane/<every other sibling a manifest reaches> -> ~/git/<same>

Two conditions, and both matter. The worktree directory must be named znippy, because that is the name nornir's ../znippy/znippy-common asks for — a worktree at ~/git/au-znippy collides exactly as documented above. And the farm needs every sibling any manifest in the graph reaches, not just nornir: draupnir was the second one missing here and cargo names it plainly (failed to read .../au-lane/draupnir/Cargo.toml), so add them until it stops.

MEASURED in that layout: cargo metadata resolves, cargo test --workspace builds including xtask (Compiling xtask v0.1.0 (~/git/au-lane/znippy/xtask)), and 636 tests ran. So xtask/examples/nornir-bench.rs is runnable from a worktree too, and none of [workspace] members has to be edited.

The old workaround — drop "xtask" from [workspace] members for the length of the run — still works and is still the answer if you are already in a badly-named worktree and do not want to move it. It is not a change to commit, because -p xtask is how the release gate invokes the bench suite.

The consequence for benchmarks: xtask/examples/nornir-bench.rs can only be run from /home/rickard/git/znippy, never from a worktree. A real fix means giving nornir a published znippy-common dependency instead of a path one; that is a nornir-side call, not a znippy one.

Release procedure

See WORKSPACE.md §Release gate protocol for the full gate.

Short form:

cargo bench -p <crate> — run benchmarks
Compare against bench_history.json — no metric drops >10% (same machine)
Append result to bench_history.json
Update design.md / README.md if architecture or benchmark numbers changed
git tag vX.Y.Z && git push --tags
cargo publish -p <crate>

Publish order when znippy-common changes:

znippy-common
→ znippy-compress, znippy-decompress
→ znippy-plugin-maven, znippy-plugin-python
→ znippy-cli → znippy        (znippy DEPENDS on znippy-cli, so cli goes first)

Decompressor publish order (when linflate changes):

linflate → lgz, ljar, lzip-parallel → znippy-zoomies → osm-katana

Intra-workspace dependencies are declared with both version and path (e.g. znippy-common = { version = "0.9.1", path = "../znippy-common" }). Cargo builds against the local path and publishes the version, so local dev and cargo publish both work with no [patch.crates-io] block. Publish bottom-up (per the order above) so each crate's deps are already on crates.io when the next is verified.

Bench history

bench_history.json at workspace root — one JSON line per run. Decompressor crates each have their own bench_history.json. See WORKSPACE.md §Benchmark column schema for required fields.