nordisk/Skidbladnir

skidbladnir ZnippyArchive → real znippy-core (one canonical archive format)

TRIAGED 2026-07-27 → history. This is finished work, not backlog. The shipped shape is documented in archive-design.md §ZnippyArchive — feature znippy (the real-znippy adapter, the on-disk layout, selective read, delta/apply, the traversal guard, and the four round-trip tests). Nothing here is outstanding; the plan below is kept only as the record of how it was decided. Do not action it.

DONE (2026-07-13). Shipped by consuming the EXISTING znippy crates directly — no new znippy-core crate was needed. Under the znippy feature, Skidbladnir's ZnippyArchive is now a thin adapter over znippy-compress (compress_dir seal)

• znippy-common (ZnippyArchive::open/ZnippyReader list, get_file selective read, decompress_archive unpack, verify_archive_integrity). The bespoke SKIDCAS1 zstd-JSON store (magic SKIDCAS1/SKIDCFTR, ZnippyIndex/write_bundle/ read_footer/load_index/BlobSource) was deleted. Public Archive API/shape unchanged; the TarZstd default is untouched. Bundles are now genuine .znippy (Arrow-IPC index, blake3 chunks) — znippy CLI / DuckDB / Polars readable.

Original plan (a new lean znippy-core crate) preserved below as history. ───────────────────────────────────────────────────────────────────────────── skidbladnir hand-rolled a JSON-indexed archive that imitates znippy and whose doc falsely claimed it mirrors znippy's Arrow-IPC layout. Replace it with the REAL format, so bundles are znippy/DuckDB/Polars-readable.

Good news: clean extraction

• skidbladnir's ZnippyArchive is behind the znippy feature (OFF by default) — never shipped, zero back-compat debt. Default stays TarZstd.
• The real format's index code (znippy/znippy-common/src/index.rs) is already lean: deps = arrow 58 + blake3 + fst + serde — no rayon, no znippy-cli, no compress engine. So a znippy-core extraction is additive, not a churn.

The real format (the truth)

[blobs…][Arrow-IPC index][8-byte LE index_offset footer] (v0.6); v0.7 adds sub-indexes + a manifest + ZNPYMIDX magic + reserved modules (lookup/trie/sign). Index schema (index.rs:39–70): relative_path·chunk_seq·fdata_offset·compressed·uncompressed_size·blob_offset·blob_size·checksum(BLAKE3). ZNIPPY_FORMAT_VERSION=3 in schema metadata; check_format_version already guards it.

Plan

Phase 1 — new znippy-core crate (znippy workspace, additive): re-home the lean index/footer surface: compose_index_schema, build_metadata_batch, build_arrow_metadata_for_config, check_format_version, ManifestEntry + reserved-module consts, read_footer/write_footer, BlobMeta/ChunkMeta. Deps: arrow/anyhow/serde/log/once_cell only. No compress, no CLI, no rayon. (Real znippy-common can later re-export from it.) Phase 2 — skidbladnir consumes it: add znippy-core dep behind the existing znippy feature; implement RealZnippyArchive: Archive that writes the REAL Arrow-IPC format (pack/read_entry O(1) seek/delta/apply), delete the hand-rolled ZnippyArchive (archive.rs:354–765). Callers (machine.rs:413 ImageCache, fetch.rs verify_entries) see no change (Archive trait seam). Fix the false doc claims. Phase 3 (optional, holger): holger's parallel .tar.zst engine (agent/lib/src/operations.rs:254–506) can gain a transfer_to_znippy() export (holger already deps znippy-compress) — tar.zst stays for non-nordisk interop.

Gate (compat, not perf)

Round-trip: skidbladnir packs a tree → the real znippy CLI lists/extracts it byte-identically AND the Arrow index parses (DuckDB SELECT * FROM <archive>). O(1) selective read stays O(1). No back-compat to preserve (feature was off). Source: read-only scout 2026-06-28.