nordisk/Skidbladnir

archive — reasoning

Class: reasoning. Why the archive layer is the way it is, and what we rejected. The descriptive shape is in archive-design.md.

Why tar+zstd is the default, not znippy

The default engine has to be the one that always works, always offline, with no feature flags. tar + zstd are pure Rust, universally understood, and need nothing beyond the always-on dep set — so a cargo build with no features still packs and unpacks a bundle. znippy is better (dedup, delta, O(1) read), but it is an optimisation, and an optimisation must never be the thing standing between you and a working offline build. So znippy lives behind a feature and tar+zstd is the dependable fallback that the pipeline can always reach for.

Why a sidecar manifest instead of an in-archive index

The manifest is a separate <bundle>.manifest.json, not a header inside the tar. That keeps the manifest readable and verifiable without decompressing the archive — the far side can inspect what should be there, and the whole-file out-of-band SHA can be checked, before any extraction. For tar (a linear format with no seekable index) an in-archive manifest would have to be read by streaming anyway; a sidecar is simpler and engine-independent. znippy does additionally carry a seekable in-archive index — that is what its O(1) read needs — but the portable contract is the sidecar.

Why verify rejects extra entries, not just changed ones

A naive verify checks "is every manifest entry present and unchanged?". That misses the attack where a bundle gains an entry (a planted file) the manifest never listed. So verify_entries matches the sets both ways: an archive entry absent from the manifest is as bad as a manifest entry absent from the archive. This closes the extra-entry gap (regression-guarded by verify_rejects_extra_entry_absent_from_manifest).

Why the znippy engine consumes the real znippy crates (L5 dedup)

The znippy feature now links the REAL znippy stack (../znippy): znippy-compress::compress_dir seals the bundle and znippy-common (ZnippyArchive::open/ZnippyReader, get_file, decompress_archive, verify_archive_integrity) reads/verifies/unpacks it. A Skidbladnir bundle IS a genuine .znippy archive (Arrow-IPC sub-index, blake3-checked chunk blobs) — znippy CLI / DuckDB / Polars readable. There is ONE canonical archive format in the constellation, not a look-alike twin (L5).

The earlier design re-implemented a skidbladnir-native store (magic SKIDCAS1, a zstd-JSON index) to dodge znippy's arrow/rayon/CLI footprint — but that was a twin whose doc falsely claimed znippy-compatibility. It has been deleted. The dependency cost (arrow + OpenZL + the gatling pool) is acceptable because the znippy feature is off by default and never shipped: the lean offline core and the embedded client stay on the always-Rust TarZstd fallback and never compile znippy. History: znippy-core-consolidation-unsorted.md.

Why O(1) read matters enough to carry an index + footer

Reading one file out of a multi-gigabyte bundle by decompressing the whole thing is unacceptable when the target only needs one artifact (a single product's binary out of a whole-lake bundle). znippy's lookup sub-index + Arrow-IPC index let read_entry (znippy_common::get_file) locate the entry and pread only its chunk blobs (blake3-verified), never touching the rest. Selective read and the content-addressed delta both fall out of that same index.