nordisk/holger

znippy format-version pin — design

Class: design · Function: znippy-format-pin (used by storage, serve). Guide: znippy-format-pin-guide.md · Beginner: znippy-format-pin-for-idiots.md. Whole-system view: design.md · storage floor: storage-design.md.

The defect

A znippy archive records the on-disk format version it was written with. Holger is a registry: archives arrive from an agent across the airgap, from a USB stick, from a pull-through cache, from a client PUT. Until now holger parsed whatever arrived. artifactory-parity-design.md recorded the consequence and rated it P0:

format version is not enforced — a newer-written archive crashes an older reader and takes the server down.

An artifact registry that any client can kill by uploading is not a registry. Boot-time containment already existed (instantiate_backends disables a repo whose backend fails — graceful_backend_init.rs); what was missing was a version verdict, and the request-time path where a writable repo re-opens its archive on every read and can therefore meet a new archive long after boot.

The chosen design

holger pins the format it reads, itself, in front of the reader.

traits::znippy_format (in traits/ because every znippy-* repository backend already depends on it — one implementation, L5):

The rules, exactly:

• Newer than the pin → refused. So is a recorded version that is not a number.
• Equal, older, or absent → reads exactly as before. The gate is additive; it refuses nothing that works today. (Absent matters: znippy's append writer records no version — see Known gap.)
• Undeterminable → not this gate's verdict. Unreadable file, corrupt manifest, or a panic inside the index reader all return "undetermined" and the normal open path reports its own error, unchanged. The gate never converts an existing failure mode into a new one.
• The version read runs inside catch_unwind, because path is attacker-controlled and the whole point is that hostile bytes do not escape this function.

Why holger pins it rather than trusting znippy

znippy-common grew an internal check_format_version on its read path, and it is welcome. holger must not depend on it: holger pins znippy by semver and releases against published crates (the local [patch.crates-io] is a dev convenience), so the reader linked into a given holger binary may predate that check entirely. The pin is a statement about what holger can read and must keep refusing regardless of which znippy is linked. Defence in depth on the first path hostile input reaches.

Where it is wired (znippy-rust)

with_archive / with_archive_and_url   → ensure_supported_format() BEFORE ZnippyArchive::open
                                         → repo disabled at boot, every other repo lives

try_open_reader()  (writable dev-mode, re-opens per read)
                                       → FormatGate::check() → Err propagates to get_file()

handle_http2_request()  (first thing, before any route)
                                       → FormatGate::check() → 503 + the reason

open_reader() survives unchanged as the Option-returning form for listing/probe callers whose signatures cannot carry an error; the read paths that answer a request use try_open_reader() so the refusal reaches the client. Nothing was removed (L2).

Cost

recorded_format_version reads the manifest and one sub-index's Arrow schema — not the archive, not the index rows, no blob bytes. On the per-request re-open path FormatGate memoises the verdict against (len, mtime), so the steady state is a single stat; a file that changed — including one swapped under a running server — is re-gated.

Known gap this exposed (znippy side, reported not fixed)

znippy's append writer (meta_sink_append::{create_archive, append_files} — the path a writable holger repo and cargo publish take) seals its sub-indexes with the bare lookup_schema(), which carries no schema metadata, so archives holger writes itself record no format version at all. Read-side that is safe (absent = read as before), but it means a holger-written archive cannot be version-checked by a future reader either. The fix belongs in znippy's write path (lookup_schema() → a schema carrying build_arrow_metadata_for_config's version key); logged in workspace_nordisk/holger_agents_problems.md. format_version_gate.rs asserts the current behaviour so the day it changes, the premise is re-examined rather than silently drifting.

Evidence (RED-when-broken)

server/repository/znippy-rust/tests/format_version_gate.rs and server/lib/tests/graceful_backend_init.rs craft a real archive — packed by znippy-compress — and then edit only its declared version. The edit is length-preserving: the flatbuffers value slot is [len:u32-le][ascii][NUL][pad → 8] followed by [21]["znippy_format_version"], and a 1-, 2- or 3-character value occupies the same 8-byte slot, so "3" → "39" shifts no offset in the manifest or footer. The archive therefore differs from a working one in exactly one thing, its declared format version — which is what makes these version tests rather than corruption tests.

Verified red: raising MAX_SUPPORTED_ZNIPPY_FORMAT to a value that admits everything turns both gate tests red.

Bumping the pin

MAX_SUPPORTED_ZNIPPY_FORMAT moves only together with a reader that actually understands the new layout. the_gate_admits_the_current_format_and_stays_out_of_the_way fails loudly if the linked znippy starts writing a version above the pin, which is the signal that the bump (and the reader work behind it) is due.