nordisk/holger

Storage — design

Class: design · Function: storage (used by serve, proxy, embed). The whole-system view is design.md.

Two storage floors

• znippy archives — immutable, content-addressed (blake3 per chunk, verified on read, not optional), one archive per repository version. Arrow-IPC framed multi-index (znippy-common): reserved __znippy_lookup__ + __znippy_trie__ give O(log n) single-file random access, so ZnippyReader::extract_file(path) pulls one file without decompressing the whole archive. Built internet-side by the agent, carried across the airgap, served read-only.
• filesystem — a {repo}/{path} tree for the writable dev backends: file-rust (Cargo sparse) and file-oci (OCI v2 CAS).

StorageEndpoint (server/lib/src/storage/mod.rs) names a location (znippy | rocksdb) and a path; repos reference it by name in their ron_in / ron_out.

file-rust — the writable Cargo sparse index

RustRepoFile (server/repository/file-rust/src/lib.rs) serves a real Cargo sparse registry out of a {ns}/{name}/{version} disk tree. RustRepoFile::new(name, base_dir, base_url); three handle_http2_request routes:

Three facts that are load-bearing:

• The download URL is config, not hardcoded. base_url = the ron_base_url knob (the only thing config.json advertises as dl); a stock cargo forms every download URL from it. There is no 127.0.0.1 fallback — the public host must be set so clients behind the airgap reach the right host.
• The per-crate index is text/plain, deliberately. Cargo's sparse protocol serves the index as newline-delimited JSON with text/plain (not application/json); RustRepoFile matches that exactly, one JSON object per line.
• Checksums are real and streamed. The cksum per line is the live SHA-256 of the stored bytes. all_versions_of streams the file through the hasher in 64 KiB chunks (crate_sha256_hex_streaming) — never resident whole — and memoizes path → (FileStamp{len,mtime}, sha256) in a Mutex-guarded map. Stored artifacts are immutable (atomic write-then-rename in put), so a cached digest whose (len,mtime) stamp matches is reused; a changed stamp forces a re-hash. No stored versions ⇒ 404.

file-oci (server/repository/file-oci) is the writable OCI Distribution v2 backend (blob/manifest upload + get, digest verify) over a disk CAS. Both file-* are the only writable backends; put on any znippy-* backend errors.

Integrity & concurrency

• Read integrity is znippy's mandatory per-chunk blake3 verify; the serve hot path is unchanged by adding writable backends.
• Write-corruption safety: the writable-znippy put holds a per-backend write Mutex across the whole check-then-mutate section, so two concurrent puts (including the proxy calling put during a cache-miss fetch) can't race the single .znippy into corruption.
• Path-traversal: check_component on the file-* paths rejects empty / . / .. / / / \ / NUL components before they touch disk.

Because writable file-backed repos live on a single ReadWriteOnce volume, the deployment is single-replica by design; scale-out needs RWX storage (not yet supported) — back up the PVC instead.