nordisk/edda

znippy storage mechanics, and where gunnar plugs in

Written 2026-08-04 for external review. Every claim carries the file:line it was read from. Claims marked INFERRED were not executed and are the ones a reviewer should attack first.

Source trees read: znippy at release/0.9.13-prep (f871d5c), gunnar at 81c2ff0.

1. The archive on disk

znippy-common/src/meta_sink_append.rs:26-28:

[ blob_0 … blob_N ][ data sub-idx(es) ][ lookup sub-idx ][ trie ][ manifest ][ ZNPYMIDX ][off]
^0                 ^blob_end           └────── the whole metadata tail is rebuildable ──────┘

Two regions. Blobs grow forward from offset 0. The metadata tail is derived and is thrown away and rebuilt on every seal.

A file is stored as chunks. znippy-common/src/archive.rs:37-52:

struct ChunkInfo {
    blob_offset: u64,
    blob_size: u64,
    fdata_offset: u64,
    compressed: bool,
    checksum: [u8; 32],   // blake3 over the UNCOMPRESSED bytes (Design law 3)
}
struct FileEntry { uncompressed_size: u64, chunks: Vec<ChunkInfo> }

A file is a Vec<ChunkInfo>. There is no representation for "this entry is a base plus a delta". That is the structural reason delta rows need a new trait rather than a new field — see §7.

Reads are positioned I/O on a shared fd (archive.rs:56-60), so extract_file is concurrency-safe without locking, and only the Arrow index is loaded on open.

Compression: what is and is not compressed

Decided in two stages (znippy-common/src/precompressed.rs:97-104): skip_by_path during enumeration, before a byte is read; skip_by_bytes inside the compressor once a chunk's real bytes are in hand.

The module states the trap explicitly (precompressed.rs:73-76): "A rule of the shape 'an oid-keyed entry is already compressed' would get the second case backwards and silently store compressible data raw." Two things called "a git object" need opposite answers.

Consequence for gunnar: in the pack tier nothing is compressed on write and nothing is decompressed on read. znippy hands pack bytes back untouched.

2. The five reserved tables

Names are constants in znippy-common/src/index.rs:328-357. pkg_type discriminant for reserved entries is i8::MIN (index.rs:360), which is how ordinary list / decompress / iceberg readers skip them.

__gunnar_oid__ — index.rs:325-328

Raw (non-Arrow) stree payload over the first eight bytes of every object id, mapping to that object's first lookup row. The doc is emphatic: "Raw bytes, never an Arrow IPC stream — it must never be parsed as a sub-index."

__gunnar_graph__ — index.rs:330-335, schema at graph.rs:graph_schema

oid            Utf8      not null
parents        List<Utf8 nullable>  not null
tree           Utf8      nullable
committer_time Int64     nullable
generation     UInt32    not null

Generation numbers follow gix-commitgraph / git convention (graph.rs:9-13): a commit with no parent inside this archive has generation 1, otherwise 1 + max(generation of parents present). So ancestry becomes an integer comparison — if gen(a) <= gen(b) then b cannot be an ancestor of a.

A parent absent from the archive contributes nothing to the max (graph.rs:15-18), which the doc defends as the honest reading for a cold tier: a partially-repacked archive is a subgraph and its generation numbers are only ever compared within it.

__gunnar_reach__ — index.rs:337-340, schema at reach.rs:reach_schema

commit_oid  Utf8    not null
cardinality UInt64  not null
bitmap      Binary  not null   // serialized ROARING bitmap over object ordinals

Note the serialisation: roaring, not EWAH. Git's own .bitmap files are EWAH. Any converter between the two is unwritten.

__gunnar_refs__ — index.rs:342-351

One RecordBatch per push. The transaction boundary is the IPC frame boundary. pushlog.rs:13-22 gives the on-disk shape:

[ schema | RecordBatch | EOS ]  ← push 1, fsynced
[ schema | RecordBatch | EOS ]  ← push 2, fsynced
[ schema | RecordBatch          ← push 3, TORN by a crash

Repeating the schema per frame is what buys independent parseability: recovery is "read forward while frames parse", with no side-car, no length table, no journal. D18 removed redb on exactly this argument.

Measured overhead (pushlog.rs:27-31, refs schema, 4 refs per push, 2026-08-03): a frame is 2056 bytes — 448 schema (21.8%) + 1600 batch (77.8%, 400 B/ref). So schema repetition is the smaller half; the larger half is Arrow's 64-byte buffer alignment paid six times on a four-row batch. Merging frames into one stream recovers only ~21.8%. The real win is compaction, which amortises padding as well as schema.

__gunnar_secrets__ — index.rs:353-357

Same one-batch-per-push shape, already-encrypted material only. znippy never sees plaintext and ciphertext is never compressed. No consumer exists in gunnar today.

3. Which tables an archive actually gets

Not a feature flag — a constructor choice. plugins/native/znippy-plugin-git/src/sections.rs:19-31:

with_section — the refs and secrets push logs — works identically on both, "because it is the server's log and has nothing to do with how the objects are laid out". That is why both tiers route through one type instead of growing a second writer.

Oids are derived by hashing the stored bytes (sections.rs:4-8), so the indexed oid and the relative_path cannot drift — there is no second place an oid is spelled. Payloads are retained only for commits and trees; blob bytes are dropped once hashed (sections.rs:10-13).

A correction worth recording. gunnar-coldtier/src/lib.rs:431 justifies pack_tier emitting none of the three with "the pack's own .idx IS the oid index". That is sound for __gunnar_oid__ and wrong for the other two: .idx is an oid→offset map and says nothing about parents, committer time, generation or reachability. One table's justification was generalised to three.

4. Store

1. Caller hands GitIndexBuilder the canonical bytes of every object it is sealing.
2. Builder hashes each to derive its oid; keeps commit and tree payloads for the reachability closure, drops blob bytes.
3. into_reserved_builder (sections.rs:15-17) defers oid-index construction, because the oid index needs lookup row numbers, which only exist once the sink has sorted the lookup — so that step happens inside ArrowIpcSink::finish.
4. Sink writes blobs, then the derived tail: data sub-indexes, sorted lookup, fst trie, manifest, ZNPYMIDX footer.

5. Retrieve

ZnippyArchive::extract_file(relative_path) → look the entry up in the loaded Arrow index → pread each ChunkInfo → decompress the chunks marked compressed → concatenate. Parallelism is per chunk. extract_file_verified additionally checks the per-chunk blake3; the fast path does not (archive.rs:44-48).

Gunnar's read half is gunnar-coldtier/src/unfold.rs. It writes the pack and .idx back into objects/pack data first, index last, because gix-odb and git both discover a pack through its .idx — so the pack becomes visible exactly when the index lands and never before (unfold.rs:4-8).

unfold.rs:10-17 states why byte-identity is asserted rather than object equivalence: "A read-back that returned equivalent objects rather than the same bytes would have quietly lost that, and every functional test would still pass — the clone works, the objects are right, and P-001's 18.3× on wire bytes is gone."

6. GC — and the answer to "is the graph rewritten whole?"

Yes. Whole, every time. There are two distinct operations.

Re-seal / append — meta_sink_append.rs:29-44

open_existing reopens a sealed archive and:

1. reads the full manifest including reserved entries;
2. computes blob_end = the lowest index_offset over all sub-index and reserved sections — where blobs stop and the rebuildable tail begins;
3. recovers existing data rows from the sorted lookup sub-index (one cheap read, no per-sub-index rescan);
4. sets the cursor to blob_end, so new blob bytes overwrite the now-stale metadata tail;
5. on finish(), re-sorts the merged rows and re-emits lookup + trie + manifest + footer.

So blobs are append-only; the entire metadata tail — including every __gunnar_* section — is truncated and rebuilt. There is no incremental patch of the commit graph. Adding one commit rewrites the whole __gunnar_graph__ section.

A hazard a reviewer should check. meta_sink_append.rs:380 accumulates lookup columns from non-reserved sub-indexes only (if !is_reserved_module(...)). Reserved sections are deliberately excluded from row recovery. INFERRED: that means an append which does not re-supply __gunnar_graph__ / __gunnar_reach__ / __gunnar_oid__ drops them, since the old tail was truncated and nothing carries them forward. If true, every append must rebuild all three from the full object set, making append O(repo) rather than O(push) whenever those sections are present. I did not execute this. It is the single most important thing to verify.

Fold / compact

gunnar-coldtier has unfold.rs and a fold_and_clone test. Compaction rewrites the blob region and reclaims superseded blobs. pushlog.rs:36-38 names it as the real win for the push log — folding many small batches into one large one amortises both the schema repetition and the 64-byte alignment padding.

Nothing reclaims a superseded blob short of a fold. Re-seal only rewrites the tail.

7. Where gunnar plugs in

gunnar-store/src/object.rs:23 defines ObjectStore with four methods. Three answer what an object says, fully inflated. The fourth answers where an object lives:

fn packed(&self) -> Option<gunnar_pack::PackedObjects> { None }

fs.rs:434 implements it:

let store = self.odb.store();
Some(gunnar_pack::PackedObjects::new(move || {
    let mut handle = store.to_handle_arc();
    handle.prevent_pack_unload();
    Box::new(handle) as Box<dyn gix_pack::Find + Send>
}))

gix_pack::Find is a trait. gix does not require a filesystem — the filesystem is only how gix_odb::Store happens to implement it. Integrating znippy means implementing that trait over a znippy archive. That is the whole seam.

object.rs:53-79 records what the inflating path costs, measured against stock git 2.53.0 on the same repository: 108.70 MB against 5.93 MB, and 4.875 s of server CPU against 0.022 s — 18× bytes, 221× CPU. And it names the failure mode directly: "a znippy cold tier that stores inflated content per oid has no pack entry to copy either, which is P-005's whole objection to it."

So: znippy stores pack entries. packed() returns Some. Nothing inflates. The trait doc also explains why packed() lives on the trait rather than being wired in by hand — "an optimisation that has to be remembered is one that is silently absent in production while every test is green."

Two details worth copying rather than rediscovering, both in fs.rs:425-441: prevent_pack_unload() before handing out the handle, and the deliberate use of the default refresh mode rather than RefreshMode::Never, because a pack landing after the last scan would otherwise report every object missing, and a miss on that path is a failed clone.

Current state, 2026-08-04

mem and fs are the only backends. The znippy backend of §4 was never written. gunnar-coldtier (944 lines, 5 test files) depends on znippy and nothing depends on gunnar-coldtier — it is a workspace member that compiles and never links. In the serve path the five section names appear only in three doc comments (upload_pack.rs:96, graph.rs:17, graph.rs:1684).

8. Delta rows — why a trait is required

Rickard's proposal is iceberg-style historization: first row the original, following rows deltas. FileEntry cannot express it — a file is a Vec<ChunkInfo> and a chunk is a byte range, not a diff against another row.

ArchiveTypePlugin (plugin.rs:81) is the wrong seam: it is a metadata trait — extract_metadata, schema_fields, matches_path, supports_batch — deciding which Arrow columns a plugin contributes. It says nothing about how bytes are stored.

The right seam is a new trait at the entry-reconstruction layer, between FileEntry and ZnippyReader, with two implementations: chunk (today's behaviour, moved behind the trait unchanged) and delta (base row plus delta rows, reconstruct by applying the chain). ZnippyReader::extract_file keeps its signature, so no caller changes.

Whatever the delta implementation stores must carry an integrity property equivalent to the per-chunk blake3 over uncompressed bytes (Design law 3), and it must state what that is.

9. Questions for a reviewer

1. §6, the append hazard. Does an append actually drop reserved __gunnar_* sections unless the caller re-supplies them? If so, is append O(repo) whenever they are present? This is INFERRED from meta_sink_append.rs:380 and untested.
2. Delta rows on the pack tier. with_section works on both tiers, so a delta section should be able to ride the pack tier and leave P-001 entry-copy intact. Is there a reason it cannot?
3. Roaring vs EWAH. __gunnar_reach__ is roaring; git's .bitmap is EWAH. One structure with two serialisations, or one format plus a converter?
4. Generation numbers across a fold. Generations are relative to the objects present in the archive. A fold changes that set. Are stored generations still valid afterwards, or must they be recomputed?
5. __gunnar_oid__ is stree over the first 8 bytes of an oid. What is the collision behaviour at repository scale, and does the mapping to "first lookup row" handle a prefix shared by several objects?