Could the delta be the row? — a design review of gunnar's storage shape
Read 2026-08-04 against gunnar @ origin/master (3a1bc71), znippy @ master, nornir @ 5d34cd1. Every byte figure below was produced here, on oden, and every one of them is load-independent; the two wall-clock figures are stamped with the load they were taken at (loadavg 39–88 across the session, three other lanes building in the same tree).
Everything in .nornir/ was written by lanes solving one problem each. This document reads it whole and asks one question:
Apache Arrow IPC expertise on append? Or at least Iceberg is export and saving history — there are many similarities with git: immutable, etc. Could delta be saved directly per row in znippy, instead of splitting blobs into rows?
0. The answer, before the argument
Delta-per-row is the right shape and gunnar already has it — for refs. It is the wrong shape for objects, and not for the reason anybody expected.
The reason it is wrong for objects is not bytes. I costed it on a real pack and it is +2.01% — essentially free (§3). It is wrong because git's packfile is already a delta-per-row store with a better index, and because the one thing gunnar's read path is fastest at (P-001, wire ratio 1.000) is byte-copying out of a real packfile through gix. A row store cannot serve that copy without reimplementing gix_pack's output pipeline, and D21 says the things gunnar builds are the things gix would take — an Arrow-backed ODB is not one of them.
The reason it is right for refs is that a ref update already is a delta (name, old → new, push_seq), gunnar already writes it as one, and the fold throws the deltas away and seals a snapshot instead. That is the defect Rickard's Iceberg parallel actually points at, and it is the cheapest thing in this document to fix.
Three further findings came out of asking the question, and two of them are bigger than the question:
- znippy already contains the two-phase architecture gunnar is missing (
znippy-iceberg: a dynamic Iceberg/skade-backed metadata store, plusseal()that freezes it into the static.znippywith blobs copied verbatim). gunnar's cold tier uses only the static half, so every fold is a full rewrite. §5. - Everything in gunnar and znippy is the Arrow IPC stream format. The file format's footer costs 1 426 B on a 69.6 MB archive and buys O(1) seek to any batch — measured at 0.173 ms to reach the last of 49 batches. Nothing in either repo uses it. §6.
- gunnar's
appendpath re-seals its whole index on every append. That is exactly the O(N²) that skade pays for Iceberg'sfast_appendrewriting the manifest list every commit, rediscovered independently in a different format. §7.
And one correction to the corpus (LAW 7, and it moves in gunnar's favour): 64-byte buffer alignment is arrow-rs's default, not Arrow's. arrow-cpp and pyarrow have always written 8 — measured two ways here. The 64 → 8 change stomach.md §3.2 landed did not move gunnar to the format's minimum tolerance, it moved gunnar to what the C++ reference implementation does, and that retires the open worry stomach.md §5 records against it. §6.2.
1. What the current shape gets right
Worth stating plainly, because the rest of this document is criticism and the criticism is only worth reading if the baseline is respected.
P-005— hand the tier the pack, not the objects.FoldRequestcarriespack: &Pathand no object iterator, so "do not inflate git objects into an archive" is a type, not a comment somebody has to obey (crates/gunnar-store/src/repack.rs:143-156). This is the single best decision in the storage design and §3 below only strengthens it.SkipPolicy::already_compressedover the magic probe. Exact, free, unoverrulable, and it closes a measured 35.6 s / 2.6 GB trap (crates/gunnar-coldtier/src/lib.rs:26-35).- The frame boundary as the transaction boundary. No database wedged between two archive formats (D18). Recovery is "read forward while frames parse", with two positive checks — exactly one batch, and the EOS marker — because Arrow's
StreamReadertreats an unexpected EOF as a clean end of stream and the naive version silently swallowed torn pushes (znippy-plugin-git/src/pushlog.rs:110-125). That is a real trap correctly closed. - Absent beats empty.
pack_tierreturnsOk(Vec::new())rather than emitting empty__gunnar_graph__/__gunnar_reach__sections, because an empty section states in a section a consumer trusts that the archive holds no objects (znippy-plugin-git/src/sections.rs:244-249). Right call, and it should survive everything proposed here. - The fold's position in
repack()— strictly between "new pack durable" (sync_dir,repack.rs:543) and "first unlink" (:603), a hardErrand never a logged warning. Correct, and unchanged by anything below.
2. Where the shape fights the grain
Rickard's parallel is exact. Git objects are immutable and content-addressed; Iceberg is append-only with snapshot lineage; Arrow IPC is a framed stream where every message is self-describing. All three say never mutate, only append, keep the history addressable. gunnar obeys the first two clauses and drops the third.
F1 — the portable artifact carries state, not history
ZnippyColdTier::seal_with creates a fresh refs.log in a staging directory and does one RefLog::push of the entire ref snapshot (crates/gunnar-coldtier/src/lib.rs:183-190). FoldRequest.refs is documented as a snapshot (repack.rs:154). So a sealed archive answers where are the refs now and cannot answer how did they get there.
Meanwhile gunnar's hot path already has the history, in the right format, on disk, in two places:
- the live
refs.arrow— one self-contained IPC frame per transaction, with a bounded reflog window kept live after compaction (crates/gunnar-store/src/arrow_log.rs:52-56,CompactionPolicyat:216); refs-archive/— one complete, self-contained Arrow IPC stream per compaction, written to a scratch name and renamed, explicitly so that "a SQL question just reads two globs instead of one file" (arrow_log.rs:67-79,REFS_ARCHIVE_DIRat:185).
The fold reads neither. In fact no production code reads the archive at all: REFS_ARCHIVE_DIR's own doc comment says "Never read at open" (arrow_log.rs:184), it is written at arrow_log.rs:1515, and its only two other mentions in the whole workspace are inside #[cfg(test)] (arrow_log.rs:3625,3733, the module's test block starts at :2322). Zero hits outside arrow_log.rs — a grep of the same shape returns 3 for ArrowLogRefStore, so it is reading the tree.
So gunnar writes a complete, chronologically-sortable, self-describing history of every ref transaction it has ever performed, and nothing but its own tests opens it. That is a stronger version of F1 than the fold alone: the history is not merely absent from the archive, it is write-only everywhere.
An archive is the thing you copy off the machine, and it is the one artifact in gunnar that cannot answer a historical question. Git would call that shipping a checkout instead of a repository.
This is the delta-per-row question answered in the affirmative, on the one table where the delta already exists — and the answer today is that the fold deletes it.
F2 — the frame boundary is a durability mechanism used as a storage mechanism
pushlog.rs:1-56 states the invariant as "one RecordBatch per push", carried all the way into the sealed section. Repeating the schema per frame is what buys independent parseability, and on a live, fsynced, crash-recoverable log that is exactly right.
In a sealed section it is not right, and the cost is measurable. Same refs schema, one ref moved per push, pyarrow 24.0.0 (byte figures, load-independent):
| pushes | history, 1 batch/push | B/push | history, 1 batch total | B/push | what gunnar seals today |
| 10 | 6 152 | 615.2 | — | — | 1 512 |
| 100 | 57 992 | 579.9 | — | — | 1 512 |
| 1 000 | 576 392 | 576.4 | 92 128 | 92.1 | 1 512 |
| 10 000 | 5 760 392 | 576.0 | — | — | 1 512 |
| 100 000 | 57 600 392 | 576.0 | 9 125 880 | 91.3 | 1 512 |
Carrying 100 000 pushes of ref history costs 9.1 MB if the push boundary is a push_seq column instead of a batch boundary — and 57.6 MB if it stays a batch boundary. 6.3×, for a property (per-frame independent parseability) that only means anything while a writer can still be interrupted mid-frame. A sealed archive is by definition past that point: znippy has already fsynced it, renamed it into place and fsynced the parent directory (crates/gunnar-coldtier/src/lib.rs, the crash-safety module docs at :44-53).
This is precisely what Iceberg does and gunnar does not: the snapshot boundary is a sequence-number column inside a manifest, never a file boundary. Once committed, a snapshot needs no framing to prove it was whole.
(These figures are at 8-byte alignment — see §6.2, where a corpus claim had to be corrected: pyarrow/arrow-cpp's default output alignment is 8, not 64. So they are directly comparable to gunnar's post-§3.2 log, and the 6.3× is not an alignment artefact. It cannot be: alignment pads buffers, and the framing cost F2 is about is metadata, which alignment never touches — §6.2.)
F3 — content-addressing stops at the archive boundary
Git's whole trick is that a name is a hash of the content, so an object can be copied anywhere and verified anywhere. znippy's chunk rows carry a checksum: [u8; 32] per chunk (znippy-common/src/index.rs, ChunkLoc), and the pack carries its own 20/32-byte trailer. But the archive is named by a path and a version, not by a digest of what it holds, and FoldReport reports bytes_out and entries — not an identity. Two folds of the same repository at the same refs are not provably the same artifact.
Iceberg has the same gap (Part A of the research: files are identified by path, not by a hash of content) and pays for it with a catalog. Git does not need one. Since gunnar's fold input is a pack whose name is pack-<sha>.pack, an archive identity is nearly free and is not computed.
3. Delta-per-row for objects, costed on a real pack
The interesting part of the question is the object tier, so I measured it instead of arguing about it.
Corpus: nornir @ 5d34cd1, its single packfile — 38 099 387 B, 13 431 objects, 10 147 of them deltas (75.5%), chain lengths to 50. Method: git verify-pack -v for the per-object row (oid, kind, inflated_size, size_in_pack, offset, depth, base_oid); the pack sliced at [offset + header_len, offset + size_in_pack) to recover each object's exact deflate stream; the header length computed from the type nibble and the OFS_DELTA / REF_DELTA variant. Built as a pyarrow table and written as a real Arrow IPC stream. Script: .nornir/assets/deltarow.py, committed with this document so the claim and its control travel together.
Red-first control (LAW 2), because a slicing bug would make every figure below meaningless and an off-by-one produces plausible-looking garbage: 400 non-delta payloads were zlib-inflated and their lengths compared to verify-pack's own size column. 400 checked, 0 wrong. A second control falls out of the arithmetic: the entry slices account for 38 099 355 of 38 099 387 B — the pack minus exactly 32 B, which is its 12-byte header and 20-byte trailer. The slices are the pack.
| shape | bytes | × pack |
| the git pack (today, stored opaque in znippy) | 38 099 387 | 1.0000 |
| its entry headers, discarded by a row store (redundant with the columns) | 44 339 | 0.0012 |
arrow delta-per-row, one batch — oid, base_oid, kind, inflated_size, payload |
38 863 536 | 1.0201 |
| + zstd on the buffers | 38 417 872 | 1.0084 |
| pack + .idx, which is what actually sits in the archive | 38 476 527 | 1.0099 |
So delta-per-row is +2.01% against the pack and +1.01% against pack+idx — and the row form carries base_oid, kind and inflated_size, which the .idx does not. It is, to within one percent, byte-neutral. The bytes argument against it does not exist. That is the opposite of what the corpus assumes, and it needed measuring.
zstd on top buys 1.2% (1.0201 → 1.0084), which is the third independent confirmation of the same fact P-005 rests on: a git pack is already deflated, and recompressing it is CPU spent on nothing.
3.1 The granularity that does bite
A push adds a handful of objects. If each push were its own batch:
| objects per batch | batches | bytes | × pack | framing overhead |
| 13 431 (one batch) | 1 | 38 863 536 | 1.0201 | — |
| 4 096 | 4 | 38 864 712 | 1.0201 | +0.00% |
| 256 | 53 | 38 883 656 | 1.0206 | +0.05% |
| 32 | 420 | 39 028 032 | 1.0244 | +0.42% |
| 10 (a realistic push) | 1 344 | 39 394 024 | 1.0340 | +1.37% |
| 1 | 13 431 | 44 317 144 | 1.1632 | +14.03% |
One batch per object is +14% — the same disease as F2, at a granularity 1 300× finer. One batch per push is +1.37%, affordable. Below ~256 rows a batch, Arrow's framing is the dominant cost, and that number is worth remembering as a general rule for this codebase rather than a fact about packs.
3.2 The one thing delta-per-row genuinely buys: a projectable delta DAG
| bytes | B/object | |
metadata columns only — oid, base_oid, kind, inflated_size |
700 952 | 52.2 |
| the same, zstd | 424 872 | 31.6 |
the .idx that exists today |
377 140 | 28.1 |
The .idx is oid → offset and nothing else. Recovering "which object is a delta against which" from a packfile means touching every entry header, scattered through 38 MB. In row form it is a column projection: 1.86× the .idx uncompressed, 1.13× with zstd, and you can read it without touching a single payload byte.
"Without touching" is a real claim, not an aspiration, and it is worth pinning which layer makes it true. The format permits it — every buffer in a RecordBatch body is independently addressed by a Buffer{offset, length} descriptor. Whether an implementation exploits it is a different question, and the two diverge: arrow-rs's FileReaderBuilder takes a column projection and its own documentation says unselected columns are "neither read nor deserialized"; pyarrow's low-level RecordBatchFileReader.get_batch(i) has no projection parameter at all and materialises the whole batch (pushdown appears one layer up, in pyarrow.dataset.Scanner). gunnar is an arrow-rs consumer, so the projection is genuinely available to it — but note that this is a file- format reader. On the stream format there is no FileReaderBuilder, which is §6 again from a different direction.
That is a real capability gunnar does not have, and it is the one argument for the proposal. It is not enough. See §4.
4. Verdict on delta-per-row
For the object tier: NO — and write down why
Four reasons, in descending order of how hard they are to argue with.
P-001is bound to gix's pack abstraction, not merely to "a pack".write_pack_copytakespacked: &PackedObjectsand hands it togix_pack::data::output::entry::iter_from_countsinMode::PackCopyAndBaseObjects(crates/gunnar-pack/src/copy.rs:237,282). That mode resolves each object to aPackLocation— a byte range in a real packfile — and copies the entry verbatim. A row store has no such location. ServingP-001from Arrow rows means writing a new pack emitter over a new ODB backend: the largest single win in gunnar, rebuilt from scratch, to gain 1% of bytes and a column. This is the whole of the "no".
It is worth being precise about what does not break, because the obvious objection is wrong: content-addressing survives fine. The oid column is still the hash of the inflated object; the delta payload was never the thing hashed, in git either. And the deflate streams could be re-emitted verbatim — what cannot survive re-ordering is the OFS_DELTA header, which names its base by distance backwards, so a subset fetch must rewrite it (gunnar's own emitter already keeps the (offset, valid) table for exactly this, copy.rs:296-299). Converting to REF_DELTA costs ~17 B per delta ≈ +0.45% on the wire for this corpus. Recoverable. It is the plumbing, not the format, that says no.
-
A packfile is a delta-per-row store with a better index and 20 years of tuning. git's delta chains here reach depth 50 and cover 75.5% of objects. Re-expressing them as rows preserves the chains but not the tool that produces them:
git gc's window/depth search is the thing that makes the 4.35×-growth curve collapse 5.19× in 2.4 ms (crates/gunnar-coldtier/src/lib.rs:328-329). An append-only row store cannot repack itself from inside, whichstomach.md§4.1 already established for a different reason and which applies here verbatim. -
A reserved Arrow section is read whole.
read_reserved_section_bytesreturnsVec<u8>(znippy-common/src/index.rs:396). An__gunnar_objects__section is a 38 MB allocation per open, where today the pack is a data entry with per-chunkChunkLocrandom access. This is fixable — the manifest carriesindex_offset/index_sizeper sub-index, and §6's IPC file footer would give batch-level seek — but it is fixable work that must be done first, and it is not free. -
It stops being a git repository. Today the archive holds
pack-<id>.packandpack-<id>.idxunder their real names, hardlinked in. Unfold it andgitreads it. That is a disaster-recovery property nobody wrote down as a requirement and everybody would miss.
Retained from the "yes" side, and worth building separately: the delta DAG as a projectable column (§3.2) is genuinely useful, costs 1.13× the .idx with zstd, and does not require moving object payloads out of the pack. It is __gunnar_graph__'s neighbour: a small derived section over (oid, base_oid, depth), built from the .idx and the entry headers at fold time. That is the proposal that survives.
For the ref tier: YES, and it is nearly built
The row already is the delta. Do three things:
- R1 — seal the history, not the snapshot. Give
FoldRequestthe live log's path (or its recovered batches) instead of&[(String, Target)], and seal what is there. The bytes are already on disk in the right format; the fold currently re-derives a snapshot from them and discards the rest. - R2 — one batch,
push_seqas a column. In the sealed section only. 6.3× smaller than one-batch-per-push, measured (§F2). The live log keeps its per-frame framing, because that is what makes a torn push recoverable, and nothing about R2 touches it. This is the hot/cold split stated in bytes. - R3 — write the sealed section as an Arrow IPC file, not a stream. 1 426 B for a footer, O(1) seek to any batch (§6). That is what turns "the archive has history" into "the archive has time travel", which is the Iceberg half of Rickard's question.
Cost of all three: a signature change on FoldRequest, a writer swap, and a reader that already exists. No new format, no new crate, no measurement risk — the numbers above are the answer.
5. What znippy already has that gunnar is not using
This is the largest finding in the document and it is not about deltas.
znippy-iceberg (znippy/znippy-iceberg/src/lib.rs:1-47) is an ArchiveMetaSink that materialises znippy's metadata layer as real Apache Iceberg tables, one per sub-index, over skade's redb-backed RedbCatalog and a file:// warehouse. The blob pipeline is untouched: compressed blob bytes still live in the .znippy sidecar. And seal() (znippy-iceberg/src/seal.rs:1-28) freezes that dynamic store into the static v0.7 container — copying the blob bytes verbatim and appending only a metadata tail, with SealReport.blob_bytes_copied as the applied-output proof that nothing was rewritten.
That is, exactly:
| git | Iceberg | znippy | gunnar today | |
| mutable, appendable, queryable | loose objects + reflog | snapshots + manifest lists | IcebergSink + catalog.redb |
refs.arrow, refs-archive/ |
| immutable shipped artifact | packfile | compacted table | seal() → .znippy |
ArrowIpcSink → .znippy |
| how you get from one to the other | git gc |
expire_snapshots / compaction |
seal() |
repack() — a full rewrite, every time |
gunnar's cold tier uses only the right-hand column. It has no dynamic store, so ColdTier::fold is seal on every call — the whole archive re-written — and the append alternative was built to dodge that. §7 is what append actually costs.
gunnar should not build a dynamic metadata store. znippy has one, it is tested (znippy-iceberg/tests/lifecycle_seal.rs, round_trip.rs), and it is already on skade. The open question is whether the tokio + iceberg + parquet dependency it carries is admissible under D17's dependency budget on a no-OS appliance — which is a real question and a different one from "does it exist".
6. What Arrow IPC offers that gunnar is not using
Spec read at arrow.apache.org/docs/format/Columnar.html plus format/*.fbs; implementation facts from arrow-rs's arrow-ipc and from pyarrow 24.0.0 on this box. Where spec and implementation differ, both are stated — that distinction turned out to matter twice.
6.1 The file format's footer, for 1 426 bytes
Measured here, oden, 2026-08-04, pyarrow 24.0.0, loadavg 39:
| bytes | note | |
| 200 000 rows / 49 batches / 69.6 MB, IPC stream | 69 613 168 | what gunnar and znippy write, everywhere |
| the same as an IPC file | 69 614 594 | +1 426 B |
get_batch(48) on the file: 0.173 ms, straight through the footer's Block{offset, metaDataLength, bodyLength} table (format/File.fbs). The stream has no footer, so the same question is a linear scan.
Nothing in gunnar or znippy writes the file format. For 1 426 bytes on a 69.6 MB archive — 0.002% — a sealed section gains O(1) random access to any batch, and (§3.2) a reader that can push a column projection down. It is the cheapest capability in this review.
And it is exactly wrong for the live log, which is why this is a hot/cold split and not a global swap: the footer sits at the end and carries every block's offset, so appending one batch means rewriting it. There is no incremental-footer mechanism in the spec and no append API in either arrow-rs or pyarrow. Stream for the log, file for the seal.
6.2 The correction: 64-byte alignment is arrow-rs's default, not Arrow's
stomach.md §3.2 records the 64 → 8 change as "arrow-ipc defaults to a 64-byte buffer alignment ... 8 is the format's own stated minimum; 64 is a recommendation". The first clause is true only of arrow-rs. The spec says "allocate on aligned addresses (multiple of 8- or 64-bytes)" and "if possible, we suggest ... 64-byte" — a suggestion — while the hard requirement is 8, baked into the framing rule "the complete serialized message must be a multiple of 8 bytes so that messages can be relocated between streams."
And arrow-cpp/pyarrow's default write path already uses 8. Measured two independent ways on this box: a 1-row Int64 batch writes a body of 8 bytes (4 rows → 32), and a Utf8 offsets buffer of 20 logical bytes pads to 24, not 64.
So the 64 → 8 change did not move gunnar to the format's minimum tolerance — it moved gunnar to what the C++ reference implementation has always written. That is a stronger justification than the one recorded, and it retires the "whether 8-byte alignment costs anything on a machine that maps the log" worry in stomach.md §5: every pyarrow-produced Arrow file in the world is already 8-aligned. arrow-rs's IpcWriteOptions::try_new accepts exactly {8, 16, 32, 64} and rejects anything else, so 8 is a first-class value, not a tolerated edge.
6.3 Alignment is not the lever on small batches — batch size is
A 4-row × 6-column batch (3 Int64, 3 Utf8), byte-inspected on the wire: 648 B total = 424 B flatbuffer metadata + 8 B frame header + 216 B of actual buffers. 67% is framing. The metadata is dominated by the per-buffer descriptor — 15 buffers × 16 B (Buffer{offset, length}) = 240 B, plus 6 FieldNode × 16 B = 96 B.
Alignment pads buffers. It cannot touch those 424 bytes at all. So on a tiny batch there is a floor the 2.258× win does not reach, and the only lever that does is fewer, larger batches — which is F2's argument arrived at from the format side rather than from the measurement side. Two independent routes to the same conclusion.
6.4 Per-buffer compression, with an opt-out sentinel
BodyCompression{codec: LZ4_FRAME|ZSTD, method: BUFFER}. Granularity is the buffer, not the batch: "Buffers in the flat sequence of a message body must be compressed separately using the same codec", each prefixed with its uncompressed length — and a length of -1 means that buffer was left uncompressed.
That is exactly the shape §3 needs and gunnar has no use for today: a delta-per-row table could zstd the oid / base_oid / kind / size buffers (where §3.2 measured 1.86× → 1.13×, a 40% cut) and leave the already-deflated payload buffer at -1, instead of the all-or-nothing compression=zstd that buys 1.2% and burns CPU on deflate output. Not costed; the mechanism is there and nothing in either repo reaches for it.
6.5 Delta dictionaries
DictionaryBatch.isDelta — "its vector should be concatenated with those of any previous batches with the same id". Legal in both stream and file; the file format additionally forbids replacement dictionaries, so a long-lived dictionary in a sealed section must be delta-only. arrow-rs implements it (DictionaryHandling::Delta); pyarrow gates it behind emit_dictionary_deltas, default off "for maximum stream compatibility".
A ref log is thousands of batches over a few dozen distinct refnames — the exact shape this was built for — and nothing in gunnar dictionary-encodes anything. Flagged, not costed. Note the interaction with R3: if the sealed section becomes a file, replacement dictionaries stop being legal, so this has to be decided deliberately rather than inherited.
6.6 Concatenated streams need a reader per frame — confirmed twice
One open_stream over two independently written streams yields only the first stream's batches: the reader hits the EOS marker, reads meta_len == 0, and returns Ok(None). Silent truncation, not an error. (If the first stream had not been EOS-terminated, arrow-rs instead errors with "Expected a record batch, but found a schema" — a different failure for the same cause.)
Measured here directly, and independently confirmed against arrow-rs's StreamReader::maybe_next. znippy's scan_frames re-opening a StreamReader per frame (pushlog.rs:126) is therefore required, not defensive. Recorded because it looks like belt-and-braces and is exactly the kind of thing a simplification pass deletes.
7. What Iceberg offers that gunnar reinvents
From the spec (iceberg.apache.org/spec/) and from skade's live use of iceberg-rust 0.9.1 (skade/skade/src/write.rs, delete.rs, read.rs, skade/src/atomic.rs). skade is a genuine Iceberg engine — real manifests, real manifest lists, real v2 equality-delete files, real snapshot lineage — on an embedded redb catalog instead of REST/Hive.
What gunnar already has, independently and correctly:
- Commit by CAS on one pointer. Iceberg atomically swaps the table-metadata location in a catalog, conditioned on it still equalling the base it read (
skade/src/atomic.rs:300-336). gunnar's ref transaction isMustExistAndMatch(old)pre-checked before gix, because gix short-circuits same-value edits (crates/gunnar-store/src/fs.rs:651,668). Same primitive, arrived at from the git side. Nothing to import. - Staged write, fsync, rename, fsync parent. Both.
What gunnar reinvents, badly:
- The O(N²) append. Iceberg's
fast_appendregenerates the entire manifest list on every commit — carrying forward every prior manifest entry — which is O(N) per commit and O(N²) over N commits until compaction. skade documents this as its known weak spot. gunnar'sZnippyColdTier::appendgoes throughArrowIpcSinkAppend, which "re-sorts the merged rows and re-emits lookup + trie + manifest + footer" on everyfinish()(znippy-common/src/meta_sink_append.rs:37-45). The measured shape is identical: append CPU 0.0007 → 0.0016 → 0.0078 s at 10/100/1000 pushes — 4.9× for the 10× from 100 to 1000 (crates/gunnar-coldtier/src/lib.rs, theappenddoc block). Two formats, one bug. - Nothing prunes. An Iceberg manifest entry carries per-column
lower_bounds/upper_bounds/null_value_counts, and the manifest list carriesfield_summaryper manifest, so a reader skips whole files and whole manifests without opening them. gunnar's archive carries no statistics at all. The obvious one, and it is free at fold time:(min push_seq, max push_seq, min/max updated_ms)on the sealed ref section, so a reader answering "did anything change after T?" reads the manifest and stops. That is the manifest- listfield_summaryidea, in gunnar's terms, for four integers.
What Iceberg does not have, and this matters for the question: there is no row whose value is a binary delta against another row. Iceberg models a change as delete-tombstone + fresh insert — position deletes, equality deletes, or v3 deletion vectors — never a diff. Rickard's instinct that Iceberg and git are the same shape is right about immutability and snapshot lineage; it is wrong about deltas, and git is the only one of the three that has them. So delta-per-row is not an Iceberg idea to import. It is a git idea, and gunnar's object tier already stores it in git's own encoding.
8. Decisions this would change, by number
| decision | verdict |
D5 — objects: incoming packs → cold znippy, folded by repack(), no database |
Stands, reinforced. §3 makes the byte case for it stronger than the corpus does: a row store is +2.01% and loses P-001. Amend the reason recorded against it — it is not "rows would be bigger", it is "the copy path is gix's". |
D14 — repack() writes a new archive and atomically swaps, never mutates in place |
Stands as an invariant, but is doing double duty. It is currently also the reason there is no dynamic tier at all. §5: the swap is the seal, and a dynamic store in front of it does not violate D14. Worth restating so nobody reads D14 as forbidding §5. |
| D18 — refs and secrets are Arrow IPC streams; one storage technology | Amend. "Stream" is doing unintended work. The live log must be a stream (append without rewrite, per-frame recovery); a sealed section should be an Arrow IPC file (§6, +1 426 B, O(1) seek). One technology, two framings, and the difference is exactly the hot/cold boundary. |
D7 — all three git indexes in scope (__gunnar_oid__, __gunnar_graph__, __gunnar_reach__) |
Retire as written; replace. All three are Ok(Vec::new()) on gunnar's path and §4 argues two of them should stay that way (the .idx is the oid index; __gunnar_reach__ has no consumer — 0 roaring, 0 ewah rows anywhere in the graph, per write-read-cycle.md §3). Replace with one section that pays for itself: the delta DAG (oid, base_oid, depth), 1.13× the .idx with zstd, buildable from the .idx and entry headers without inflating anything. |
D12 — one .znippy per git repo |
Stands. Nothing here wants more archives. |
| D21 — anything upstreamable is built donation-shaped, gix conventions, first candidate delta computation | Stands, and it is the sharpest argument in §4. An Arrow-backed ODB is the opposite of donation-shaped. Note it explicitly against D21 as a rejected candidate. |
D26 — secrets sealed at rest, KEK outside anything FoldRequest reaches |
Unaffected, and the R1 change must not disturb it. R1 widens FoldRequest to carry ref history; the invariant "FoldRequest carries no secrets" (repack.rs:136-141) is load-bearing and must be restated in the same commit. |
| new | The wiring question every proposal here inherits: crates/gunnar-coldtier is excluded from the workspace (Cargo.toml:55), 0 znippy rows in Cargo.lock, and RepackOptions::cold_tier() has 0 call sites in any server crate. See §9. |
9. How any of this gets wired — the precondition nobody has discharged
Everything in §4's "yes" column and §5 is dead code until the cold tier is in gunnar's build. The exclusion is deliberate and the reason is good: znippy's codec cmake-builds C/C++ during cargo build, and cargo reads a path dependency's manifest at resolve time even with the feature off. So it is not a default = [] away.
The two shapes that actually work, neither of them free:
- Split the codec. A
znippy-common-only dependency that does not pull the cmake build, with the codec behind its own crate. Whether the reserved-section API can be reached without the codec is not determined here — it needs a resolve-graph experiment, not an opinion. - Keep the boundary, cross it at runtime.
ColdTieris already a trait object; a separate binary or adlopen'd plugin honours D17's dependency budget on the appliance and lets the fold be an out-of-process call. Costs the in-processFoldReportcontract.
Until one of those lands, total_cold_bytes: 0 and "no znippy cold tier is built" (crates/gunnar-cli/src/server/control_plane.rs:903) is the honest state, and the control plane says so out loud — which is the right behaviour and should not be quietened.
10. What I could not determine
- Whether the 1.13×-the-
.idxdelta-DAG section is worth its fold cost. The size is measured; the build cost (walking 13 431 entry headers at fold time) is not, and neither is the read-side win, because no consumer exists yet. §3.2 is a size result, not a benefit result. - Whether
znippy-icebergis admissible on a no-OS appliance. It carries tokio, iceberg, parquet and arrow-57 alongside gunnar's arrow-58. I read its module docs and its dependency declaration; I did not resolve the graph or measure the binary. D17's budget may simply refuse it, and that would be a legitimate end to §5. - Whether the reserved-section API can be reached without znippy's cmake codec. §9's first option. Not tested — it is a
cargo treeexperiment on a quiet box, and this one is at loadavg 39–88. - Everything about a second corpus. All §3 figures are one packfile from one repository.
norniris source-heavy, 75.5% delta, chains to 50. A binary-heavy or LFS-shaped repository would shift the payload/metadata ratio and could move the +2.01% materially. One corpus is one corpus. - The 0.173 ms
get_batchfigure is a single warm-page-cache read on a contended box. The 1 426 B footer cost is exact and load-independent; the latency is indicative only. - Nothing here was run against a live gunnar server. §3 and §6 are format measurements with pyarrow; §F2 is a format measurement. Every gunnar and znippy claim is a source read at the refs named at the top, verified by call site, not by execution — and
gunnar-coldtiercannot be executed from gunnar at all (§9). - I did not re-run any bench arm. The
stomach.mdandwrite-read-cycle.mdfigures quoted here (2.258× alignment, 4.35× growth, 5.19× reclaim, 0.0007 → 0.0078 s append CPU, wire ratio 1.000) are carried as written down, at the status their own documents give them. - arrow-rs's default
DictionaryHandlingvariant (delta vs resend) — both code paths were confirmed to exist; which one you get without asking was not pinned. §6.5 says "implemented", not "on". - Whether
arrow-rs'sFileReaderBuilderprojection actually avoids the I/O, as opposed to avoiding the deserialisation. Its own documentation says "neither read nor deserialized"; I did not verify that against a byte counter, and §3.2's benefit claim rests on it. - The 2.258× alignment win's split between buffer padding and everything else. §6.3 proves alignment cannot touch the 424 B of metadata in a 6-column batch, so on gunnar's 7-column single-row ref batch there is a floor the win does not reach — but I did not decompose gunnar's own 1536 → 680 B against that floor, and it would be worth doing before anyone budgets a further alignment change.
11. Reproducing §3, §F2 and §6
git -C <any-repo> verify-pack -v <any-repo>/.git/objects/pack/*.idx > vp.txt
python3 .nornir/assets/deltarow.py vp.txt <any-repo>/.git/objects/pack/pack-*.pack
Needs only pyarrow and git; it does not build or link gunnar, which is the point — it measures the format, and the format's answer does not depend on whether gunnar-coldtier is in the workspace (§9).
The script prints its own red-first control line first — 400 base-object payloads inflated, 0 wrong -> SLICES ARE REAL — and every byte figure after it is meaningless if that line does not read 0 wrong. It is deliberately the first thing on stdout rather than an assertion buried in the middle, because a measurement script that fails its control silently produces output shaped exactly like a clean result.