nordisk/edda

The stomach — gunnar's storage and retrieval engine

What skade and holger actually do, which of it transfers to gunnar, and what each change here measured on both sides.

Every number below is dated and stamped with the machine and its load. oden is 32 real cores / 512 GB / RTX 4090; /home/rickard/scratch is xfs on a two-NVMe RAID0 (md1), which turns out to matter (see §3.1).

1. What made skade best in class

Read 2026-08-04 against /home/rickard/git/skade at its then-current tree.

1.1 Two headline claims did not survive contact with the source

Worth recording before anything else, because both were in the brief:

• "~200 M rows/s written on 31 cores" is not in skade. Its write tops out at 44.4 M rows/s (all-core encode + commit, oden 32c tmpfs, 2026-07-24, skade/.nornir/README-full.md:420) or 92.7 M rows/s on uncompressed synthetic NVMe. The ~200 M figure is the read path (234.1 M rows/s at 9.5 cores busy; 263.7 M on the OSM corpus). "31 cores" comes from the zstd:9 row of a different table (skade/.nornir/s3-ingest-concurrency.md:148), where write is 17.0 M rows/s at 97% CPU. The two were never one measurement, and skade's own doc says the uncompressed write path is memory-bandwidth bound at about 4 cores — more cores do not help it.
• The 2.0 M ops/s catalog is a HAMT, not the static search tree. table_exists (skade/src/catalog.rs:1047) is one ArcSwap load into an imbl::HashMap with a foldhash hasher — lock-free, no redb on the hit path. The STree64 serves time-travel snapshot-id lookups only (skade/src/static_index.rs:41).

This is exactly the LAW 7 correction-downward the brief asks for, and it changes what is worth copying: skade's read discipline is the transferable part, not a write-side fan-out gunnar does not have the shape for.

1.2 What transfers

1.3 What does not transfer

• skade's whole write fan-out. Its unit of work is one Parquet file, handed to gatling_map_owned over a shared atomic cursor. gunnar's ref transaction is one row that must be durable before it is acknowledged; there is no N-way work to steal. The transferable idea is group commit (§4), not fan-out.
• Arrow buffer alignment is not addressed anywhere in skade. Zero mentions repo-wide — it never constructs Arrow buffers, it hands RecordBatches to ArrowWriter. So §3.2 is not a technique borrowed from skade; it is a lever skade's shape hides and gunnar's shape exposes.

2. What made holger best in class

2.1 What transfers

2.2 The measurement that matters most to gunnar's cold tier

znippy/.nornir/arrow-ipc-benches.md:80-120, threadripper-3975wx 32c, znippy 0.9.0, 2026-06-18 — an Arrow-indexed compressed archive with no cached reader:

Latency grows linearly with N because every cold call re-decodes the entire lookup sub-index. Extracting 1% of a 100 k-file archive runs at 55 files/s, and a prefix scan at 1% selectivity costs the same as at 100%.

This is the shape of the failure gunnar's cold tier would hit, and it is the strongest argument found in either repo for the append/re-seal threshold the brief says nobody has taken. Recorded as the next lever, not as a result.

2.3 Negative results holger paid for, worth not repeating

• SIMD everywhere: REJECTED by benchmark. A six-agent audit named "the build runs at SSE2, AVX2 paths are dead" as its P0 lever. Forcing inflate_fast into an AVX2 codegen context regressed every corpus (+5.3% text, +4.0% rle, +1.1% binary on revert) — real DEFLATE decode is Huffman-bit-decode bound, not match-copy bound. Reverted. The actual win was a build flag.
• A "mini" client that was 2 MB bigger — mini = ["gui"], and a Cargo feature cannot subtract a dependency. Measured 1 050 137 592 B against 1 048 139 472 B.
• WebAuthn linked but never dispatched: +14 crates and the entire native openssl stack for zero reachable functionality. Withdrawn, and the guard strengthened on removal.
• No cold-cache measurement exists anywhere in holger. Every bench warms up first. For an engine with a cold tier that is the blind spot not to inherit.

3. gunnar, measured

3.1 The baseline, and where the time goes

4 000 single-ref transactions against a never-compacting ArrowLogRefStore on md1, refs_write_stats().fsync_nanos separating the sync from the encode. oden, 2026-08-04, loadavg 17.4:

push_per_s=16 252   us_per_push=61.5   fsync_share=0.829   bytes_per_push=1536.1
open: 6 145 984 B / 4 001 batches replayed in 13.9–20.8 ms

fsync is 83% of a durable ref transaction. Everything else is 10.6 µs.

3.2 LANDED — the Arrow write alignment, 64 → 8

arrow-ipc defaults to a 64-byte buffer alignment. A ref transaction is a one-row batch of seven columns, so the padding is an order of magnitude larger than the data. 8 is the Arrow IPC format's own stated minimum; 64 is a recommendation for readers that want to hand a buffer to a SIMD kernel, and nothing maps this log.

Four interleaved A/B pairs from two binaries differing only in write_options(), loadavg 2.6–7.0 throughout. Medians:

Every pair favoured 8 on every metric. The byte figures are exact and load-independent.

The throughput win is a consequence of the size win, not a separate effect. fsync's share of the wall clock is 0.830–0.838 in both arms — the sync did not get relatively cheaper, there were 2.26× fewer bytes for it to force through the XFS log.

Correctness:

• Full gunnar-store suite green (60 lib + 34 integration).
• Red-proven both ways. Reverting write_options to the default fails one_ref_move_is_framed_at_the_8_byte_alignment with "one ref move framed to 1536 bytes" — the exact 64-byte figure, so the guard is reading the mechanism and not a coincidence. It also fails a_64_byte_aligned_log_is_still_read_and_can_be_appended_to on its anti-vacuity assertion — "the planted frame (1536 B) is not larger than the appended one (1536 B), so this log is NOT mixed-alignment" — which is the guard catching itself becoming blind, the failure mode LAW 2 is about.
• No migration. Every IPC message carries its own bodyLength, so a stream written at 64 and appended to at 8 is read message by message. Asserted against real mixed bytes produced by arrow's own 64-byte writer, not a hand-rolled imitation.
• The external-reader promise holds. pyarrow 24.0.0 reads the 8-aligned stream: 51 batches, 51 rows, 7 columns, values correct.

3.3 WITHDRAWN — fsync → fdatasync

The obvious first move against an 83% fsync share, and it measured nothing: 16 252 → 16 449 durable pushes/s, inside the run-to-run spread.

Why: /home/rickard/scratch is xfs. An append changes the file size, and xfs commits that size update through its log whether or not the timestamp metadata goes with it. The part fdatasync skips — mtime/atime — was never what was being paid for. On ext4 the answer could differ; on this box it does not. Reverted, and recorded in Backing::sync's neighbourhood so it is not tried a third time.

3.4 LANDED — group commit

The 83% fsync share is not a disk problem, it is a serialisation problem, and the store's own refs_lock_wait_nanos counter said so before a line was written. Same fixture, transactions spread over N threads each moving its own ref so no CAS ever conflicts:

(One diagnostic run, loadavg 33.95 — not a row from the A/B set below, which was measured separately with matched loads. It is quoted for its shape, not its absolute values.)

Throughput flat across a 32× change in concurrency (1.14×) while lock wait rises linearly with the writer count is the signature of a serialised barrier. All 32 pushers were queueing for one fsync at a time.

The fix is leader/follower group commit. Log::append now writes the frame and returns a commit sequence; transaction releases the writer mutex and only then takes the barrier, on a dup of the log's descriptor held apart from it (SyncGate). The leader reads written_seq before syncing and publishes it after, so its one fsync is credited with every commit that had already reached the kernel — exactly the set fsync is guaranteed to cover. Everyone at or below that watermark returns without syncing.

Five interleaved A/B pairs, loadavg matched within each pair (11–26 across the set). Medians:

Lock wait at 32 writers: 8.5–10.5 s → 0.27–0.66 s, about 19×. The single-writer row is the one that had to not move, and did not.

The flaky guard this caught, which is the part worth reading

The first grouping test raced eight threads and asserted syncs < transactions. It passed on md1 and failed on /tmp with 320 syncs for 320 transactions — because /tmp is tmpfs on oden, an fsync there costs nothing, and no pusher ever lives long enough for another to queue behind it. A guard whose verdict depends on which filesystem TMPDIR names is not a guard, and it was its own anti-vacuity assertion that said so; the durability half was perfectly green throughout.

Replaced with a constructed queue: the test holds the gate shut, waits until every writer's frame has provably reached the kernel (written_seq), and only then opens it. Exactly one leader, on any device.

Red-proven, all three ways:

• dropping the gate.barrier call fails concurrent_pushers_are_all_still_durable with "durable 448 < written 223488: a push was acknowledged that a power cut would lose", and the pre-existing single-threaded durability test with it;
• putting the sync back inside the writer mutex fails a_queue_of_pushers_costs_exactly_one_sync with "only 1 of 8 frames reached the kernel in 30 s; the writers are not getting past the writer mutex, which means the sync is still inside it" — the deadline assertion diagnosing the exact pre-change shape;
• inverting the watermark check fails a_follower_that_never_syncs_is_still_covered_by_the_leaders_sync, which drives the follower path deterministically with no threads at all.

The one property that changed, stated plainly. The acknowledgement barrier is untouched: transaction still does not return Ok until its bytes are durable. What changed is reader visibility — the in-memory ref map is published under the writer mutex, which is now released before the sync, so for the duration of one barrier a concurrent reader can observe a ref move whose bytes are not yet on the platter. Nobody has been promised anything, but a fetch served inside that window and followed by a power cut would have handed out a ref the log no longer names. That is the ordinary read-uncommitted trade a group-committing log makes; it is written into SyncGate's own docs rather than left to be discovered.

The bug this change introduced, found by reading every call site

Splitting Log::append into stage-then-barrier silently un-synced the secret path. ArrowLogSecretStore::write_locked called append and discarded the returned sequence, so a freshly minted SSH host key was acknowledged with its bytes still in the page cache.

Every test stayed green. The ones that reopen the store read the secret back happily, because a page-cache read cannot tell a synced file from an unsynced one. Only a power cut could, and there is no test for that. It was found by grepping the three append/commit call sites and asking what each did with the value, not by anything that went red.

Fixed by appending and barriering under the writer mutex — deliberately not the group commit the ref log does, because a secret is written when a host key is minted or rotated, which is rare and never concurrent, so there is nothing to group and releasing the lock first would only widen the window in which a rotated secret is still readable. a_secret_is_durable_before_the_write_returns now guards it, red-proven at durable 288 < written 728.

The general lesson, and it is the one worth carrying: when a function that used to do two things is split into two, the compiler will not tell you which callers only took the first half — the second half's return value was () before and is () again if you drop it. Grep every call site; the tests cannot.

The hazard that has no test and is prevented by construction instead: a compaction renames a replacement over the live log and reopens it, so the gate's dup would point at the old, now-unlinked inode and every later barrier would flush a file nothing reads — a silent loss of durability no test of the contents could see. Log::rewrite is the single writer for both swap paths (compact_refs and the snapshot path both go through it), and it rebinds the gate there, so there is one place this can be got wrong rather than two.

3.4b WITHDRAWN — reusing the frame buffer, and the ceiling that says why

Group commit moves the bottleneck: with the fsync outside the writer mutex, the mutex is the serial section, and everything in it — validate, encode, frame, lseek, write — is what every other pusher queues behind. That makes the arithmetic worth writing down, because it bounds the whole approach:

perf stat, 16 writers × 2 048 pushes, oden 2026-08-04: user 0.347 s / 32 768 pushes = 10.6 µs per push, sys 0.731 s = 22.3 µs. Against an fsync of roughly 50 µs, about 50 / 10.6 ≈ 4.7 writers fit into one sync round, so ~4.7× is the ceiling group commit can reach on this box, and §3.4 measures 2.59–2.79×. Halving the serial section roughly doubles that ceiling — which is the case for attacking it, and the reason this was tried at all.

Log::append called Vec::new() for every frame. Profile (perf record --call-graph=dwarf, same workload): _int_malloc 2.41% + cfree 2.01% = 4.4% of total CPU, i.e. ~14% of the user side; arrow_ipc::writer::write_array_data 1.36%; <Mutex>::lock_contended 3.53%.

Reusing the buffer across appends (mem::take, clear, hand it back — zero allocations after the first push) measured, three interleaved A/B pairs on instructions retired, which the brief prefers on a loaded box:

0.3%, with the two ranges completely overlapping. Withdrawn. glibc's tcache makes a repeated same-size ~680-byte alloc/free nearly free, and the 4.4% in the profile is the Arrow builders inside build_ref_batch plus the IPC encoder's own internal allocations — not the one frame Vec that was easy to reach.

And a caveat on the metric itself. Instructions retired is not clean for this workload: an earlier perf stat of the identical pre-change binary reported 98 534 instructions/push against ~84 800 here, a 16% run-to-run spread. The cause is visible in the profile — lock_contended spins burn instructions, and how much spinning happens depends on the box's load. So on a contended benchmark, instructions retired inherits the load-sensitivity it is usually chosen to escape. It still separated these two arms (both measured under the same spread, interleaved), but it would not detect a change smaller than a few per cent, and it should not be quoted as an absolute for this arm.

3.5 The two rows that persist, and what makes them falsifiable

Both are nornir-bench arms in bench/examples/nornir-bench.rs, appended at the tail because three agents are editing that file and a tail arm is the cheapest thing to rebase. Fixtures on md1, never /tmp — /tmp is tmpfs on oden and a durability arm measured there is measuring a machine gunnar will never run on.

gunnar.reflog_append — one durable transaction, end to end. bytes_per_push 680.1, fsync_pct 82.1, push_per_sec 15 353, open_ms 17.04 for a 4 001-batch log. Its red is a byte ceiling, not a wall clock: the two alignments are 2.26× apart in bytes and only 1.12× apart in time, so a time-based guard could not see the regression it exists to catch.

gunnar.reflog_group_commit — the same total work at 1 and at 16 writers. group_commit_x 2.22–3.54 over two runs at loadavg 5.5 and 20.2 (push_per_sec_1 13 072, push_per_sec_16 46 277 on the better one), lock_wait_ratio_x 1.70. The range is quoted rather than the best figure: this arm is contended-box sensitive by construction, since what it measures is threads queueing. Both runs are far above the 1.3 threshold and further still from the 1.08 the pre-change library produces.

Red-proven against the real pre-change library, not by argument: rebuilt on the commit before group commit, the arm fails with

16 concurrent writers reached only 1.08× the durable throughput of one. Group commit is not working: every pusher is paying its own fsync inside the writer mutex.

1.08 is the number the message predicts. No single-threaded arm in gunnar's suite can see that regression, which is the whole reason this row exists.

The red is conditional on lock_wait_ratio_x, and that is not a hedge — it is the same tmpfs lesson as §3.4's flaky guard, applied to a bench arm before it could bite. A flat ratio has two causes that need opposite answers: the pushers queued and it bought nothing (group commit broken), or nobody ever queued (fsync is free on this filesystem, and the run says nothing either way). Only the first is a failure. The second emits grouping_observable = 0 and a loud stderr line, so it can never be read as a passing verdict.

4. Next levers, in order, none of them taken yet

1. The cold-tier re-seal threshold — RETIRED, and the reason saves a lane. The proposal is "append on every push, re-seal at 100 pushes or ratio ≥ 2.0". Two facts kill it as stated:

   • ZnippyColdTier::append has no production caller. Grepped 2026-08-04: the only callers are crates/gunnar-coldtier/tests/append.rs and that crate's own bench. repack() reaches the tier through ColdTier::fold, which is seal — the full re-seal — every single time. So the threshold would be policy for a caller that does not exist.
   • A cold-tier re-seal could not reclaim the measured growth anyway. The 4.35× came from packs accumulating, not from dead payload: the crate's own measurement records rows_replaced at 0 throughout and the per-push append flat at 1556/1558/1557 B. The 5.19×-in-2.4 ms reclaim was sealing the consolidated pack git gc had produced. Consolidating packs is a git repack, which an archive append cannot do from inside itself.

So the threshold does not belong on the cold tier; it belongs on when repack() runs, and repack() already re-seals on every run. The open question is a repack schedule, which is a different question from the one the growth curve was read as asking. §2.2's 1 177 → 3.5 lookups/s cliff still stands as the reason archive size matters — it just is not reachable by this lever.

1. Streaming the fold. ZnippyColdTier::append does std::fs::read(pack) into a Vec before handing it on — the whole pack in RAM, which is the floor under the brief's measured 1.61 bytes of peak per pack byte. Chunking that one call is the obvious first cut.

2. Frame-buffer reuse in Log::append. Tried and withdrawn — see §3.4b. 0.3% on instructions retired, ranges overlapping.

3. Shrinking the serial section for real. §3.4b's arithmetic is the case: the group-commit ceiling is roughly fsync ÷ serial_section ≈ 4.7× today, so halving 10.6 µs roughly doubles it. The frame Vec was not where the time is. The candidates that remain, in descending size, are the Arrow builders rebuilt per call in build_ref_batch (a one-row batch constructs seven builders and their buffers), and the write syscall itself, which is 22.3 µs of kernel time inside the mutex.

The write is the bigger prize and the more dangerous change: reserving the byte range under the mutex and pwrite-ing outside it would take the syscall out of the serial section entirely. It is not a drop-in. Frames would then complete out of order, so the durable watermark would have to become the contiguous prefix rather than the maximum, or a barrier could acknowledge frame N+1 across a hole where frame N has not landed. Worth doing, worth doing carefully, and worth doing on a quiet box.

1. O_APPEND instead of seek(End) — one lseek per push inside the same serial section. Named for completeness; an lseek is tens of nanoseconds against a 10.6 µs section, so it is bookkeeping, not a lever.

5. What could not be determined here

• Nothing about __gunnar_oid__ could be measured from gunnar. Its three defects are real and read correctly from the source, but gunnar-coldtier is excluded from the workspace, so znippy is not in gunnar's resolved graph at all and no gunnar build, test or bench reaches that code. Fixing it is a znippy change, benched in znippy.
• The 938 durable pushes/s figure is not this measurement. That is the full push path with a real pack; §3.1's 16 252/s is the ref log alone. The two are not comparable and the ratio between them is where the rest of the push actually goes — unmeasured here.
• Nothing cold. Every number in §3 is warm, which is precisely the blind spot §2.3 says not to inherit. No page cache was dropped between runs.
• Whether group commit helps a real push. §3.4 measures ref transactions in isolation, with N threads doing nothing but move refs. A real concurrent push also indexes a pack and writes objects, so the fraction of it that is this barrier is smaller and the end-to-end ratio will be lower than 2.79×. What is measured is that the barrier stopped being the serialisation point; what is not measured is what fraction of a whole push it was.
• How much of the 32-writer spread is the box. The five pairs ran at loadavg 11–26 with three other agents building in the same tree. Medians are quoted and each pair is internally matched, but a quiet box would give a tighter interval and possibly a different number.
• Whether 8-byte alignment costs anything on a machine that maps the log. Nothing in gunnar does, and pyarrow reads it, but a hypothetical consumer that mmaps the file and hands buffers to a SIMD kernel would now get 8-byte-aligned ones. No such consumer was found; none was looked for outside this tree.