nordisk/znippy

Package-aware plugins — design

Class: design · siblings: plugins-guide.md · iceberg-design.md · signing-design.md. Strict: this file states only the shape we run — the trait, the registry, the built-ins. No alternatives, no why.

f0 — the plugin system (the whole)

znippy is a content-addressed, seekable compressor: an Arrow-IPC index over zstd/lz4 chunks with blob dedup. A plugin teaches that index about one packaging ecosystem (rust, python, maven, media, …). One archive carries one package type, selected at write time by --format.

A plugin is any type implementing znippy-common/src/plugin.rs trait ZnippyPlugin (:82). The writer composes the archive's Arrow schema as base columns + the union of every registered plugin's fields, laid out as a DenseUnion; each plugin fills only its own columns. Two rules make the system hold:

Reuse map (what the plugin subsystem uses): plugin.rs depends only on arrow (schema/arrays) and the archive's BatchItem borrow. It is used by the writer (schema composition), by every native plugin crate, and by wasm_loader.rs (.wasm plugins). The whole workspace is tokio-free here — plugins do no async I/O.

f1 — the ZnippyPlugin trait

pub trait ZnippyPlugin (znippy-common/src/plugin.rs:82). Six members:

fn meta(&self) -> HandlerMeta;                                 // :93  DISCOVERY record
fn run_command(&self, cmd: &str, args: &[String]) -> Result;   // :107 handler subcommands
fn extract_metadata(&self, path, data: &[u8]) -> RecordFields; // :112 per-file
fn schema_fields(&self) -> Vec<Field>;                         // :117 the Arrow columns it adds
fn matches_path(&self, path) -> bool;                          // :121 which files it claims
fn extract_batch(&self, items: &[BatchItem]) -> …;             // :131 optional zero-copy, once

• meta() returns a HandlerMeta: name, aliases, ecosystem, extensions, subcommand API. It is the only discovery surface — the CLI never hard-codes handler names.
• matches_path decides file ownership; schema_fields declares the columns the plugin contributes to the composed schema.
• extract_metadata runs per file; extract_batch is an optional zero-copy pass that borrows &[BatchItem] and runs once, before compression, for handlers that benefit from seeing the whole set at once.

f2 — the PluginRegistry and schema composition

PluginRegistry (plugin.rs) holds the registered handlers and is the seam the writer uses to build the final schema and fill the union:

Each plugin fills only the union branch it declared; the reader recovers per-ecosystem columns through typed views (f4).

f3 — the built-in register, and the two places a handler's source lives

Built-ins register at one point: znippy-cli/src/handlers.rs::builtin_handlers(). The CLI autodiscovers the register through meta() — there is no second name table to keep in sync. Both tiers below register through that one function and are indistinguishable to znippy handlers and to --format; the tier decides only where the source sits.

P-5 — the tier rule

A handler's source lives in znippy-common/src/plugins/ if and only if it needs nothing beyond std, arrow and znippy-common itself. The moment it needs a third-party parser, a decompressor, a network fetch, its own crate-type or its own release cadence, it is a znippy-plugin-* crate under plugins/native/. If it is an ABI shim over one of those, built for wasm32-unknown-unknown, it is a crate under plugins/wasm/. There is no fourth place.

The reason is one-directional: znippy-common sits under every other crate in the workspace, so a dependency added there is paid by all of them, on every target including wasm32. Neither shape is a staging area for the other — a handler is born into the tier its dependencies dictate, and a znippy-plugin-* crate at src/ root is the normal Rust layout, not a misplacement.

Why tier A did not move into plugins/native/ (2026-08-05). The obvious tidy-up — one crate per handler, all under one directory — is blocked by a package cycle, not by taste. A znippy-plugin-cargo crate needs znippy_common::plugin::ArchiveTypePlugin, so it must depend on znippy-common; keeping the old path alive would need znippy-common to depend back on it, which cargo rejects. Without the re-export, the published path znippy_common::plugins::cargo_native::CargoPlugin disappears — and it has a live out-of-workspace consumer: holger/traits/src/lib.rs names six of the handlers in non-test code, plus four holger repository crates in tests. The move would strand holger until six new crates were published. The dependency-weight argument that would justify it does not exist either: MEASURED 2026-08-05 on oden, gunnar's cargo xtask guard reports 281 crates / 267 third-party both before and after the plugin move, and znippy appears zero times in that graph — gunnar-coldtier is excluded from gunnar's workspace on purpose. The six handlers import only std and arrow, so they add no crate to anyone. What they cost is compile time in every znippy-common consumer, which is real but is not what the guard measures.

Tier A — in-common handlers (znippy-common/src/plugins/native/)

Promoted, real handlers: cargo_native.rs (1), npm_native.rs (6), rpm_native.rs (8), deb_native.rs (9), gem_native.rs (11), conda_native.rs (14). Alongside them skeletons.rs holds the stubs — filename-only coords/extract_metadata, promoted in place once someone writes real parsing (go 4, nuget 5, elf 7, flatpak 10, docker 12, helm 13, snap 15, appimage 16, composer 17, hex 18 …). mod.rs is the module list.

These files import only std, arrow and crate:: — zero third-party crates. That is P-5 holding. A .wasm plugin loads through wasm/wasm_loader.rs (WasmPlugin, behind feature wasm-plugins).

The tier-A source is itself split native/ + wasm/ (2026-08-05). Same rule as the repo-root plugins/ directory, one tier down: no plugin source sits loose beside plugins/mod.rs. The two root directories hold crates, the two here hold modules — which tier a handler is born into is still decided by its dependencies, and the move changed no handler code (nine pure git mv renames). mod.rs pub uses each module back to its old path, because znippy_common::plugins::<x>_native::<X>Plugin is API of a published crate: holger/traits/src/lib.rs names six of them in non-test code, and several holger/server/repository/* crates take znippy-common by version, so they follow the published path and cannot follow a directory rename. znippy-common/tests/plugin_source_layout.rs guards both halves from outside the crate.

Tier B — separate znippy-plugin-* crates, all under plugins/native/

All six are rlib compiled-in host handlers. maven and python additionally declare cdylib, but nothing loads that host .so; the WASM artifact comes from tier W below, not from these crates.

Where the ArchiveTypePlugin impl sits is not uniform, and does not need to be. maven/python/media put it in a private native.rs re-exported from lib.rs (pub use native::NativeMavenPlugin); git/skidbladnir/rust-toolchain implement it in lib.rs directly. member.rs is not the counterpart of native.rs — it is a pure path→MemberKind classifier with no trait impl in it at all.

Publishing consequence. Tier-B crates are published separately and are depended on by version, from outside this workspace: holger's znippy-maven / znippy-python server crates name znippy-plugin-maven and znippy-plugin-python from crates.io. Folding a tier-B crate into znippy-common breaks those consumers. The 2026-08-05 move into plugins/native/ changed directories only — every package name and every published version is untouched, so crates.io is unaffected; what did move is the path = "…" string in every sibling that names one of these crates (holger, nornir, skade/bench, dwarves, dwarves/bench, gunnar-coldtier).

Tier W — the wasm shims, under plugins/wasm/

A tier-W crate is defined by what it does not contain. znippy-plugin-maven-wasm declares one dependency — znippy-plugin-maven with default features — and reuses its pom.rs, its linflate JAR scan and its row.rs field writer verbatim. row.rs is the single writer of the column set: the native schema_fields() builds Arrow fields from MAVEN_SCHEMA, the wasm plugin_schema serializes the same constant, and both call maven_fields() — so the two surfaces cannot drift by construction (LAW 5).

What does not cross to wasm32-unknown-unknown, and why:

The JSON ABI is lossy in one specific way and the code accounts for it rather than papering over it: parse_json_to_row produces only ExtensionValue::Str and knows no null, so fields_to_json omits a null column instead of emitting null (which would land as the literal string "null"), and it replaces , : " in a value — that parser splits on them before it strips quotes, so an unescaped separator would shear a value into the wrong column.

tests/maven_native_wasm_agree.rs is the guard: it loads the real .wasm through the real WasmPlugin and compares rows against NativeMavenPlugin over a bare .pom, a stored JAR, a DEFLATE JAR (the case where the two decompressors genuinely differ), a classified artifact and a JAR with no pom, plus schema_fields() and matches_path(). It fails — never skips — when the .wasm is missing or older than any maven source.

f4 — the read side (typed views)

Because each ecosystem's columns live in a known union branch, the reader exposes a typed view per ecosystem — maven_view, rust_view, python_view — that round-trips that ecosystem's columns out of the DenseUnion (integration: plugin_read_api_test.rs).

f5 — bomb defense

A plugin's parsers are hardened: garbage or oversized inputs fall back to filename parsing and never panic (*_bomb_falls_back_to_filename tests). Metadata quality degrades to the filename; the archive still writes.

znippy-plugin-git — the indexer's memory model (lane P, 2026-08-30)

MEASURED 2026-08-29 21:10 UTC on the gunnar appliance (16 GB, 4 vCPU, no swap, PID 1 = gunnar-server): a push --mirror of torvalds/linux (8.12 GiB pack, 13 948 244 objects, 11 215 872 deltas) was acked, then the post-ack indexer grew past the box and the kernel panicked (System is deadlocked on memory). The table in resolve.rs's module doc is the per-object rule; this is the whole worker, after the fix:

The watchdog (memguard.rs): before every pack and every 4096 objects inside one, MemAvailable off /proc/meminfo; under the floor (ZNIPPY_GIT_MEM_BUDGET_PCT, 20 %) the resolver drops every held base, flushes the sink and the row batch, and waits (250 ms re-check; one WARN index.paused, one INFO index.resumed; cancel ends it). It parks the indexer worker and nothing else — it is taken before the absorb gate.

The two checkpoints. Absorber::absorb takes one between packs, deliberately before the absorb gate, so a parked worker holds no gate, no row and no base; Resolver::checkpoint takes one every 4096 objects inside a pack. Both call the same pause_until_ok: it parks while MemAvailable is under the budget and returns when MemAvailable reaches budget + budget / 8. That eighth is hysteresis — without it an 8 GB box flapped fourteen pause/resume pairs in a minute — and it is why a pause that trips on a shortfall of a fraction of a percent needs 12.5 % of headroom to end. The flush runs once as the pause opens and again every 10 s while it holds, because a reader can build a table after the first flush.

What a pause drops. Absorber::spill clears the readers' scanned pending-pack rows and calls ExplodedTable::drop_caches (the Arrow oid index and the batch cache); Resolver::checkpoint additionally drops every held base, flushes the sink and writes the row batch. index.spilled's scanned_rows_dropped counts only the first of those, in rows, so it reads 0 whenever no reader ever had to answer out of a pending pack — a push with no concurrent clone or fetch drops nothing there. It says nothing about the exploded caches, which go unconditionally and uncounted, and clear() releases the map's Arc, not a clone a reader is still holding.

The livelock. What a parked worker still holds is the part that cannot be dropped: the walk, ofs_deps/ref_deps, oid_to_offset and the remaining/stuck vectors are all O(pack) and all live until the pack ends. When those — and not the held bases — are what the box is short of, the flush frees nothing, MemAvailable never climbs by budget / 8, and pause_until_ok, which has no timeout and no give-up, parks until the store is cancelled. The floor still does its job: the box does not panic, the control plane and the read path keep answering. Bounding the walk-sized structures is what lane Q's lazy explode has to reach; more spilling cannot get there, because there is nothing left to spill.

Nobody waits on the indexer. Reads never take the gate: the rows already batched in are in the index, and an oid the worker has not reached is resolved read-only from the pending pack on the reader's thread (Absorber::scan_pending, same resolver, bounded bases, no sink). reachable absorbs for itself if the gate is free and otherwise refuses by name with the progress numbers. AccountIndexer::drop cancels before it joins.

The lazy read path lane Q builds on: resolve::Resolver::materialize(offset) — one object's content from a pack and an offset with only its chain inflated — and Absorber::scan_pending — one oid (rows + content) from the pending packs.

Measured (resolve::tests::a_bounded_base_set_…, release): facett pack, 9668 objects: unbounded peak 731 051 B; budget 1 B → peak 287 000 B (one base at a time), 839 evictions, 7028 re-inflations, identical rows; budget 861 000 B → 0 re-inflations. Plugin lib suite 220/220.

MEASURED 2026-08-30 on the oden KVM rig (lane P), gunnar v1.1.3-180-g01ce6c58, 4 vCPU, 4×75 GiB raid0 xfs /data, no swap — the same push --mirror of torvalds/linux at two RAM sizes. The ack path holds: the 8.12 GiB pack (8 723 679 817 B, 13 948 244 objects, 2874 refs) was spooled to disk over the 64 MiB threshold, mapped, and ingested in 269 s on both boxes with no packfile written. Neither box panicked — the 2026-08-29 kernel deadlock is gone at 8 GB and at 16 GB. Both then parked and neither index finished. At 16 GB the worker reached 78 % (11.0M/13.9M) before index.paused at MemAvailable 3 286 511 616 B against a budget of 3 361 581 460 B — a shortfall of 75 MB, 2.2 %; at 8 GB it reached 22 % (3.2M/13.9M) and paused 1.65 MB under a 1 672 962 040 B budget, 0.1 %. Neither resumed: 0 index.resumed in 31m44s and 1h45m41s of parking, index.spilled every 10 s throughout (190 and 627 of them) with scanned_rows_dropped=0 on every one — no reader had scanned a pending pack, so the spill had nothing to give, and at 8 GB MemAvailable had fallen to 1 276 284 928 B by the time the pause ended, further from the resume line than when it opened. Both pauses ended only on cancel. The consequence is the push: the connectivity check that follows the ingest never completed, no reference moved, and the client sat unacked for 2h14m30s (16 GB) and 2h12m24s (8 GB) against stock git's ~11 min for the same mirror. What kept answering while the worker was parked: the control plane instantly, and ls-remote in 4 ms.

The exploded tier's cost, measured the same run. repo.purged reports what one mirror had written when it was cancelled: 195 704 627 966 B (182.3 GiB) at 78 % indexed, and 46 641 441 342 B (43.4 GiB) at 22 % — 22.4× and 5.4× the pack, both projecting to ~24–29× at 100 %, so linux.git needs roughly 200–235 GiB of /data for an 8.12 GiB pack. Delete reclaims all of it: bytes_after=0 in 604 ms and ~2 s, /data back to 33 MiB on the band.

znippy-plugin-git — the ACK path, and the two hours it used to take (lane Z, 2026-08-30)

Lane P2 measured an appliance that survived a kernel-sized push and never reported it. This is where that two hours went, and what the ordering is now.

What receive_pack.checking was doing

The connectivity check reads objects by oid. This arm acks a push's bytes before its oids exist — that is §13.9, and it is what lands an 8.12 GiB mirror in 4m29s — so the check's very first graph.peel of a pushed tip missed the exploded table, missed the objects index, and fell into Absorber::resolved's third arm, scan_pending. That arm calls scanned_pack, which runs the full resolver over the whole pending pack on the connection thread and then throws the rows away:

07:43:49  git.receive_pack.checking   commands=2874
09:20:21  index.scanned  "a reader scanned a pending pack for itself:
                          13948244 objects in 5792442 ms"

1h36m32s at 16 GB, 1h44m14s at 8 GB, for one answer. It is not a per-ref walk — S-062 already batched that — and it does not wait on the indexer. It is worse than waiting: it is the same work, twice, at the same time, two resolvers over one pack on four vCPUs, each holding its own PackWalk::entries and sorted oid table. That second copy is also why the watchdog paused, and why index.spilled then reported scanned_rows_dropped=0 on all 190 and 627 of them: the scan the spill would have dropped was the one still being built, and a half-built scan is not in the scanned map yet. After the scan finally landed at 09:20 the check still had 13.9 M objects to walk one at a time; the delete cancelled it 32 min later.

The ordering, and it is git's

GitServe::index_now() (git-storage-trait, default Ok(0)) — "make every object this store has accepted addressable by oid, now". receive_pack::apply calls it once, before the frontier is built, and everything after it is a point lookup. That is index-pack and then check_connected, which is what stock git has always done.

GitStore::index_now → Absorber::index_pending_now: take the absorb gate, absorb every pending pack, keep the rows. If the account worker already holds the gate this blocks on it and then finds is_absorbed set — whoever got there first did the work exactly once.

The read path is deliberately NOT this

Nobody waits on the indexer still holds for every other caller, and a_read_before_the_drain_is_served_from_the_pack_and_never_waits is its red test: a clone, an ls-remote and a control status must be answered while the drain is parked, and they still are, through scan_pending, without touching the gate. The first version of this change had every reader index and that test turned the suite from 50 s into a hang.

The connectivity check is the one caller for which waiting is the right trade, because it cannot answer without the pack resolved and the only question is once or twice.

IN_ABSORB (thread-local, InAbsorb guard) is what makes the re-entry safe: the resolver asks this same store for delta bases while the gate is held, and that call must absorb the other pending packs inline (lane U's thin-push-on-a-pending-pack case) and must never take a non-reentrant mutex it already holds.

The watchdog has three ways out now, not one

A watchdog that parks for two hours and never resumes is worse than one that says I cannot index this pack on this box: the pack's bytes stay durable, nothing partial is indexed, and the operator gets a sentence naming what was available, what the floor is, how far the pack got and the three things they can do about it.

What an operator can see while it runs

progress::ProgressTicker logs index.progress every 10 s for as long as a pack is being absorbed, carrying peak_rss and reinflations — which rode only on index.finished, so lane P2's two boxes, which never finished a pack, printed neither, in exactly the failure the two numbers exist for. It is a thread and not a hook in the resolve loop because a hook does not run while the resolver is parked, which is when the numbers matter most.

Read peak_rss with care. It is VmHWM, which counts resident file-backed pages, and this arm maps the whole pack — so on the linux mirror it reads ~11–13 G on a 16 GB box while MemAvailable is still 11–12 G. mem_available on the same row is the honest number and the one the watchdog acts on.

MEASURED — the same mirror, the same rig, before and after

oden KVM, 4 vCPU, 4×75 GiB raid0 xfs /data, no swap, 16 GB. push --mirror of torvalds/linux: 8 723 679 817 B pack, 13 948 244 objects, 2874 refs. Before = lane P2, gunnar v1.1.3-180-g01ce6c58. After = lane Z, v1.1.3-207-g83ce408.

Where the 66 minutes are, stated plainly: 53m46s of it is the single-threaded resolve. Resolver::run's fixpoint is a for over remaining on one thread — 13.9 M objects at ~4 200/s on a four-vCPU box — and it is now the whole cost of an ack. The connectivity walk that was blamed for the two hours is 8 minutes. Stock git acks the same mirror in ~11 min and does it by resolving the pack on every core. That is the next thing to fix and it is a gatling/znippy-zoomies job, not another ordering change: each entry has at most one base, so the dependents form a forest and every root's subtree is an independent chain.

Why there were 0 pauses. Not because the watchdog was lucky — because the second resolver is gone. Two O(pack) copies of PackWalk::entries + oid table is what put lane P2's 16 GB box 75 MB under its floor; one copy finishes with 4.95 GiB to spare. The livelock fixes above are still load-bearing (the 8 GB box has a 1.67 GB floor and far less room), and they are covered by unit tests carrying the rig's own numbers rather than by this run.

znippy-plugin-git — the resolver is a FOREST, and the forest is the parallelism (lane AM, 2026-08-31)

Lane Z left one number on the table: of the 66m14s an ack took, 52.6 minutes was Resolver::run's fixpoint on ONE thread — 13 948 244 objects at ~4200/s on a four-vCPU box, with three vCPUs idle. Stock git acks the same mirror in ~11 minutes and does it on every core.

The shape

Every pack entry has at most one delta base. One parent per node is the definition of a forest, so the dependents resolve.rs has always collected (ofs_deps, ref_deps) are not an incidental index — they are the children lists, and two roots' subtrees share no object and no ordering constraint. [crate::forest::DeltaForest] states that: parent, CSR child lists, and the two root sets, built from PackWalk::entries and nothing else (LAW 5). One round of the fixpoint is now a gatling_run fan-out over that round's roots (NEVER rayon — LAW 3), and each unit is a depth-first walk of one subtree.

What the shape buys besides the cores

The fixpoint held a base from the moment it resolved until its last dependent's position in the pack, a distance the format does not bound — which is what lane Q's byte ceiling and Belady eviction exist to cap. A depth-first walk holds the path from the root to the current node and nothing else, and on a path the shallowest frame is by construction the one needed last, so Belady needs no next_use table to know what to drop. MEASURED, the kernel pack: peak held bases 627 670 217 B (lane Z) → 87 000 000 B at one worker, 145 400 000 B at four. The ceiling is one atomic across every worker — N workers do not get N budgets.

Determinism: keyed by ROOT, and the measurement that says why

The stream out is fixed and worker-count independent: roots ascending, and inside a root the depth-first sequence the ascending children lists determine. the_same_pack_resolves_to_the_same_bytes_at_1_2_and_4_workers compares the whole tape — rows, batch boundaries, sink calls, in sequence — byte for byte, and was seen RED by letting the drain take whichever root was at the front of the buffer instead of the cursor's.

Keying it by entry index is the obvious choice and it is unusable. MEASURED with examples/forest_stats on torvalds/linux:

The subtrees are tiny — ideal parallel work — but a five-object subtree routinely spans five million entries, because pack-objects puts a base beside its first dependent and the rest wherever they fall. An entry-ordered cursor is therefore pinned by any subtree in flight whose span reaches the top of the pack. MEASURED, kernel pack, 4 cores: 1 worker 1454.7 s, 2 workers 1572.9 s — slower, 1.24 cores of CPU, 286 686 stalls at the lookahead ceiling. Keyed by root the cursor moves at the rate roots finish, and gatling hands roots out in ascending order, so it follows the workers instead of pinning them: stalls 0.

The memory budget of the parallel resolver, declared

DeltaForest replaces HashMap<u64, Vec<usize>> (2.7 M keys, 2.7 M heap Vecs — ~230 MB on the kernel pack) and the fixpoint's remaining/stuck index vectors (111 MB each) with three Vec<u32> (156.5 MB) and a bitset (1.7 MB where the round's index list was 56 MB). The parallel resolver's own footprint is smaller than the fixpoint's, and the numbers below say so.

MEASURED — what it does, and where it does not fit

Same rig, same mirror, same clock as lane Z's run.sh: oden KVM, 4 vCPU, 4x75 GiB raid0 xfs /data, no swap, 16 GB, gunnar v1.1.3-207-g83ce408 — lane Z's own measured build, so the only thing that differs is znippy's resolver.

The honest verdict: the resolve is an order of magnitude faster and the 16 GB appliance does not have the headroom to let it finish a kernel-sized mirror. Lane Z's own run ended with 4.95 GiB available against a 3.36 GiB floor — 1.6 GiB of margin — and the parallel resolver spends it. It is not the resolver's own structures (peak RSS is below lane Z's): what changes is that a depth-first walk of a subtree whose index span is five million entries touches the 8.13 GiB mapped pack all over, at four cursors at once, so far more of the mapping is resident at any moment and MemAvailable — which counts only half of page cache — falls accordingly. The fixpoint walked the pack in entry order and had perfect locality.

When the floor is breached the box says so and stops, which is lane Z's law working: index.gave_up ... refused by name, the pack's bytes durable, nothing partial indexed. One give-up now stops the whole round rather than one per worker — before that fix a two-worker run burned a second, full 300 s deadline after the first had already given up.

znippy-plugin-git — what a push costs on DISK (lane Q, 2026-08-30)

Lane P's table above is what the indexer holds in memory. This is what the same push leaves on the filesystem, and it is the question Rickard asked when the appliance filled up: "we cannot have one repo take all 300 GB disk."

The rule

At push time the store writes the pack verbatim, the index, and the graph, and nothing else. The index is oid → (offset, len, type, base, uncompressed size); the graph is the commit and tag payloads, because those are what graph/reach/a tag peel read and none of them may stall. Trees and blobs — 90 % of the objects and essentially all of the bytes — are not written at push time at all. The first reader of one inflates that one object's delta chain out of the mmapped pack (resolve::materialize_at, d inflates and d−1 delta applications) and hands it to a write-back thread, which puts it in a budgeted cache for the next reader.

The four files

What an open store costs in descriptors — 3, and why the number matters

It is not a housekeeping number. gunnar's server divides its RLIMIT_NOFILE by exactly it to decide how many repositories it may keep open at once (gunnar_server::limits::open_repo_budget = min((soft − 256) / fds_per_repo, MemAvailable × 20 % / 8 MiB)), so the count is the divisor of an appliance's concurrent-repository ceiling. Lane O measured the descriptor exhaustion it causes when it is wrong: on the rig the 202nd repository touched since boot failed with open journal …: No file descriptors available (os error 24) and the ssh accept loop died with it.

It was 5 until 2026-08-30 (lane AH): objects.pack was opened three times in one process — the writer's read/write handle, Absorber::reader's read-only pread fallback, and IndexerPool::archive's read-only handle for the index workers. All three address the file positionally (pwrite / libc::pread / read_exact_at), the file is append-only and gc truncates nothing, so no cursor was shared and no ordering was observable between them: two of the three bought a kernel object and a page-cache alias per open repository and no isolation at all. They are now one Arc<File> handed out by ArchiveWrite::blobs_shared(), asked for inside PushPath::with_absorber so that no caller can forget it.

Nothing about durability moved. The writer still owns the append cursor, the flock and the blob-then-journal fsync order before the ack; sharing an open file description changes when it is closed, not what is written or in which order. One thing improved by accident: the flock now lives as long as the last of the three holders, so a reader outliving the writer can no longer drop the writer lock out from under a store that is still open.

tests/fd_per_store.rs counts them off /proc/self/fd per store and asserts the breakdown above — a descriptor is a kernel object, and reading the code counts the File::open calls you can find rather than the handles that exist.

Eviction is a segment unlink and not a rewrite, and that is the whole reason the tier can have a budget at all: the exploded table is one append-only Arrow IPC stream, so deleting a row in the middle costs a rewrite of up to 17 GB. Cut into segment files, reclaiming the coldest bytes is remove_file — O(1), no rewrite, no read amplification, and a crash during it leaves a pool that is merely shorter.

The budget, and the arithmetic that makes it safe

min(10 % of the filesystem holding the store, 8 GiB) per repository (exploded_cache::default_budget, ZNIPPY_GIT_EXPLODE_BUDGET, or the account's quota from gunnar's control document). On the shipped 300 GB appliance the ceiling binds, so twenty repositories cost at most 160 GB of cache together — against 141.7 GB for one before the cap existed. On a 64 GiB /data the percentage binds instead and each repository gets 6.4 GiB.

Nothing the budget bounds is data. Every row in either pool is re-derivable from the pack by materialize_at, so an eviction is never a loss and never an error a client can see — it is a later read paying ~0.5–2 ms to inflate its own chain again.

MEASURED — torvalds/linux, oden, 2026-08-30

8 731 794 676 B pack, 13 948 244 objects, 11 215 872 deltas, into a fresh store on md1; st_blocks * 512 per file; budget 8 GiB.

Graph's remaining 3.2× was one thing and one thing only: objects.exploded held 28 244 349 440 B of tree payloads. They were eager because reach::ObjectFacts wanted &HashMap<String, Vec<u8>> — every tree in the repository, resident, keyed by a hex String, rebuilt on every fold, which is also why peak RSS was 50.9 GiB on a box that ships with 16 GB. That map is now reach::TreeSource, one lookup method; the eager arm is still a HashMap (it implements the trait and returns Cow::Borrowed, so it copies nothing) and the store's arm resolves the tree the way any other object is read.

The plateau — the answer to "does lazy explode take the disk anyway?"

Yes, unless the tier is capped: a client that browses a whole history reads every object once, and a write-back-on-first-read tier would then hold exactly what the eager one held. So the budget is not an optimisation, it is the design.

MEASURED, same run, reading all 13 948 244 objects once, 390.8 s, 0 missing: the store stayed between 4.57× and 4.70× the pack for the entire scan while the cache took 174 412 527 296 B (162 GiB) of write-backs and gave every one of them back, 324 segments unlinked. Without the budget those 162 GiB would have been on the disk. exploded_cache::tests::the_tier_plateaus_at_its_budget_under_a_full_scan is the assertion; examples/lazy_explode_cost.rs is the harness that produced the table.

What it costs on the read side

MEASURED, same store, 200 objects sampled evenly across the pack:

The p50 barely moves because most objects sampled are commits and trees; the p90 is the blob tail, and 44 µs is what one delta chain out of the page cache costs. The design note's estimate was 0.5–2 ms per object, so the real number is an order of magnitude better than budgeted — the note assumed a disk seek per link and the pack is mapped.

znippy-plugin-git: the writer arms (status 2026-08-30)

archive_write.rs has three WriterArms with the SAME journal and, for the two durable ones, the SAME contract — the blob's bytes are durable before the journal row that references them:

uring_write::probe() is the door for a server that has to choose before its first store: one io_uring_setup of a two-entry ring + the opcode probe, no registration, no file. gunnar's PID 1 calls it when the medium's korp.data.io=uring token asks for the arm and falls back to safe on a row that names the errno (ENOSYS = # CONFIG_IO_URING is not set). The crate itself never falls back — UringWriter::create refuses — because a pwrite number reported as an io_uring number would be a lie. Ceilings the arm carries (module docs of uring_write.rs): one ring + one pinned page per open store, charged uid-wide against RLIMIT_MEMLOCK unless the process has CAP_IPC_LOCK (PID 1 on the appliance has it); MADV_NOHUGEPAGE on the staging page so mimalloc's huge-page arenas are not charged 512× . The uring-vs-safe numbers on gunnar's appliance rig (same ISO, token flipped): gunnar .nornir/best-results-benchmarks.md §1b.

The measurement that made uring the shipped arm (lane S, oden 2026-08-30). One installed image, the korp.data.io= token rewritten on its ESP, five runs per arm, 4 vCPU / 8 GB q35/KVM, xfs /data. At 1–8 clients the arms are indistinguishable — sequential push-create 172 ms vs 174, 8-parallel 1145 vs 1175, a 50 MB blob 1848 vs 1808, clone and fetch a tie — all inside a ±26 % noise floor taken from the three phases that never reach a writer (sequential clone, parallel clone, incremental fetch). The one row with a signal is the deepest queue tested, 200 push-creates from 64 clients at once, where the five run-medians do not overlap: safe 5130–6149 ms against uring 4433–4682 ms — 1.14× per accepted op, at ±3 % run spread against safe's ±10 %. Four syscalls were never the bottleneck below that depth; what the ring buys is the tail and the repeatability.

Two warnings for anyone re-running it. The 24- and 64-parallel wall clocks are admission rows, not writer rows: gunnar admits 19 concurrent transfers and refuses the rest by name, so at 64 clients only ~250 of 1000 ops reach a writer and a run that refuses more finishes sooner. The first single run of that matrix read 1.4–1.55× for uring purely because the two arms refused different counts. Compare medians of ACCEPTED ops, never phase wall clock.

Durability, on the arm that ships: three SIGKILL power cuts in the middle of an 8-parallel push storm, verifying every (repo, sha) the client had seen acked — 133 of 133 present at the acked sha, 0 missing, 0 wrong, 34 sampled repositories clone --bare + cat-file -e + fsck clean, XFS log recovery on /data at every boot.

Measured on the KVM rig 2026-08-30 (16 GB, SeaBIOS band, linux mirror): the push acked at 4144 s (the reader's one scan of the pending pack, index.scanned 13 948 244 objects in 2944 s, 0 re-inflations, is what the ~800 ref updates waited on — their own work, not the worker's); the band's row climbed 19% 2.7M/13.9M resolve 2.9M/s → 58% 8.1M → 80% 11.3M paused 1p while ram went 6.4G → 10G → 13G; the watchdog paused at 3.08G available < 3.13G budget. The pause held because the readers' scanned table and the exploded oid tree were not the worker's to drop — they are now (Absorber::spill on every pause). An 8 GB and a 12 GB box died before the ack, in gunnar's read_pushed_pack Vec (fixed in gunnar: spooled to disk above 64 MiB).

znippy-plugin-git: the ref log costs one pass per push, not two per push (lane Y, 2026-08-30)

refs.rs is an append-only Arrow push log and fold replays it into the current namespace: "reading the current refs is therefore a scan of a structure sized by pushes, not by repository size". True, and it was being paid twice for every single reference update — GitOps::update_ref folded the whole log through ref_state() to compare the caller's expectation, and then RefLog::push scanned it again to re-derive push_seq. RefLog::push even said so in its own doc, and offered append_batch to a server holding its own counter, which nothing took.

Two O(pushes) passes per push means the cost of landing a reference grows with every reference the repository has ever landed. Measured on oden, release build, one update_ref (fold the namespace, derive the sequence, append the frame), medians over 20:

13.5 µs per prior push, with no ceiling. A repository at ten thousand pushes burned 135 ms of pure rescan before its next reference could land.

RefLog now carries the folded namespace and the counter forward in memory, loaded by one lazy scan the first time a handle is asked anything. The log is still the authority and the frame is still the transaction. The crash contract is a property of where the cache is advanced: only after PushLog::append has returned, i.e. after the frame is on the platter, so a failed append leaves the namespace and the counter exactly where they were, and a restarted process rebuilds both with the single scan recovery has always done. The assumption added is one writer per store, which GitStore already required — it serialises reference work behind a process-local ref_gate, and two processes over one store were unsound before this because that mutex could never have seen the other one.

PushLog::scans counts the only O(log) operation the type has, so the guard asserts passes over the file rather than a wall clock — one pass for a handle that has pushed 1, 100 or 1000 times, where it was two per push, seen red at every N. No push-create benchmark can see any of this: a create's log is empty, and on the 2 vCPU appliance rig the sequential push-create row moved 131.4 → 130.6 ms, i.e. not at all (gunnar .nornir/best-results-benchmarks.md §1c).

And the empty store's two barriers, which were 1.5 ms and 0.0 ms

SafeWriter::create and UringWriter::create fsynced the journal's schema message into a fresh journal. A store with no journal row has acked no pack, and gunnar's create is transactional, so a create that does not finish leaves no catalog row pointing at the store; the first row a writer claims fsyncs the same file and takes the schema with it. What the barrier bought — that a torn schema could not be inherited as whole — is bought by archive_write::journal_head, one function both arms ask (LAW 5): an empty file gets the schema, a file that opens with exactly this build's schema message is left alone, anything else no longer than the schema was interrupted before a row could exist behind it and is written again, and anything else longer carried rows and is reported as corrupt rather than truncated away.

GitServe::set_head_projected is the same idea one level up: a caller that keeps HEAD's authority elsewhere and re-applies it on every open (gunnar's catalog does) may have the write without the barrier. A detached HEAD is derivable from nothing and keeps it.

Neither barrier was the cost. On the appliance rig an fsync is ~0.22 ms (4k sync 4517 ops/s), and removing these two moved their spans by 1.48 ms and 0.28 ms — one fsync each — against the 8.9 ms and 6.0 ms those steps measure. The step is not the barrier.

znippy-plugin-git — a THIN pack, and why the archive grows one delta link per push (lane AI, 2026-08-30)

What a thin pack is

A git push does not send whole objects when it does not have to. Having read the server's ref advertisement, the client knows what the server already holds, so it sends a thin pack: entries whose REF_DELTA header names, by oid, a base object that is not in the pack. A one-line change to one file is three entries — commit, tree, blob — of which two are usually deltas against objects the server acked a moment earlier. This is the ordinary shape of every push after the first, not an edge case.

Stock git resolves that at receive time and then completes the pack: git index-pack --fix-thin appends the resolved base objects into the pack it just received, so what lands on disk is self-contained and its delta chains never leave it — bounded by the pushing client's own pack.depth, 50 by default.

What §14 does instead, and the consequence nobody had measured

This store appends the client's bytes verbatim. That is the whole reason a clone can be a byte-range copy rather than a re-pack, and it is not negotiable. But it means the REF_DELTA keeps naming a base in an earlier pack, forever:

  pack 0   blob c1     whole
  pack 1   blob c2     REF_DELTA -> c1        depth 1
  pack 2   blob c3     REF_DELTA -> c2        depth 2
  ...
  pack N   blob cN+1   REF_DELTA -> cN        depth N

A repository pushed one commit at a time grows its delta chain one link per push, without bound. Nothing repacks; nothing is supposed to.

What the absorber does about it

Absorber::absorb_one resolves each pending pack with self as the BaseSource, and a base outside the pack is answered in this order — cheapest first, and every step is a point lookup:

1. §14's exploded table (ExplodedCache::content) — resolved content by oid, no inflate, no chain.
2. the objects index — the base's archive offset, then resolve::materialize_at follows its own chain out of the mapping: d inflates and d − 1 delta applications, nothing else in the pack touched.
3. the pending packs, read-only, on this thread (Absorber::scan_pending) — for a base in a pack that is acked but not yet indexed. Lane Q's re-entrancy rule holds here: a nested call may LOOK UP an already-built scan and may not BUILD one.
4. nothing — and only then is the pack refused by name, as ResolveError::missing_base.

Ok(None) from a base source means absent and is the only answer step 4 may be reached from. Err means held and unreadable, and it travels: it is the store's failure, not the pushed pack's, and the two must not arrive as the same sentence.

🔴 The ceiling that refused the 101st link

resolve::MAX_DELTA_LINKS = 100 was a refusal — "a chain longer than this is not a deep delta, it is a loop" — and step 2's walk hit it as soon as a repository had been pushed a hundred and one times. Absorber::resolved_costed then fell through to step 3, found nothing there (the base's pack was long since absorbed), returned Ok(None), and BaseSource for Absorber had already dropped the real error with .ok().flatten(). So the client was told, about a blob this repository was holding:

  this pack is thin: it deltas against 509be6a7… which is not in it, and no base source
  could produce that object's content — §14's exploded table does not hold it and it
  could not be re-derived from the verbatim packs either, so this repository genuinely
  does not have that object.

MEASURED on oden, 140 sequential single-commit pushes into one repository: 21 refused, first at push #120, and then every push after it, forever — because the absorb loops ran for job in &jobs { self.absorb_gated(job)?; } in ordinal order, so the one pack that could not be indexed was hit first and aborted the loop for all the packs behind it. At 8 / 24 / 64 clients: 168/1120, 504/3360, 1344/8960 — the same 21 per repository, every time. Lane AD's two-appliance EventGap leg never converged for the same reason: the primary's GitOps::reachable failed on its own store and the standby was re-asked every 300 s for 21 minutes.

A count cannot tell a deep chain from a loop. The ceiling is a termination proof now: resolve::CycleGuard records the archive offsets a walk has stood on past MAX_DELTA_LINKS and refuses a repeat, which is what a cycle actually is. It is strictly stronger — a three-link loop is caught at its first repeat instead of 100 links later — and it never refuses a legitimate chain. Under the threshold it is two integer operations and no allocation, so the hot read path is unchanged. One guard, two walkers: serve::resolved_type_at had a hand-written copy of the same ceiling and shares this one (LAW 5). After: 0 of 140, 0 of 1120, 0 of 3360, 0 of 8960, and the EventGap leg converges in 4.1 s.

What bounds the cost, since the depth no longer does

Nothing caps chain depth, so a read of the newest object of an N-push repository is O(N) inflates the first time. What makes that affordable is lane Q's cache, not a ceiling: Absorber::keep_for_the_next_reader writes back what a read cost (ReadCost::depth), a deep chain is worth_caching by construction, and the next push's base is that object — answered at step 1, in one lookup, with no walk at all. The cost is paid once per depth, by the reader that hit it, and never again.

A pack that cannot be indexed STOPS, and says why

MAX_ABSORB_ATTEMPTS = 3. Three and not one, because an absorb can fail for a reason outside the pack — the memory watchdog gave up, the repository is being deleted, a transient read error — and one bad moment must not condemn durable bytes. Three and not thirty, because the resolve is deterministic in the store's own state and every attempt is paid for by a client waiting on a push.

Absorber::absorb_each is the one loop all four entry points run. It runs to the end and reports the first failure afterwards, so a pack that cannot be indexed no longer stops the push that arrived after it from being indexed and acked; and a pack that has been given up on is recorded in Absorber::unindexable with the backend's own root cause, logged once as index.unindexable, skipped from then on, and readable through GitStore::unindexable_packs(). Its bytes stay durable and untouched, its objects are genuinely absent from the index, and every read path already knows how to say that: a push that deltas against one is refused by name at external_bases_exist, and a clone that needs one fails its connectivity check — both immediately, neither after another futile whole-pack resolve.