nordisk/gunnar

The twin holds every transaction — the transaction log, its wire, and the twin that applies it

Written 2026-09-15 on branch twin-holds-all. The owner's rulings (2026-09-14 late, and the morning after): the twin must hold each transaction sent to the master; if it has missed something it must ask the master for the missing part before continuing to append; the twin is NEVER promoted; restore = replay into a fresh primary; the twin holds packs, LFS and every control-database write — never caches, never repack scratch; it runs as its own tunnr appliance (gunnar-twin) on ONE slow disk and grows on fill.

gunnar_core::txlog is the storage half (frames, segments, index, torn tail). This note is the rest: what a transaction is per kind, how the primary writes one BEFORE its state changes, the wire, the twin's loop, the gap, the snapshot, restore, and what is deliberately not logged.

One number

Every durable mutation on the primary gets the next seq of ONE log, <data-root>/txlog/. Not per repository, not per table: a twin holding 1..=n holds everything the primary was told in that window, in the order it was told, and a restore is a forward replay of one stream. The bus's seq is not this (the bus is not durable); history's push_seq is not this (derived per repository at read time).

What a transaction is, per kind

CAS is replayed, not resolved. A Refs or ControlDoc entry carries the claim the primary made (expected / old). The twin makes the same claim against its own copy. Because the twin applies exactly the primary's sequence from the same starting state, a claim that held on the primary holds on the twin, and a claim that lost on the primary loses on the twin — Ok(false), not an error. That is what makes "log BEFORE the state changes" safe: an entry whose mutation never took effect on the primary replays to the same no-op.

Before, not after

The primary appends (and fdatasyncs) the frame, THEN performs the mutation. (One refinement, 2026-10-06: a control document's SHARD BLOB is part of no state until the ControlDoc frame that moves the pointer publishes it, so its frame is staged and its row committed without an fsync of its own, and the ControlDoc frame's fdatasync and the pointer's durable redb commit make both durable — still before anything that can be seen. §"Group commit".) A crash between the two leaves an entry the primary never applied; the twin applies it; the primary, on its next boot, does not — and that is the one asymmetry a restore has to know about: the twin can be at most one transaction AHEAD of a crashed primary, never behind. (A restore replays the twin, so the restored primary has that transaction. The crashed primary, if revived, must NOT be re-used as the primary without a replay from the twin. The twin is the truth.)

The tee is a decorator on the three stores (gunnar_core::txlog::tee): LoggedGit around the engine's GitServe, LoggedControlStore around ControlStore, LoggedSecretStore around SecretStore, and LoggedLfs around LfsStore in the server. Nothing in a call path changed; ServerStores::open wraps at the one place each store is constructed. A box whose log cannot be opened does not serve: a primary that cannot write its log is a primary with no backup, and that must be loud at boot, not discovered at the restore.

The wire (Mirror, additive)

from < first_seq answers FAILED_PRECONDITION naming first_seq: the twin has a hole retention has closed, and the only honest answer is a SNAPSHOT (below). A twin never guesses.

Mirror.FlushReplication (branch data-set-grow) keeps its per-standby verdict and gains, when that branch merges, one more clause: caught_up is also applied_seq == head_seq on every twin — the watermark. Until the merge the same fact is read from the twin's LogHead.

The twin's loop

open own txlog (own head = h)          own stores (same layout as a primary, /data on one disk)
loop:
  H = primary.LogHead()
  if H.first_seq > h+1        → SNAPSHOT NEEDED: stop, say so, do not append anything newer
  if H.head_seq > h           → LogFetch(h+1 ..= H.head_seq): for each entry, in order, AS IT ARRIVES:
                                  begin(seq) a frame in own txlog, write each piece, commit on the last
                                  (durable)  → take: control plane applied, data KEPT (see below)
                                  h = seq
  report fill = bytes used / size of the volume   (what the watchdog grows on)
  sleep(poll) or wake on Events.Watch

The gap rule is the loop's shape: the twin never appends seq unless it holds seq-1, and append_at refuses anything else. A restart mid-fetch resumes at its own head: whatever was appended is applied again (every apply is idempotent or a CAS replay), whatever was not is fetched again.

One entry is never whole in memory, on either box (twin-streaming, 2026-10-06). The fetch's byte ceiling (64 MiB) bounds how many entries one LogFetch carries, never the size of ONE: a 256 MiB pack used to be read whole on the primary (fetch_range), cut into a second whole copy of 1 MiB messages (chunks_of), and glued back into one Vec on the twin (GrpcSource::fetch) before it was appended — the shape of test2's "1 GB twin silent under 256 MiB packs" (2026-09-27). Now:

• the primary reads each entry off its segment a piece at a time (Log::read_pieces) on a blocking thread behind a 4-piece window (ControlPlane::log_fetch_stream, twin::stream_range); the entry's LAST piece leaves only after its payload crc checked, the check Log::read always made. The refusals (FailedPrecondition for a range retention dropped, InvalidArgument beyond the head, [primary-growing]) are still the call's status, answered before the first piece;
• the twin hands each piece, as it arrives, into a txlog::Pending at the primary's number (LogSource::fetch_into → FrameSink; the LogChunk protocol checked by twin::ChunkFeed) and commits on the last. The length is held both ways (more than declared is refused at the piece, less at the end); a cut, short, jumbled or refused entry is cut off the segment — and so is any Pending dropped without a commit, which before this left its half frame in the file and broke the next append's read. The take reads the frame back only when it needs the bytes (twin::take_logged): a twin keeps its data frames without reading them again.

MEASURED (tests/twin_stream_memory.rs, one 512 MiB entry over loopback LogFetch, both boxes in one process, peak resident set over the reset): +1031 MiB before, +15 MiB after. A following twin says its own peak resident set at INFO (twin memory: peak …), once, and again each time it climbs 32 MiB — the line an operator reads on its log door. Still whole: a box being RESTORED (Bootstrap.Restore, restore_from) gathers each entry, because apply of a pack needs it whole.

An entry larger than one call of the restore lane (restore-large-entry, 2026-10-07). MEASURED by the r5 integration test (R1-recharge-main-lost): a recharging main stuck at a 200 MiB pack, 41 calls of monetize's restore lane, each "the stream ended inside an entry; resend from the box's head + 1". monetize's doors cut every stream a guest opens 30 s after it opens (half the 60 s call), the lane resumed per ENTRY, and Bootstrap.Restore gathered each entry in memory and dropped it when the stream ended inside it: one entry slower than a call could never be restored. Now:

• the box stages an entry that spans more than one piece and is over 4 MiB (or any entry it holds part of) in <data-root>/txlog-restore-partial/ (twin::intake, per box, never framed): entry.blob its bytes so far, entry.rec its seq, kind, length, meta and how many of its bytes are DURABLE — rewritten (atomic, fsynced) after an fdatasync every 8 MiB, at every end of a stream (clean, broken, the sender's cancel — tonic reads a client's RST_STREAM as a clean end — or a refusal) and when the intake drops. Bootstrap.Status.restore_partial_{seq, bytes} say it. The entry is applied only WHOLE: its declared length, and the crc32c over meta ‖ blob equal to LogChunk.entry_crc — the twin's own payload crc, which Log::read_pieces checks before the last piece leaves (Located.payload_crc). Then append_at + apply, and only then is the staging removed, durably: nothing is left on the volume. One stream at a time holds a box's intake; a second waits up to 20 s for an abandoned one to let go.
• the twin starts the entry at the byte the box holds (LogFetchRequest.from_offset): it reads the blob from that byte only (Log::read_pieces_from) — a resumed entry of gigabytes costs it the rest per call, never a re-read that would eat the call's budget on a slow volume — and stamps the frame's STORED crc on the last piece, unchecked there: the box that put the entry together checks it over the whole. Every LogChunk says its offset. An offset past the blob is not the entry the asker knew (one erased since): it goes whole from 0, checked, and the length tells the box.
• the lane (gunnar-monetize-guest) asks for from: head + 1, from_offset: restore_partial_bytes and ends its OWN streams at a piece boundary when its budget is spent — two thirds of the stream deadline the host says (host_env("stream_timeout_ms"), monetize-product-wasm), 20 s from an older host — so the box commits and answers where to resume; the phase stays replaying, and the dead-main watch calls again. A call cut by the host anyway loses only what arrived since the last commit. The global timeout is NOT raised.
• crash-safe and idempotent: an entry is applied once (append_at takes head + 1 only; the staged entry must be the one at head + 1 or it goes when the intake opens); a tail past the durable count is cut back; a file SHORTER than its count is not trusted (the entry starts again at 0); bytes put together across calls that fail the crc are dropped and the entry starts again.
• older peers: an older twin ignores from_offset and sends the entry from byte 0 with no offset; the box skips what it holds, which makes the stream NO shorter — so an entry needing more than one call still cannot pass an old twin. The box says so by name once a stream of such a sender ends without a byte past what was held: REFUSED [restore-twin-too-old] the twin is too old for a resumed restore; upgrade the twin (the estate upgrades its twin with its main). An older lane strips the new fields the same way (same refusal). An older main ignores them and gathers entries as before.
• still whole: the apply reads the staged entry back whole (apply of a pack needs it), as the gather did; between calls the box holds nothing of it in memory. gunnar control twin replay (no call cap) resumes per entry as before. A twin's snapshot base is resumed too since restore-after-snapshot (below).
• monetize's LINKED gunnar plugin (monetize/products/gunnar) still resumes per entry, and nothing that ships runs it (checked 2026-10-07): it is a DEV-dependency of the open monetize-server and of monetize-product-wasm (their tests), and a normal one only of monetize-impl's monetize-server-vetra, whose deploy (monetize-impl cargo xtask refresh-front, retired 2026-10-12) was the pre-appliance Ubuntu front. No estate runs a monetize (owner 2026-10-12: exactly ONE monetize, the ONE box); the ONE box's [products.gunnar] is this guest. Its vendored protos carry the new fields (monetize c00cd3a); the day it ships again it adopts the same resume.

MEASURED (dwarf): gunnar-server/tests/restore_large_entry.rs — a 24 MiB pack, 7 MiB per call: before, 12 calls holding [0, 0, …] bytes; after, held 7340032 → 14680064 → 22020096, done on the 4th call; a main restarted mid-entry with an uncounted tail resumes from its durable count; an old twin is named. gunnar-monetize-guest/tests/the_restore_lane_resumes_inside_a_large_entry.rs — the real .wasm, calls cut at 2 s, a 3 MiB/s link, a 16 MiB pack: before, 24 calls each "CANCELLED after 5 chunk(s)", never past it; after, 4 MiB more per call, done on the 5th.

A main recharged from a twin whose log starts after its snapshot (restore-after-snapshot, 2026-10-07). FOUND by the round 5.1 integration test: after a twin took a full resync it keeps the snapshot as its BASE (txlog-snapshot-base) and its log starts at at + 1 (first_seq 3344 there, a 617 MB base). monetize's restore lane read first_seq > 1 as refusal 4 — "… retains its log only from sequence 3344 … Say accept_lost" — every minute, and accept_lost would have lost the control database and the refs, which are in the base. gunnar's restore_from and control twin replay stream the base first; the lane never did, and a base larger than one capped call could not arrive anyway (Bootstrap.Restore staged snapshot pieces per stream and dropped them when the stream ended; a twin sent its base only from the start). After any full resync — a silent twin (cov-13), twin-resync-space giving its copy up, a new twin — a lost main stayed fresh. Now:

• the twin says its base and streams it from a position: LogHead.snapshot_at / snapshot_bytes (only while its copy is whole); SnapshotFetch{from_item, from_offset, expect_id} reads its base from that item and byte only (snapshot::stream_base_from), every piece naming snapshot_item, offset and snapshot_id (a blake3 of the base's item table, its crcs and its size — two bases differ in it whenever any item differs); each item's last piece carries its crc32c (meta ‖ blob), which every Stager now computes as it stages and a twin keeps beside its base (crcs; a base kept before has none and serves no crc). A base that changed since is REFUSED [snapshot-changed], never spliced. A primary's own snapshot is a new cut every time: it is not resumed and its pieces name no position.
• the box being restored stages the base ACROSS streams (snapshot::RestoreSnapshot, in <data-root>/txlog-restore-snapshot/: blobs + restore.rec — the base's id, the items staged whole with their crcs, the open item with its running crc32c state, the DURABLE byte count), committed (fdatasync, then the record, atomically) every 8 MiB, at every end of a stream and on drop; Bootstrap.Status.restore_snapshot_{resumable,id,items,bytes,whole} say it. A piece is placed by its item and offset: what the box holds is skipped, a gap is refused naming where to resume, an item is checked against the twin's crc at its end (a mismatch drops the staging and refuses by name — the base is taken again from its first item), and the terminator installs the base (snapshot::install, whole) and removes the staging; the log follows from at + 1. A crash leaves bytes past the count, cut when the next stream opens; a file shorter than its record is not trusted; a base staged whole but not installed is installed when the next stream opens. Pieces with no snapshot_id (a primary's snapshot, an older twin through twin replay) take the one-stream path as before.
• the lane (gunnar-monetize-guest): first_seq > 1 and the box below first_seq - 1 → the twin's base first, from restore_snapshot_items/bytes (unless the box holds it whole), in the same Bootstrap.Restore stream, ended at a piece boundary by the call's budget like the log, then the log from first_seq. Nothing is lost, so lost is 0. A twin that does not say its base is asked for its snapshot: pieces without a position are an OLDER twin's — upgrade the twin; a box whose Status does not say it can stage one — re-lay the main; both refused by name, neither with an accept_lost path. Only a twin that keeps no snapshot at all ("keeps no snapshot", or Unimplemented) meets refusal 4 and accept_lost; anything else is asked again next tick.
• still: each record rewrite carries the whole item table (a base of thousands of control rows writes a few hundred KB per 8 MiB committed); the base is installed at its terminator inside the stream that carried it, so a large install runs past that call's budget (the next call finds it done).

MEASURED (dwarf): gunnar-server/tests/restore_after_snapshot.rs — a 24 MiB pack in the base, 7 MiB per call: before, [(0, 0) ×16] held; after, 19 items + 7340032 → 14680064 → 22020096 bytes of the next, done on the 4th call; a main restarted mid-base with an uncounted tail holds exactly what it said; a base that changed between calls is taken again from its start. gunnar-monetize-guest/tests/the_restore_lane_recharges_from_a_snapshotted_twin.rs — the real .wasm, calls cut at 2 s, a 3 MiB/s link, a 16 MiB pack in the base: before, 40 calls each refusal 4; after, 4 MiB of the base per call, done on the 5th; the recharged main is the primary.

Group commit (perf twin-apply-fsync, 2026-10-06). The load test on the UpCloud mockup measured the twin (1 vCPU, one HDD-tier volume) at ~10 write operations and ~3 flushes PER entry while following a storm of sign-ups and small pushes: ~2,800 write IOPS at 0.03–0.2 cores, a lag of ~93,000 entries, and every emergency growth of main refused because the twin was not caught up. Counted in one process (tests/twin_apply_io.rs, strace on the twin's data root, 1,351 entries): 12,984 writes and 2,503 fsyncs — the frame's header, meta, piece and trailer as four writes, an index write and an fdatasync per frame, and ~5.7 redb page writes and one more fsync per control frame. Now:

• the log — the follower writes each entry of a fetch with Pending::commit_unsynced (frames staged and coalesced; a large piece goes straight out), and ends the group (GROUP_ENTRIES 2,048, GROUP_BYTES 32 MiB, GROUP_AGE 500 ms, and always at the end of a fetch) with ONE Log::sync: one fdatasync, one index write. A frame is visible to readers and counted in the head ONLY once it is durable, and the head is what the twin acknowledges (applied_seq), so a twin never acknowledges what a crash could take back.
• the control plane — after the group is durable, its control frames are taken into ONE redb write transaction (ControlDb::begin_group/commit_group) committed once, with the applied marker twin.applied (table box_marks, per box, never framed or snapshotted) inside the same commit. The log is durable BEFORE redb is written, so after a crash the log may be a whole group ahead of redb, never behind — and redb is EXACTLY the state through its marker, so the restart re-takes (marker, head] in order (Follower::retake_from_marker). That exactness is what makes a re-take of more than one entry safe: a compare-and-swap the primary LOST loses again only when it is replayed onto the state it lost against (the old rule re-applied the head alone, exact only while every entry was durable on its own). A twin from before the marker keeps the old rule once and gets a marker; a snapshot install sets it to at; a ControlDoc the fresh-box rule still names commits on its own (its fresh_take writes a file durably first). A run of data frames moves no redb page and is marked every MARK_EVERY (4,096) entries.
• MEASURED (same test): writes 12,984 → 214 (9.61 → 0.16 per entry), fsyncs 2,503 → 5 (1.85 → 0.004). Crash consistency (tests/twin_crash_mid_group.rs): a following twin killed mid-group and again between the log's fsync and redb's commit comes back with nothing it acknowledged lost and, followed to the head, every document pointer, every one of 160 versions exactly once and every blob equal to the primary's; re-taking only the head (the old rule) after the second kill is RED (385 blobs missing).
• the primary appends with the same stage-then-sync: concurrent appenders share one leader's fdatasync (taken outside the writer lock, Log::sync_through); one appender alone pays what it paid before. A document change's shard blobs ride on its pointer move (LoggedControlStore:: put_blob: Log::stage + put_blob_unsynced, made durable by the ControlDoc frame's fsync and the pointer's durable redb commit; a box being RESTORED keeps durable blob rows, because its head-only re-apply could not re-put a blob several frames back). Measured (tests/main_doc_write_io.rs, counters Log::frame_syncs / ControlDb::durable_commits): 30 sign-ups = 540 frames, 540 + 540 flushes before (6 + 6 per document change), 90 + 90 after. Its audit trail stages its frame and waits with the chain unlocked, and the line in audit-chain.jsonl follows the durable frame without an fsync of its own (the file is rebuilt from the log at every start). Repeated plain reads are AGGREGATED, never dropped (P3, grpc_gate::READ_WINDOW): the first read of a 60 s window per (method, principal, scope, decision, …) is written at once and one more entry carries the count and the first and last time of the repeats — the trail still answers who read what, and when to the window. Measured (tests/audit_read_io.rs): 300 reads by one principal on one scope = 300 frames, 600 fsyncs, 250,880 B before; a few frames after; security decisions on eight threads 2.0 → 0.26 fsyncs each, every one an entry of its own.

Snapshot (built 2026-09-15, Lane K; twin::snapshot). When retention on the primary has dropped the range the twin needs, the twin takes a full copy, never a partial one, and never appends past the hole. Mirror.SnapshotFetch (Admin or group twin) answers the MUST-NOT-LOSE classes at ONE sequence at, spelled as the log's own entries with seq 0 — Kek, every ControlBlob, every ControlDoc pointer, every Secret row (tombstones included), every framed file (txlog::FRAMED_FILES) whole, and per repository RepoStore{created}, one Pack (the engine's own export: select over every tip, then emit_pack, the byte-range copy a clone takes), one Refs with HEAD, every LfsPut — then a SnapshotAt{at} terminator.

Consistent at at: the primary produces it under Log::freeze. Every framed mutation holds a Log::mutation guard from before its frame until its store has the write (the tee, the LFS tee, the push/fetch ingest, purge_repo, every sink writer); the freeze stops new ones and waits for those in flight, and only then reads the head and copies. Without the wait, a frame at the head could still be on its way into the store, and a twin resuming at at + 1 would never see it. Writes pause for the CUT, not for the copy (R5.1-A, 2026-10-07): snapshot::Cut::take fixes, under the freeze, the head, a redb read view of the control database (MVCC), each framed file as an open handle and its length, and every repository's refs and tips; the freeze ends and the bytes stream from the cut while writes go on. A snapshot pin (ServerStores::snapshot_pin, taken before the freeze) holds back a gc and a repository store's removal until the stream ends (at most 600 s, then that snapshot fails and is asked for again); LFS objects are listed after the cut (a later LfsPut/LfsDelete is a later frame either way). MEASURED (twin::tests::a_framed_write_during_a_ slow_snapshot_is_not_held_for_the_stream, a 3 s stream): a namespace created during it took 3.03 s before (the whole stream), 3 ms after. Account purges no longer stop appends either (R5.1-B): Log::erase holds the writer lock only to publish and read its plan, and costs two fsyncs per segment (300 frames in 7 segments: 307 fsyncs, appends blocked → 14, appends free).

Staged beside, room made (twin-resync-space, 2026-10-06): see §"A full resync on a volume the size of main's set" below.

Installed whole: the twin marks txlog-snapshot-installing, removes every repository store it holds, every framed file and every live secret the snapshot does not carry, force-sets every document to the snapshot's pointer, applies the rest with the same apply, and Log::restart_after(at) — its log is empty with head at, and following resumes at at + 1. A crash mid-install finds the marker and asks for a snapshot again. A source that cannot make one (a bare log) still stops the twin with snapshot-needed.

A restore from a snapshotted twin: its log starts after 1, so gunnar control twin replay streams the twin's own SnapshotFetch first; Bootstrap.Restore buffers seq 0 items until the terminator installs them and continues with the log from at + 1 (twin::restore_from natively). Proof: twin::tests::a_twin_behind_a_retention_hole_catches_up_via_snapshot_and_then_follows (a tag deleted, a repository purged, a secret deleted and a principal added inside the hole; state equal after the install, equal again after following; a restore from that twin equal too), plus txlog::tests::a_freeze_waits_for_a_mutation_in_flight_and_holds_new_ones_until_it_ends and restart_after_…. The snapshot is streamed on both ends since 2026-09-15 (tests/twin_snapshot_memory.rs), and a LogFetch entry since 2026-10-06 (above).

Never promoted. There is no SetRole(PRIMARY) path on a twin. Its stores are the restore source and the replay target, nothing else; its variant (gunnar-twin) starts with --role twin, so its forge listener is loopback-only from the first boot.

The twin only stores — OWNER RULING 2026-09-22 (lane T21)

"the twin shouldnt do indexing or processing, only receive and save data … if the main is recharged from the twin it could do the indexing then, at recharge time."

What the code does since agent/t21-twin-store-only (twin::take, twin::twin_applies):

• The log is the copy. A twin's log is never trimmed (log_retain refuses on a follower), so every pack byte the primary was told is in it, at the primary's number.
• Snapshots (a retention hole on the primary): the twin installs the control items and KEEPS the staged file as <data-root>/txlog-snapshot-base/ (blobs + items), renamed into place BEFORE restart_after(at). Mirror.SnapshotFetch on a twin streams that base (snapshot::serve), so twin replay = base, then the log from at + 1.
• Recharge (gunnar control twin replay, twin::restore_from, cargo xtask mock restore main) runs the full apply on the NEW main — that is where every index is built.
• Backfill and retention are unchanged: the gap rule, snapshot-needed, the retention guard (9212808).
• Proofs: twin::tests::a_store_only_twin_builds_no_derived_index_and_keeps_every_frame_byte_for_byte (RED when the store-only branch is neutralised: 6 derived files under repos/), a_recharge_from_a_store_only_twin_rebuilds_every_index_on_the_new_main; every other twin test now reads "the twin holds X" through a main recharged from it.
• Measured (twin::tests::twin_cost_per_frame_on_the_td8_workload, TD8's seed_repos and step-5 control plane, 75 frames of which 18 data, 229 MB of blob; debug build, tmpfs): CPU 37 970 ms → 1 770 ms (506 → 23.6 ms per frame, −95 %), disk 459.8 MB → 229.6 MB (6.13 → 3.06 MB per frame, −50 %: the second copy — the applied store — is gone).
• The wire word applied_seq now means "held in the twin's log"; it is not renamed.

A full resync on a volume the size of main's set (twin-resync-space, 2026-10-06)

The gap (cov-13, found by the growth-twin-silent work). A twin behind a retention hole takes Mirror.SnapshotFetch and stages the snapshot in ONE file beside the copy it replaces — its kept base (txlog-snapshot-base/) and every log entry after it. Only the install swaps them. So a resync needs the old copy AND the new one on the twin's volume: about twice main's data. The emergency path grows main ALONE while its twin is silent, the twin is then grown to main's set ([twin-too-small] until it is) and resynced — and a twin grown to main's set filled up part-way through the staging, failed with ENOSPC, and failed the same way on every later poll. Each attempt held main's writes for the length of the stream (the snapshot was produced under Log::freeze; since R5.1-A only its cut is).

The fix: (b), adaptive. A twin's Stager makes room (Stager::making_room):

• before each piece the volume is measured (snapshot::Volume::space, one statvfs). While the piece leaves at least staging_floor free (1 % of the volume, 1 MiB..256 MiB) the snapshot is staged BESIDE the old copy, as before: a twin with room keeps its old copy until the new one is whole;
• when it would not (or the volume answers ENOSPC first), the twin GIVES ITS STALE COPY UP, once: the marker txlog-snapshot-dropped is written first (durable: head, base at, bytes, time), then the base and the log go (Log::restart_after(own head): empty, the numbers do not move), and the staging goes on;
• still short after that: TooSmall, said with [twin-too-small] and the numbers. The twin notes the volume's size (txlog-snapshot-too-small), waits in recovering, and asks main for NO further snapshot until its volume is larger;
• the install removes both markers once the new copy is whole.

The window is explicit. From the marker to the install the twin holds NO copy and main is the only one: a WARN line ALERT the full resync does not fit beside this twin's stale copy … (event twin.resync_without_copy), the health alarm twin.resync_without_copy on the twin's Admin.Health (since the give-up, with the too-small sentence when that followed), and the state word resyncing whose why says "holds NO copy" (recovering / stalled carry the same sentence when the resync then fails). A twin in the window — or mid-install, the latent half of the same rule — serves nothing (snapshot::not_whole): SnapshotFetch and LogFetch on it are refused by name, so a recharge (twin replay, restore_from) cannot read half a set, --stale-ok or not. It never reads caught up: its applied head is its old one, below main's.

Crashes. Before the marker: the old copy is intact, the staging file is started over on the next poll. After it: the marker makes snapshot::installing true, so the next poll (after a restart too) finishes the give-up (idempotent), stages afresh and installs. Inside the install: the install marker, as before.

Why not (a), in place. A store-only twin's copy is one base FILE plus the log after it, not stores. Any mix of the old and the new is consistent at no sequence (the control plane at one point, the data at another), so a store-by-store swap has exactly the no-copy window of (b) — with a new base layout and a resumable snapshot protocol on top, and a peak no lower.

Why not (c), a preflight to ≥ 2×. The volume only ever grows: twice main's set is paid for ever, to keep through a resync a copy that is already behind a hole main trimmed. After (b) a twin of main's set holds the snapshot (main's data ≤ main's set less main's own 2 GiB floor and log budget; the twin keeps ≤ 256 MiB). The case it does not fit (a twin well below main's set) is the named [twin-too-small] wait above, not a fill.

Proofs (twin::tests::resync_space, the volume capped in-process — snapshot::Volume): resync_space_a_twin_the_size_of_mains_set_takes_a_full_resync_and_never_fills (RED before: No space left on device), …_a_crash_mid_resync_restarts_it_and_the_twin_is_whole_and_caught_up, …_a_give_up_cut_short_by_a_crash_is_finished_before_the_resync, …_a_volume_too_small_even_alone_waits_in_recovering_until_grown_and_never_hammers_main, …_a_twin_mid_install_serves_no_snapshot_and_no_recharge; each ends with a main recharged from the twin equal to the primary and the twin following, caught up. The same scenario on a REAL small filesystem: …_on_a_real_filesystem_the_size_of_mains_set (ignored; GUNNAR_TWIN_CAPPED_DIR).

Restore

gunnar twin restore --from <twin-log> --into <fresh data-root>: open the twin's log, replay first_seq..=head_seq into fresh stores with the same apply. The test that counts: after a scripted primary of pushes, ref edits, principals, keys, secrets and LFS, replaying the twin's log into a fresh root yields byte-identical packs, identical refs, identical control documents (every name, every version) and identical secrets. That test is twin::tests::a_restore_from_ the_twin_rebuilds_every_object_and_every_document and it is on the test matrix.

Not logged, and why

Caches and generated archives (recomputed), repack output (a repack is gc, not a transaction — the twin's repos are grown from packs as pushed and may gc on their own), page cache and indexes the engine rebuilds on open, the replication key, the grow journal (/serialdat, per box). The KEK IS framed (Kek, seq 1) — see §"The KEK after a restore" for why, and for the re-seal a restored box does.

The KEK after a restore — the ruling (Lane K, 2026-09-15)

Threat model (gunnar-core/src/seal.rs): the on-disk KEK is domain separation between the PORTABLE data (archives, backups, a copied control.redb, the log's Secret frames) and the MACHINE. It defeats data leaving without the disk; it does not defeat possession of the disk.

Why the twin holds the KEK. A twin is a restore source; sealed secrets without their key are a backup of nothing. So the twin's disk is, by construction, a disk that holds both — the same trust class as the primary's own disk, and it is kept like one (its own tunnr appliance, no git transport, Mirror.LogFetch is Admin or group twin).

Why a restored box must NOT keep the primary's KEK. The dead primary's volumes outlive it (returned to the cloud provider, a detached disk, a snapshot someone took). A restored box that kept that KEK would leave every secret it holds openable by whoever gets one of those volumes — the old disk carries the key, and the restored box's archives and logs carry values sealed under it. So restore_finish writes txlog-reseal and the restored box's first open runs twin::reseal_after_restore: mint a KEK (or take GUNNAR_SECRET_KEK when the box is handed one), durable as secrets.kek.new first; frame it (Kek); open every sealed value with the replayed key and seal it under the new one, each a Secret frame then a store write; rename over secrets.kek (or remove the replayed file when the key comes from the environment); remove the marker. A value that already opens under the new key is skipped, so a crash anywhere resumes. The twin following the restored box receives the new KEK and the re-sealed bytes in sequence, and a restore from it opens everything (and re-seals again under that box's own key).

What it does not do, said plainly: it does not ROTATE the secrets — the SSH host key and the issuer key are the same plaintext, as they must be for the restored box to be the primary it was. A leaked old volume still yields those plaintexts IF its holder also has a copy of the sealed values from before the restore (an old archive, the log before retention dropped it). Rotation is a separate, operator-driven act (new host key = clients re-trust), not part of a restore. Proof: twin::tests::a_restored_box_reseals_every_secret_under_a_kek_of_its_own_and_its_twin_follows_the_new_key (old KEK opens nothing on the restored box, second open does not re-seal again, the twin holds the new key, a restore from the twin opens every secret).

Retention

Segments roll at 256 MiB; truncate_before(seq) drops whole segments below seq. A primary may drop only what every configured twin has applied (applied_seq from each twin's LogHead), so a twin never meets a hole it did not cause by being away longer than the retention window.

The variant — CORRECTED 2026-09-17: the twin is a ROLE, not a medium

This section described a gunnar-twin medium from 2026-09-15 to 2026-09-17. That medium is deleted, and the correction matters more than the text it replaces.

gunnar-twin was gunnar-installer-open's [[build.installer]] row cloned with four lines changed. Cloning the KEYLESS row also cloned its variant: gunnar-server. gunnar.rs's appliance is gunnar-installer-metered, whose variant is gunnar-server-metered — and gunnar-recipe/tests/metered_symbols.rs guarantees, off the artifact's own symbol table, that gunnar-server contains no entitlement symbol at all. The twin exists to BECOME the primary. Promoted after a restore it would have served gunnar.rs from a binary with no plan enforcement in it: every suspended tenant live again, silently, with nothing on the box able to say why — because nothing on the box would have been wrong. The medium was, and a medium is read once, at mastering, months before the takeover that needs it.

The shape now (2026-09-22). One medium per LINEAGE, and the medium says nothing about what a box is. There is no role and no data_disks install parameter, no InstallRole, and no GUNNAR_ROLE env read: every box installs as a fresh appliance, the weakest is elected twin once (cargo xtask elect twin --apply, private-gunnar-ops) and made one over the wire by Bootstrap.BecomeTwin (twin.json). /data is korp-installer's own election (≥ 0.1.15: the system on the disk the firmware boots, data on the rest). The history of the 2026-09-17 → 2026-09-22 parameters is in gunnar-recipe/src/recipe.rs.

Does the role select a BINARY? No. --role twin is a runtime argv switch on the same binary — no twin feature, no twin symbol family. The axis that selects a binary is METERING, and it stays on the medium's variant. The community row installs a community primary or a community twin; the metered row installs a metered pair. The community build links no monetize_embed before this change and none after. The alternative — one medium carrying both binaries, the role picking which to lay — would put the money code ON the community medium, and "contains none of the entitlement families" is a claim about what is IN the artifact. Not widened.

The two ZNIPPY_* tokens are the role's setup rather than a medium's identity, and they stay cmdline tokens because znippy-plugin-git 0.1.4 has set_graph_cache_default and no explode twin of it (edition 2024 makes set_var unsafe; gunnar forbids unsafe). Keeping them in ONE place is the measured half: twin-memory-bound measured +36 MiB against the 64 MiB ceiling with them and +149 MiB without — over the ceiling. Open item: a set_explode_default(off) released in znippy would let a box that becomes a twin at RUNTIME close C1 the way it already closes C2, and make the token belt-and-braces instead of the only belt.

gunnar-recipe/tests/twin_is_the_same_medium.rs holds all of it, the metered defect by name. Everything below this box is the 2026-09-15 text, kept for the reasoning; read gunnar-twin as "the twin ROLE" and TWIN_KORP_ARGS as "--param role=twin".

The twin IS a build variant: gunnar-twin, a pointer variant of Atom::GunnarServer in Column::IsoInstaller with its own [[build.installer]] row (label GUNNAR_TWIN, argv serve,--grpc-bind,eth1:50051,--role,twin, korp.env.ZNIPPY_GIT_EXPLODE=off and korp.env.ZNIPPY_GIT_GRAPH_CACHE=off), emitted into .nornir/build-variants.toml from gunnar-recipe. What it is not is a new ATOM: it ships the same server binary as the primary (the stores the log applies into, and the gunnar control twin verbs), and what differs is the argv and the medium, not the assembly. Its disk is one HDD-tier volume laid by the installer's single-disk layout; growth on fill is stop → resize the one volume → start, and the xfs grows at boot. The day a distinct binary is wanted (to drop the git transport entirely) it becomes a real atom then, with its own bench. Detail: §"The variant, revised" below.

Inventory — what must not be lost, what is cache (verified against the code, 2026-09-15, Lane D2)

Line numbers are twin-holds-all as of this section. Nine classes the twin holds, eight classes it must never hold. "How the twin gets it" is the frame kind and the one place that writes it; "why never" is the rule that keeps it out.

MUST NOT LOSE — every one is a frame before it is a byte

CACHE — may be lost, and MUST NOT exist on the twin

Per box and never framed, and NOT cache (txlog::is_per_box_path): the box's own crash records and console/audit trail (crash/, log/audit-chain.jsonl; slow.rs:119-121, audit.rs:68) — a twin keeps its own, the primary's are never sent; the twin's own twin.json and twin.key (twin.rs:333, :337); and the restore markers txlog-restoring / txlog-fresh-docs.json. txlog::never_framed = cache or per box: the sink refuses to frame one, twin::under_root refuses to apply one. (Corrected 2026-09-15 after the first KVM scan: C7 had listed crash/ and log/ as cache. The scan found them, EMPTY, beside gc/ and quarantine/ — all four laid on /data by korp-installer's layout tree when a box has no /serialdat partition, korp-installer layout.rs:400-418. The rule is about files; empty directories are reported, not failed.)

What a twin's store directory holds, measured (the kept twin image of run t1-1216, repos/bench/big.git/gunnar-git.znippy/): objects.pack 201 388 684 B and objects.pack.journal 424 B (M1); refs.log 3 600 B (M2); objects.tail 1 589 248 B — the archive's own oid index, which put_refs checks every target against, so a twin without it could not apply a ref: part of M1, not optional; objects.graph 1 773 B — the commit graph the engine folds from the packs, DERIVED (C2), and closed as the next paragraph but one says; and objects.exploded 1 728 B, the eager explode pool (C1).

C2 on a twin — objects.graph (Lane K, 2026-09-15). The writer is znippy-plugin-git's refold (git_ops.rs persist_graph), so the switch was added there, on znippy branch graph-cache-switch, and released through dwarves crate-release: 0.1.3 reads ZNIPPY_GIT_GRAPH_CACHE (off neither reads nor writes the file, and an open removes a stale one), 0.1.4 adds set_graph_cache_default for a process that learns its role at runtime (edition 2024 makes set_var unsafe and gunnar forbids unsafe). gunnar uses both: the gunnar-twin medium bakes korp.env.ZNIPPY_GIT_GRAPH_CACHE=off (TWIN_GRAPH_ENV), and the server calls Engine::hold_no_graph_memo when its stores open as a twin and in mark_twin (become_twin) — so, unlike C1's medium-only closure, a box that becomes a twin at runtime holds no graph from its next store open. txlog::is_cache_path (and the drill's copy) counts objects.graph as cache, and twin::tests::no_cache_path_ever_enters_the_txlog_and_none_lands_on_the_twin holds the twin's data root to it.

C1 on a twin is closed by the medium: gunnar-twin declares korp.env.ZNIPPY_GIT_EXPLODE=off (gunnar-recipe TWIN_EXPLODE_ENV). znippy reads the policy at every store open (arms.rs ENV_EXPLODE), and ExplodePolicy::Off wants no object kind, so the file is never written (exploded_arrow.rs: the archive is only File::opened until a first write). Limit, said plainly: a box that became a twin at runtime on another medium keeps the default policy until it boots the twin medium.

C7 on a twin is closed in the server: the twin flag is runtime (ServerStores::mark_twin, set by become_twin, not only decided at open), twin::forget_mirrors removes the mirror copies a fresh box wrote at its first open (at open on a twin, and in become_twin), mirror_identity writes nothing on a twin, and ReplicationSlot::apply / WebhookSlot::apply save the file alone.

The native proofs: twin::tests::no_cache_path_ever_enters_the_txlog_and_none_lands_on_the_twin (the primary grows C1/C5, the sink refuses C4/C5/C6, every path-named frame is checked, a cache-naming frame is refused on apply, and the twin's data root holds no cache file) and a_restore_from_the_twin_rebuilds_every_object_and_every_document (M1–M8 byte for byte, including attestations.jsonl, webhooks.json and secrets.kek).

Twin mode, the fresh-box rule, and the two doors (2026-09-15, Lane D2)

Twin mode is <data-root>/twin.json. ServerStores::open then makes the log follower-only (Log::set_follower: append refuses with Error::Follower, only append_at writes), registers no sink, sets the browse flag (the control plane refuses mutators) and writes no identity mirror. A primary instead frames its KEK at seq 1 and registers its data root with the sink. serve on a twin reads its host key through the unlogged store, binds the forge listener to loopback, and starts the follower thread (twin::start_follower: twin.json

• twin.key, Mirror.LogHead/LogFetch over gunnar_grpc::client_signer, every 2 s).

The gate. Mirror.LogHead / LogFetch admit a member of group twin (gunnar_grpc::TWIN_GROUP) or a principal with the server grant — the twin's machine key reads the log without being an administrator of the box.

The fresh-box rule. A fresh appliance mints things at first start (issuer, host key, catalog seed) before anybody can tell it to take another box's numbers. twin::prepare_replay (refused unless no key is enrolled, the bootstrap window is open and no repository exists) records every document name in the box's own log in txlog-fresh-docs.json, makes the log follower-only and Log::resets it. The first replayed ControlDoc for such a name whose claim is "absent" is applied against the box's current value — the replay is into an empty store, and the fresh box's value is not a claim the primary ever made. Secrets need no rule (put is unconditional).

Bootstrap.BecomeTwin{primary} (unauthenticated, fresh boxes only — beside Enroll, under the same rule): mint twin.key, write twin.json, prepare_replay, freeze, start following; answer the public key to enrol on the primary as a machine in group twin. CLI: gunnar control twin join --at <new box> --primary <primary as the twin reaches it>.

Bootstrap.Restore(stream LogChunk) (unauthenticated, fresh boxes only, or a box already restoring — a broken stream resumes, entries below head + 1 are skipped, an entry the box holds part of resumes at the byte it holds: see "An entry larger than one call" above): restore_begin (prepare_replay + the txlog-restoring marker), append_at + apply per entry, then restore_finish. The box answers restart_required: it holds the fresh KEK in memory and the replayed secrets are sealed under the primary's; on the next open it IS the primary — same numbers, same host key, same issuer, same identity table. Caches (C1, C2) rebuild from the replayed packs as the engine opens them. CLI: gunnar control twin replay --twin <at> --into <fresh box> -i <admin key> pipes the twin's LogFetch into the fresh box's Restore.

The variant, revised (2026-09-15, Lane D2)

The twin is a variant and not a new ATOM — gunnar-twin is a pointer variant of Atom::GunnarServer in Column::IsoInstaller (gunnar-recipe/src/variants.rs) with its own [[build.installer]] row derived from the keyless one (recipe.rs, twin_installer): label GUNNAR_TWIN, argv serve,--grpc-bind,eth1:50051,--role,twin. --role twin keeps the forge listener off from the first boot; the primary is named at Bootstrap.BecomeTwin, because its address is not knowable when the medium is mastered. Single disk: the installer lays what the machine has, and the twin's machine has one volume. Emitted into .nornir/build-variants.toml by cargo run -p gunnar-build-thing -- emit.

§drill — the twin drill under KVM (Lane D2, 2026-09-15)

gunnar-growbench scenario T1 (crates/gunnar-growbench/src/twin_drill.rs), on this laptop under KVM/OVMF: a PRIMARY from gunnar-installer-open (4 × 4 GiB members) and a TWIN from gunnar-twin (10 GiB system disk + ONE 20 GiB data volume, as estate.toml), booted at once. The primary takes three repositories and an LFS object from the bench admin, user carol pushes her own repository, the fresh twin is joined with gunnar control twin join and its key enrolled on the primary as a machine in group twin, then a second wave (a repository and user dave) is followed to the watermark. The primary's disk images are DELETED, a fresh primary is installed on blank disks and pinned to the same forwards, gunnar control twin replay streams the twin's log into its Bootstrap.Restore, the box restarts, and every repository is cloned and git fsck --strict --full'd, every file sha256-compared, the LFS object pulled and compared, the identity rows compared, the twin watched following the restored primary, and the twin's data volume read from the host with xfs_db -r for cache files. Media composed from this branch through cargo xtask iso-installer --variant … --opt dhcp=on --opt wireguard=off (tunnr kernel via KORP_KERNEL, KORP_PID1_FEATURES=bench-halt), run with ~/scratch/twin-drill/run-t1.sh.

What the runs found, in order — each was a defect the native tests had not seen:

1. The rig: a twin has no git listener, so a readiness test on the SSH banner never fired, and the rig's 300 s boot hold ended the twin under the primary. Fixed in the bench.
2. A frozen twin refused Mirror.LogFetch (the browse replica's mutator refusal takes every non-Read level), and a restore stream that failed before its first entry was "finished" on an empty box. Fixed: Mirror.LogFetch is in BROWSE_STILL_ANSWERS; restore_finish refuses a restore that received nothing; twin replay keeps the stream open until the fetch succeeds.
3. A pushed pack was never a frame. A client's push enters Engine::ingest_push, which opens the engine by git_dir and never passes the OpenRepo::logged tee; its refs were framed and its pack was not. The twin's apply of those refs failed — and the follower, having appended before applying, resumed PAST the failed entry and reported applied == head with the refs missing. Fixed: before_store frames the exact spooled bytes inside both engines' ingest (M1); the follower and a resumed restore re-apply the log's head entry. Native proofs: gunnar-cli/tests/twin_push_is_framed.rs (stock git push → Pack before Refs → twin → restore) and twin::tests::an_entry_whose_apply_failed_is_applied_again_before_anything_newer_not_skipped.
4. The first scans flagged the empty gc/, quarantine/, crash/, log/ links korp-installer lays to /data/.serialdat — directories and links, not files. The scan counts regular files.

Run t1-1216 (commit 786bc63, media sha256 of gunnar-installer-open.iso in the log):

VERDICT T1a the twin follows the primary to the watermark, two waves: PASS — wave 1 head 121 (234 241 943 B of log) applied 121 caught_up true in 8 s | wave 2 head 147 applied 147 caught_up true in 2 s
VERDICT T1b restore from the twin into a re-imaged primary: every repo clones + fsck-clean, LFS identical: PASS — files 21/21 sha256-identical, repos 3/3 cloned+fsck-clean; carol/notes 2/2, 1/1; lfs-assets fsck-clean, asset.bin (3 MiB, LFS) sha256-identical
VERDICT T1c users back: PASS — 5 identity rows identical before and after
VERDICT T1d entitlements back: UNRUN — the keyless medium refuses Entitlement.Set (community build, no monetize key); the fact is an attestations.jsonl line (Append) and the document a ControlDoc, both replayed byte for byte in the native restore test
VERDICT T1e the twin follows the restored primary: PASS — applied 147 = primary 147, caught_up true
VERDICT T1f a scan of the twin's disk finds no cache file: FAIL — repos/*/gunnar-git.znippy/objects.exploded (one per repository), .serialdat/meta/identity.json, .serialdat/meta/replication.json

1. Cache on the twin (run t1-1216 T1f): an objects.exploded eager pool in every repository the twin applied, and the fresh box's own meta/identity.json and meta/replication.json from before it became a twin. Closed as the C1 and C7 paragraphs above say (commit 3fc34c4).

Run t1-1235 (commit 3fc34c4; gunnar-installer-open.iso sha256 34cbdd97767bcf00…, gunnar-twin.iso sha256 404af9c0c8737cda…) — the drill passes:

VERDICT T1a the twin follows the primary to the watermark, two waves: PASS — wave 1 head 121 (234 241 945 B of log) applied 121 caught_up true in 8 s | wave 2 head 154 applied 154 caught_up true in 4 s
VERDICT T1b restore from the twin into a re-imaged primary: every repo clones + fsck-clean, LFS identical: PASS — files 21/21 sha256-identical, repos 3/3 cloned+fsck-clean; carol/notes 2/2, 1/1; lfs-assets fsck-clean, asset.bin (3 MiB, LFS) sha256-identical
VERDICT T1c users back: PASS — 5 identity rows identical before and after (bench, carol, dave, twin-box …)
VERDICT T1d entitlements back: UNRUN — the keyless medium refuses Entitlement.Set (community build, no monetize key); the fact is an attestations.jsonl line (Append) and the document a ControlDoc, both replayed byte for byte in the native restore test
VERDICT T1e the twin follows the restored primary: PASS — applied 154 = primary 154, caught_up true
VERDICT T1f a scan of the twin's disk finds no cache file: PASS — 47 files in 30 directories on the twin's one data volume; 0 cache files (installer-laid links/dirs with nothing under them: /gc, /quarantine, /.serialdat/{meta,gc,quarantine})
VERDICT T1 twin drill (follow, destroy the primary, restore from the twin): PASS — T1a PASS, T1b PASS, T1c PASS, T1d UNRUN, T1e PASS, T1f PASS

What this does NOT prove, named: entitlements under KVM (T1d, needs the metered medium and a monetize key); objects.graph (1.7 KB derived commit graph) is still on the twin, not flagged by the file rule above; the snapshot for a twin that meets a retention hole; the KEK re-seal into a fresh box's own KEK (a restored box takes the primary's KEK, as Kek frame seq 1 carries it); LfsDelete and RepoStore purge frames are kept but not applied; runtime fill-growth of the twin's volume (L4 d).

The loose ends closed, and the drill again (Lane K, 2026-09-15, branch twin-loose-ends)

The "does NOT prove" list above, item by item: objects.graph is off on a twin and counted as cache (C2 paragraph); the snapshot for a twin behind a retention hole is built (§"Snapshot"); a restored box re-seals under its own KEK (§"The KEK after a restore"); LfsDelete and RepoStore purge frames are written and applied (M3, M9). Still open: T1d under KVM (metered medium), runtime fill-growth of the twin's volume (L4 d), and the snapshot under KVM (the drill does not force a retention hole).

Run t1-1547 (commit 6340ccc; gunnar-installer-open.iso sha256 8b521af7d5e0030c…, gunnar-twin.iso sha256 100444356c10a806…, composed with cargo xtask iso-installer --variant … --opt dhcp=on --opt wireguard=off, tunnr kernel via KORP_KERNEL, KORP_PID1_FEATURES=bench-halt). New row T1g: bench/doomed is pushed in wave 1, repo delete --purge --yes in wave 2, and must not come back.

VERDICT T1a the twin follows the primary to the watermark, two waves: PASS — wave 1 head 129 applied 129 caught_up true in 10 s | wave 2 head 160 applied 160 caught_up true in 4 s
VERDICT T1b restore from the twin into a re-imaged primary: every repo clones + fsck-clean, LFS identical: PASS — files 21/21 sha256-identical, repos 3/3 cloned+fsck-clean; carol/notes 2/2, 1/1; lfs-assets fsck-clean, asset.bin (3 MiB, LFS) sha256-identical
VERDICT T1c users back: PASS — 5 identity rows identical before and after (bench, carol, dave, twin-box …)
VERDICT T1d entitlements back: UNRUN — the keyless medium refuses Entitlement.Set (community build, no monetize key)
VERDICT T1e the twin follows the restored primary: PASS — applied 160 = primary 160, caught_up true (the restored box's re-seal frames included)
VERDICT T1g a repository purged on the primary is not resurrected by the restore from the twin: PASS — bench/doomed absent from the restored catalog (5 rows) and its clone refused
VERDICT T1f a scan of the twin's disk finds no cache file: PASS — 42 files in 30 directories; 0 cache files, objects.graph now counted (installer-laid links/dirs only: /gc, /quarantine, /.serialdat/{meta,gc,quarantine})
VERDICT T1 twin drill (follow, destroy the primary, restore from the twin): PASS — T1a PASS, T1b PASS, T1c PASS, T1d UNRUN, T1e PASS, T1g PASS, T1f PASS

§rescue — the rescue estate under KVM (lane LIFECYCLE, 2026-09-16)

The drill above proves the happy rescue. A rescue is not a happy path, so four more phases were added to the SAME estate H1 and R6 already install — one primary, one twin, five verdicts, one install. gunnar-growbench --scenarios H1,R8,R10,R9,R6 --iso <medium> --blank-system (the twin installs the same --iso and is made a twin by twin join; --twin-iso is retired), and the whole appliance life is --scenarios LIFECYCLE.

R8 closes what DATA-SET-GROWTH-DESIGN.md §20 calls the RETENTION HOLE — "nothing in gunnar-growbench trims the log behind a stopped twin and catches it up through SnapshotFetch … that drive belongs in growbench beside T1" — and what §"Still open" above called "the snapshot under KVM (the drill does not force a retention hole)". It is forced now.

Two corrections to this document, measured.

1. §Retention says "Segments roll at 256 MiB" and that a primary "may drop only what every configured twin has applied". Neither is what Mirror.LogRetain does: DEFAULT_SEGMENT_BYTES is 1 GiB, and control_plane::log_retain has no twin clamp at all — it rolls a segment and calls truncate_before, whose own log line says "a twin behind it takes a snapshot". The verb is the operator's override, which is exactly why R8 can use it to make the hole.
2. truncate_before drops a segment only when its LAST sequence is below from, so retain --from head moves nothing. --from head + 1 — what --from 0 resolves to on the server — is the one that trims.

A defect this estate found in gunnar, not in the bench: a box that has just restarted out of a restore answers SSH before its control plane is listening. R6 (e) failed on transport error while the twin, moments later, found the same box serving; every restored secret is re-sealed at open (§"The KEK after a restore") and that takes longer the more the box was given back. The bench now waits for the control plane up to 180 s and reports the wait as a number in the verdict, so a box that reopens slowly is measured rather than flaky.

What is NOT proven here. The other half of the replay-abort contract — that a stream ending cleanly BELOW the announced LogChunk.restore_head is not a restore and the box stays restoring — cannot be read from the client: an older server ignores the unknown field and finishes the short stream, which looks identical from outside. It is proven by the pump_into unit tests in gunnar-cli/src/control/twin.rs and would need a door reporting the box's restore state to be proven under KVM. The media used for these runs are composed from 0bd1779, which predates restore_head entirely.

The owner's question, answered class by class (2026-09-17, lane TWIN-COMPLETENESS)

Does the twin hold everything needed if the primary VM is lost — not just the git objects, but the configuration and state? Does a restore bring back a working box, or a box with the repos and none of what makes it itself?

It holds all of it, and until this lane nothing PROVED most of it. The inventory above (M1–M9) is right; the tests under it were not. The fixture in twin::tests::primary wrote two packs, a ref set, one principal, one generic secret, one attestation line and one document — and nothing else. It never minted an SSH host key, never created an issuer, never made a group, a membership edge, a grant, a second namespace, a visibility, a default branch or an entitlement fact. So assert_same — the function named "byte for byte, name for name: what a restore has to reproduce" — could not fail for any of them. A restore that lost the issuer passed.

What is replicated, how, and where it is now proven

The brick this lane found, measured

A fresh box's own secret refused the restored box's every open, for ever. A fresh box seals its secrets under the KEK it minted at its own first open. The replay's Kek frame — seq 1 — replaces that key on disk before any Secret frame lands. A row the replay OVERWRITES is fine; a row it has no frame for is left sealed under a key that no longer exists, and reseal_after_restore met it with a ?:

  re-sealing a restored box's secrets under its own KEK: opening fresh-only/key with the
  replayed KEK: … wrong key-encrypting key, wrong secret name, or modified bytes

The box had been restored successfully and then could not start — and could not start on the next attempt either, because the re-seal marker stays until the re-seal succeeds. prepare_replay remembered the fresh box's DOCUMENTS (FRESH_DOCS) and gave up its numbers, and kept its secrets.

Closed both ways. prepare_replay now drops the fresh box's own secrets with its own numbers — the box is FRESH by its own checks (bootstrap window open, no enrolled identity, no repository), so those are a host key it minted at first boot and an issuer it minted at first start, never anybody's data. And reseal_after_restore drops a row that opens under NEITHER key loudly and by name rather than refusing the open, so a box restored by an older build still starts; the value is unrecoverable by construction, so keeping it buys nothing but the brick. twin::tests::a_fresh_boxs_own_secret_does_not_brick_the_box_it_is_restored_into asserts both.

reseal_after_restore covers every SEALED row in the secrets table — it iterates the whole table, skips a row that already opens under the new key and a legacy plaintext row, and frames the new KEK before the first value moves. So the host key and the issuer key come back openable under a key the dead primary never held. The one row it could not open — the fresh box's own — is the brick above, now dropped by name at both ends.

What the twin deliberately does NOT hold, and it is right

• log/audit-chain.jsonl and crash/ — is_per_box_path. A restored box starts a new audit chain; the dead primary's is gone with its disks. This is a real loss of a tamper-evident record and it is a DESIGN decision, not an oversight: the chain is a record of what happened ON A BOX, and a twin keeps its own.
• grow-journal — is_cache_path, per box, on the boot disk. A restored box's growth history is its own.
• gunnar-ui's configuration — out of scope by ruling (2026-09-17): the front is declared in private-gunnar-ops/estate.toml and reproduced by cargo xtask front. The spec is its backup, and a spec is better than a copy because it is the source.

The gap this lane found in the PRODUCT

No control-plane read of the PKI document exists. gunnar issuer show takes --data-root, the appliance has no shell, and the issuer certificate is returned only by Bootstrap.Enroll — which a restored box can never answer again. So an operator cannot ask a running appliance which CA it mints with, and the KVM drill cannot compare the issuer across a restore. T1t is UNRUN and says so by name. The fix is a Pki.Show-shaped read; ServerStores::pki_readonly() already exists for exactly this and refuses every secret-store call by construction.

T1d (entitlements) is not a medium problem. There is no gunnar control entitlement verb at all, and Entitlement.Set takes a fact signed with monetize's PRIVATE key — the appliance carries only the public half, baked at compose time. A metered medium alone does not close T1d; it needs the bench to hold a monetize signing key and something to push a fact with.

Restoring onto DIFFERENT HARDWARE (2026-09-17, lane TWIN-COMPLETENESS)

The owner's case: eight devices where there were four, or two. The data set is md RAID0 over its members, so the question is whether a restore assumes the geometry it left.

It does not, and by construction rather than by luck. The two halves are independent:

• the installer lays the set for the machine it is on. korp_installer::dataset::plan_set discovers the stripe — the largest group of mutually equal eligible disks — from the disks the machine actually has; layout::pick_boot takes the remainder. Nothing reads a previous member count, and InstallReport is checked member by member (korp-data-<i>) in the bench.
• the restore streams FRAMES into a filesystem, never blocks into a member layout. Bootstrap.Restore carries Pack, Refs, ControlDoc, Secret, File, Append, Kek — the same entries the log carries. A pack lands through put_pack, a document through commit_doc. There is no image, no partition table and no member count anywhere on that path.

So 4 → 8 works: more room than the set left with, and the set is laid for eight. T2 is that run under KVM.

4 → 2 is the direction that could lose data quietly, and it could. Two devices may hold less total space than the set being restored, and nothing checked. The stream filled the volume and the apply died somewhere in the middle — recoverable, because the box stays restoring and nothing says "restored", but the operator learned it an hour in and learned it as an ENOSPC rather than as a number.

Closed here. LogChunk.restore_bytes (additive, field 8) carries what the sender is about to stream — the twin's LogHead.bytes. Bootstrap.Restore checks the box's own statvfs against it ONCE, on the first chunk, before any apply, and refuses naming the shortfall:

  NOT ENOUGH ROOM for this restore, and nothing has been written: the sender streams 1.2 GiB of
  log, which needs about 2.7 GiB on this box (2× the log — a restore writes every frame into this
  box's own log AND into the store it builds — plus 256.0 MiB the box keeps for its caches and
  slack), and this data set has 1.9 GiB free of 2.0 GiB. SHORT BY 836.0 MiB. …

restore_bytes of 0 means the sender did not say (an older client, or a snapshot whose size it cannot know in advance): no refusal, because a guess is not a measurement. An unmeasurable statvfs is not "full" either. The factor of 2 is named as an ESTIMATE in the refusal itself — a tight number would need the sender to say what its STORES measure rather than what its LOG measures, and it is a floor, not a promise: a set that only just fits can still run out mid-stream, and that failure is the ordinary resumable one.

T3 is the KVM proof and it is written red first — the set is made too small and the log too big on purpose, the refusal must name the shortfall, and the box's bootstrap door must still be OPEN afterwards, which is what proves no frame landed before it said no. A run that cannot create that situation says UNRUN with the numbers, never PASS.

And a defect the geometry work walked into

The gunnar-twin medium cannot install the estate's twin shape. estate.toml [machines.twin] is a 10 GB system disk beside a 20 GB data volume. Two unequal disks means no group, so pick_boot_without_a_group decides by class and then by the LARGEST — and the largest is the DATA volume:

  INSTALLER: data set: boot vdb (20GiB) · data 1×10GiB plain [vda (blank)]
  boot.img p1 is "korp-data-0", not the ESP

korp-installer names the answer itself: "no size heuristic can serve both … where the sizes are silent the layout has to be DECLARED" — korp.data.disks= on the medium's command line. The gunnar-twin variant's installer row carries none (its baked cmdline is serve,--grpc-bind, eth1:50051,--role,twin and nothing about disks). Until it does, a twin must be built with its system disk LARGER than its volume. The bench now does that and refuses by name rather than mis-install when handed the other shape; the drill's old comment said the opposite, which was true of the 2026-08-29 installer and is the exact inverse of today's.

Run t1c-1139 — the drill with a row per class

Media: gunnar-installer-metered (GUNNAR_INSTALL_METERED, sha256 2b5e1721be6d94df…) and gunnar-twin, both composed 2026-09-16 from v1.4.0-220-g0bd1779 — i.e. BEFORE the room check and the fresh-box-secret fix in this branch, which is why T3 cannot be green on them. Primary 4 × 4 GiB, twin 10 GiB system + 8 GiB volume, 209 frames / 245 MB of log.

VERDICT T1  twin drill: PASS — T1a PASS, T1b PASS, T1c PASS, T1d UNRUN, T1e PASS, T1g PASS,
            T1f PASS, T1h PASS, T1i PASS, T1j PASS, T1k PASS, T1l PASS, T1m PASS, T1n PASS,
            T1o PASS, T1p UNRUN, T1q PASS, T1r PASS, T1s UNRUN, T1u PASS, T1t UNRUN
VERDICT T1h the SSH host key: SHA256:QfduN0/2YsnQLtTldZKNqNuC8kvitjx/v+urnBpkm1o — identical
            before and after; no client sees a changed fingerprint
VERDICT T1r repository rows back: bench/mixed hash sha256 · branch refs/heads/trunk · access
            public ; bench/big … branch refs/heads/main · access private
VERDICT T1u the set LEFT 4 × 4 GiB and LANDED on 4 × 4 GiB; the installer laid korp-data-0..3

The four UNRUN rows, each naming what it needs: T1d a monetize signing key and something to push a fact with; T1p a medium built with the webhooks feature; T1s a pull mirror the drill could seed; T1t a control-plane read of the PKI document, which does not exist.

Three reds the first two runs found were all the BENCH's, and each is worth keeping written down: a default branch pointed at a ref that was never pushed (every clone then checks out nothing); "not implemented" is how the server spells a missing feature, not "unimplemented"; and replication list on a box with no standby answers "no standby configured" — identical before and after, a green row about nothing, which is why a class must now be SEEDED before it may pass.

Run t2-1143 — 4 → 8, and it works

Same media, same drill, the fresh primary installed on EIGHT members where the set left with four.

VERDICT T2u the set LEFT 4 × 4 GiB (16 GiB raw) and LANDED on 8 × 4 GiB (32 GiB raw); the
            installer laid korp-data-0 … korp-data-7 for the machine it was on:
            data set: boot vda (50GiB) · data 8×4GiB raid0 [vdb … vdi]
            grow: nothing to do — 8 members, every device sized for the 4.0GiB each the set uses
VERDICT T2  restore onto MORE devices than the set left with (4 → 8): PASS — T2a PASS, T2b PASS,
            T2c PASS, T2d UNRUN, T2e PASS, T2g PASS, T2f PASS, T2h PASS, T2i PASS, T2j PASS,
            T2k PASS, T2l PASS, T2m PASS, T2n PASS, T2o PASS, T2p UNRUN, T2q PASS, T2r PASS,
            T2s UNRUN, T2u PASS, T2t UNRUN

209 frames replayed whole onto a stripe of a different width, every repository cloned and fsck-clean, the LFS object identical, the host key unchanged, every class of configuration back. The geometry is not part of what is restored, and this is the measurement that says so.

Run t3-1146 — 4 → 2, the RED the scenario was written for

The set left 4 × 4 GiB and lands on 2 × 1 GiB. The twin holds 1.21 GiB of log, which a restore needs about 2.68 GiB for; the set is 2.00 GiB raw. The media are the 2026-09-16 pair, i.e. gunnar v1.4.0-220-g0bd1779 — a build from BEFORE LogChunk.restore_bytes, so the box has nothing to check against and cannot refuse. This is the shipped behaviour, measured:

VERDICT T3a the twin follows the primary to the watermark: PASS — head 120, 1 303 067 942 B of log
VERDICT T3d the installer laid the small set for the machine it was on: PASS — 2 members laid
            (korp-data-0, korp-data-1) · data set: boot vda (50GiB) · data 2×1GiB raid0 [vdb, vdc]
VERDICT T3b a restore onto a set too small for it refuses by name: FAIL — the replay failed, but
            NOT with the room refusal:
              Bootstrap.Restore: internal error: restoring seq 65: seq 65: pack into
              bench/fill-1: storing the pack verbatim: uring: op 0xf500 failed:
              Operation canceled (os error 125)
VERDICT T3c the refusal comes BEFORE the first byte lands: FAIL — the door is no longer open
            after a REFUSED restore (bootstrap closed — enrolled … by "bench"): something of
            the twin's log landed before the box said no

Two things to take from it, and the second is worse than the first.

1. The failure does not say "out of space". The volume filling surfaces through io_uring as Operation canceled (os error 125) — a linked chain cancelling its tail — not as ENOSPC. An operator reading that message has no reason to look at the disk at all.
2. The box had already closed its bootstrap door. The identity table landed somewhere below seq 65 of 125, so from outside the half-restored box reads bootstrap closed — enrolled … by "bench": a box that looks enrolled and holds half a forge. It IS still restoring — nothing said "restored" and the replay is resumable onto a bigger set — but "looks enrolled" is exactly the shape somebody would mistake for a finished restore.

With LogChunk.restore_bytes and twin::room_for_restore (this branch) the box refuses on the FIRST chunk, before restore_apply is reached, naming the shortfall — and the bootstrap door stays OPEN, which is what T3c asserts. T3 goes green only on a medium composed from a build that carries the check; that compose is the handover item. The rule itself is proven natively by twin::tests::a_restore_onto_a_data_set_too_small_for_it_refuses_before_the_first_byte_and_says_how_much_is_short.

Run t3new-1232 — the medium composed from this branch, and T3 is GREEN

The handover item above is done. Medium gunnar-installer-metered composed 2026-09-17 from a REAL CLONE of twin-completeness (/home/rickard/compose-clean/gunnar-t3, git describe v1.4.0-321-g351c792), kernel built from source in the pinned container nordisk-kernel-toolchain@sha256:78eaa55a… (--build kernel), sha256 f72c442341b7579305793844ad61921e673acd2b48677dff039883d5ee95027f. The banner carries the commit — SSH-2.0-gunnar_1.4.0 v1.4.0-321-g351c792 — and the built .config has CONFIG_HOTPLUG_PCI, _PCIE and _ACPI all y. The TWIN medium is the 2026-09-16 gunnar-twin unchanged, and that is correct: restore_bytes is filled by the CLI from the twin's LogHead.bytes, which every build already serves, and checked by the RECEIVING box. The CLI is the third piece and was rebuilt from this branch.

Same geometry and the same 1024 MiB fill as t3-1146, so the two runs are comparable.

VERDICT T3a the twin follows the primary to the watermark: PASS — head 125, 1 303 069 271 B,
            applied 125 · primary 125 · caught_up true, following in 51 s
VERDICT T3b a restore onto a set too small for it refuses by name, saying how much is short:
            PASS — the set LEFT 4 × 4 GiB (16.0 GiB raw, 15.6 GiB free) and LANDS on
            2 × 1023 MiB (2.00 GiB raw, 1.9 GiB free). The box refused, by name:
              NOT ENOUGH ROOM for this restore, and nothing has been written: the sender
              streams 1.2 GiB of log, which needs about 2.7 GiB on this box (2× the log …
              plus 256.0 MiB … ), and this data set has 1.9 GiB free of 1.9 GiB.
              SHORT BY 793.3 MiB.
            The audit line on the box is the other half: announced_bytes=1303070606
            decision="deny" — "Bootstrap.Restore refused for room; nothing of the stream was
            written".
VERDICT T3c the refusal comes BEFORE the first byte lands: PASS — "bootstrap OPEN — no identity
            has ever been enrolled here" AFTER the refusal. On the shipped medium this row read
            "enrolled … by bench" with 64 of 125 frames landed; that is the shape somebody would
            mistake for a finished restore, and it is gone.
VERDICT T3d the installer laid the small set for the machine it was on: PASS — 2 members laid
            (korp-data-0, korp-data-1) · data 2×1GiB raid0 [vdb, vdc]
VERDICT T3  restore onto FEWER, SMALLER devices (4 → 2): PASS — T3a PASS, T3b PASS, T3c PASS,
            T3d PASS · 1 of 1 PASS · 0 FAIL · 0 UNRUN

Two things the green says that the red could not. The failure is now an ENOSPC-shaped fact stated in bytes — SHORT BY 793.3 MiB — where the shipped medium gave an operator uring: op 0xf500 failed: Operation canceled (os error 125) and no reason to look at the disk. And the box is left genuinely fresh: door OPEN, log empty, so the same replay into a bigger set starts at seq 1 and loses nothing.