nordisk/gunnar

TWIN DESIGN AUDIT — 2026-09-21 (lane T1)

Read of the code at master ac02956, plus branch agent/stink-leaks (4d38319, the follower-supervision fix, NOT on master). Nothing here is quoted from a design doc without a file:line behind it — several recorded claims in this estate have turned out stale, so every answer below is read off the source.

Scope note. The owner ruled mid-audit: "gunnar twin must be built EXACTLY like the main, it MUST use the SAME iso" and "DO not build a special gunnar TWIN iso". §7 audits the code against that target. The un-metered-twin question is closed by that ruling and is not argued here.

The short version. The restore path EXISTS, end to end, and it has a name: gunnar control twin replay --twin <twin>:50051 --into <fresh>:50051. It has been driven under KVM and reached its last step. That is the good news and it is the thing the owner asked about. The design is sound in its core and wrong in three specific places: the twin can never be the recovery target (§5), the pairing has no notion of WHICH log it follows (§2, D1), and nothing refuses a restore from a stale twin (§4, D2).

1. What does the twin actually hold?

The twin holds the primary's transaction log, byte for byte, at the primary's own sequence numbers, and applies every frame through the UNLOGGED halves of its stores so it originates nothing (crates/gunnar-server/src/twin.rs:23-27, apply_raw at twin.rs:210).

Replicated — the twelve frame kinds, gunnar-core/src/txlog.rs:101-146:

Entitlements are a control document (gunnar-core/src/entitlement.rs:49, ENTITLEMENT_REF), so the tenant/entitlement state IS replicated. That matters for §7.

NOT replicated, by construction (txlog.rs:1420-1446):

• is_cache_path — gc/, quarantine/, meta/, objects.exploded*, objects.graph, serialdat/, .tmp, incoming-*, and grow-journal. The last one is real state: the data-set growth journal is per box and a restored box starts with none.
• is_per_box_path — crash/ and log/: the box's crash records and its console/audit trail. A restored main comes back with NO audit history. Nothing warns about this.
• The metadata mirror dir is actively DELETED on a twin at join (twin.rs:671-688, forget_mirrors) — identity mirror, replication.json/webhooks.json mirror copies. They are derived, and the File frames carry the real ones, so this is correct.
• Everything outside the data root: boot params (GUNNAR_NET, static_ip, hub.conf), /etc/gunnar-clock/time_ed25519, the medium's own configuration. These are install parameters, not data — they come from the medium, not from the twin.

Does any absence make a restore incomplete or silently wrong? Three:

1. log/ — the audit trail is silently gone. Every other class is either replicated or re-derivable. The audit/console trail is neither, and the restore reports success without mentioning it. Rank: D6.
2. grow-journal — a box mid-growth when it died restores with no journal. Deliberate (is_cache_path), but undocumented as a restore consequence. Rank: D7.
3. The entitlement facts survive but their VERIFIER does not. EntitlementSlot::open re-verifies every stored fact against --monetize-pubkey and, on a fact that does not verify, logs entitlement.boot.refused and leaves the namespace Free (gunnar-server/src/entitlement.rs:311-324). It is a tracing::error!, not a ?. So a restore onto a box handed a DIFFERENT monetize key silently downgrades every paying tenant to Free, and the restore still says "restored". Rank: D3 — this is the "incomplete or silently wrong" case the question asks for.

Positive finding worth recording: a fresh box's OWN secrets are given up at prepare_replay/forget_own_secrets (twin.rs:526-545) and the unopenable-row case is dropped loudly rather than bricking the box (twin.rs:919-952). Both were measured failures; both are now handled. The KEK is re-minted at reseal_after_restore (twin.rs:881) so the dead primary's key leaves the disk, and each step is framed so the twin following the restored box gets the new key.

2. The pairing

Join. gunnar control twin join --at <twin> --primary <main> → Bootstrap.BecomeTwin (gunnar-cli/src/control/twin.rs:77-88, gunnar-grpc/src/service/bootstrap.rs:161-187) → twin::become_twin (twin.rs:644-664). It mints twin.key (ed25519, 0600, per box, never framed), writes twin.json, runs prepare_replay (which refuses unless the box is FRESH: bootstrap window open, no enrolled key, no repository — twin.rs:485-506), marks the box a twin, forgets its mirrors, sets browse and writes_frozen, and returns the public key line. The operator then enrols that key on the primary as a machine principal in group twin.

Who authenticates whom.

• Twin → primary: signed. Every Mirror.LogHead / LogFetch / SnapshotFetch is an sshsig from twin.key (twin.rs:1267, 1274, 1304). The primary authorises via require_log_reader (gunnar-grpc/src/service/mirror.rs:36-54): membership of group twin is sufficient and bypasses the grant check entirely.
• Primary → twin: NOTHING. The twin verifies no identity of the primary at all. The channel is plain_channel (twin.rs:1221) — no TLS, no pinning, no host key. The twin trusts whatever answers at the host:port in twin.json.
• Bootstrap.BecomeTwin and Bootstrap.Restore are UNAUTHENTICATED RPCs (gunnar-grpc/src/auth.rs:1832-1833). Their only guard is that the target box is fresh. That is the same trust model as Bootstrap.Enroll and is defensible — but it means the CONTENT of a restore is unauthenticated: anyone who can reach a fresh box's control port can stream it an arbitrary log, and it comes up as a "primary" with attacker-chosen admin keys and KEK.

Re-join. Refused. become_twin is idempotent for the SAME primary (returns the same key) and errors for any other: "this box is already the twin of X; a twin is never re-pointed" (twin.rs:647).

Key rotation. There is no twin-key rotation verb. twin.key is minted once at join and never rotated (twin.rs:652-657). Rotation is done on the PRIMARY side by removing and re-enrolling the principal — that is what gunnar-cli/tests/key_rotation.rs:8-21 records having been done by hand on the live appliance. Rank: D8.

A re-imaged TWIN. Works, at a price. The new box is fresh, become_twin succeeds, a NEW key is minted and must be re-enrolled on the primary. Its head is 0, so if the primary has trimmed retention at all, Follower::step takes the first > wanted branch and pulls a whole Mirror.SnapshotFetch — which holds Log::freeze() for the ENTIRE copy (twin/snapshot.rs:216). On a 44 GiB estate the primary's framed writes are stopped for the whole transfer. There are stall deadlines on both halves (mirror.rs:506, control/twin.rs:564-585), so it cannot hang for ever, but re-imaging a twin is a production-freeze event and nothing says so at the door.

A re-imaged MAIN — this is the defect. twin.json pins an ADDRESS and nothing else. LogHeadResponse carries head_seq, first_seq, bytes, twin, applied_seq, primary_head_seq, caught_up, state, why (gunnar-proto/proto/gunnar/v1/mirror.proto:192-211) — no log identity, no box UUID, no epoch. So:

• a main restored FROM THIS TWIN continues at the twin's own numbers, and the twin follows on correctly — this is the designed happy path and it works;
• a main re-imaged FRESH at the same address (wrong order of operations, or restored from a different source) starts at seq 1. The twin holds head N. Follower::step computes wanted = N+1 > primary_head, the fetch loop never runs, and it returns Progress::CaughtUp (twin.rs:1146-1179). The twin reports state following and simply waits. When the new main's head passes N, the twin will begin appending the NEW box's frames at seq N+1 on top of the OLD box's state — two histories spliced, silently. The gap rule protects against holes; nothing protects against a different log with the same numbers.

Symmetry. None. The relationship is entirely one-directional: the twin pulls, the primary answers and records a heartbeat row. The primary has no notion of "my twin" beyond group membership, and any member of group twin may pull the whole log — which is every sealed secret plus the KEK (mirror.rs:36-54, and mirror.proto:147 says so out loud). Group twin is, in practice, root on the box. The design does assume main is immortal in exactly one sense: it assumes the box at that address is always the same box.

3. The restore path — the thing the owner actually wants

It exists. The verb is gunnar control twin replay.

gunnar control twin replay --twin <twin>:50051 --into <fresh-main>:50051 [-i KEY] [--ignore-room]

gunnar-cli/src/control/twin.rs:90-113 (replay) and :154-217 (pump_into). Traced concretely:

1. replay reads the twin's Mirror.LogHead as the operator (control/twin.rs:95-98).
2. If the twin's log starts above 1 (it installed a snapshot once), pump_into first drains the twin's own Mirror.SnapshotFetch — items with seq 0, SnapshotAt last (control/twin.rs:165-176).
3. Then the twin's log first..=last in ranges of 32 MiB (control/twin.rs:136), every chunk stamped restore_head = last and restore_bytes = head.bytes.
4. Both are streamed straight into the fresh box's unauthenticated Bootstrap.Restore (service/bootstrap.rs:189-276). The fresh box runs restore_begin (twin.rs:707-726: refuses a twin, requires FRESH, writes the txlog-restoring marker), checks room ONCE before the first byte lands (twin.rs:765-805, bootstrap.rs:209-218), then append_at + apply per entry, durable first (twin.rs:824-827).
5. restore_finish (twin.rs:837-856) refuses an empty stream and refuses a stream that ended below the announced head — a SHORT stream is not a restore and the box stays resumable. It writes the txlog-reseal marker before clearing txlog-restoring, so no open can see "restored" without "re-seal".
6. Answer: restart_required: true. The box restarts and reseal_after_restore (twin.rs:881-977) re-seals every secret under a KEK of its own, framing each step so the next twin gets them.

A broken fetch mid-stream poisons the request body rather than cancelling it (control/twin.rs:150-153, Frame::Poison), because tonic reads a client RST_STREAM(CANCEL) on a request stream as a CLEAN end — that is exactly how this verb used to hand a fresh box a short stream it then finished as a restore. Good fix; worth keeping.

What restore_drill.rs proves. crates/gunnar-growbench/src/restore_drill.rs scenario R6 (:38-45, R6_NAME at :457) is the real thing, under KVM, in Rust: the primary's data disks are destroyed and the medium re-installed onto its system disk; the box must read FRESH (bootstrap OPEN, the old admin refused); gunnar control twin replay streams the twin in; the box restarts only if the answer said so; and then every repository clones fsck-clean with every file sha256-identical, the user list and the repository list equal what the primary held, and the twin follows the restored primary. Beside it: R9 proves an interrupted replay RESUMES rather than starting over (:373), R10 proves a primary killed mid-push leaves no listed-but-unclonable repository, R8 proves a retention hole is closed by a real snapshot (the twin's own first_seq must move off 1, which only Log::restart_after does).

What it ASSUMES. Three things, and they are named in the source:

• the media it installs are composed from 0bd1779, which predates restore_head entirely (.nornir/twin-transaction-log.md:466-467), so the short-stream refusal is NOT proven under KVM — only by pump_into unit tests (restore_drill.rs:27-36);
• it assumes two media — --iso and --twin-iso (growbench/src/main.rs:7,73,170). Under the one-medium ruling this becomes one --iso plus a role param. See §7.
• it has never run in the estate. The twin box is dark.

The one thing that does NOT exist: an orchestrating verb. restore_drill.rs:38 describes itself as "the steps cargo xtask restore main takes". cargo xtask restore does not exist — there is no restore arm in xtask/src/main.rs and no restore.rs in xtask/src/. The re-image half is cargo xtask reinstall-appliance (xtask/src/reinstall.rs); the feed half is gunnar control twin replay; nothing joins them, nothing checks the order, nothing checks that the twin was current before the data disks were destroyed. The bench does that orchestration in its own Rust and no shipped verb does. Rank: D4.

So the honest answer to the owner: yes, main can be fed from the twin, the command is gunnar control twin replay, and it has been driven to its last step under KVM. What is missing is a single verb that does the whole rescue in the right order, and a check that the twin was current when you started.

4. Liveness and lag

Two halves, both real.

• Primary's view. Every twin poll carries a TwinReport — applied head, statvfs of the data root, the follower's state word (twin.rs:1240-1248). heartbeat::Registry::record (gunnar-grpc/src/heartbeat.rs:158-178) keeps the last one per principal plus a ring of behind_polls + 1 samples; view (:181-231) decides ok / behind / silent at answer time, never on a timer — a twin that died cannot report its own death, so absence has to read as silent (heartbeat.rs:10-13). silent after 60 s; behind only on a stuck head or a monotonically growing lag, so a healthy constant small lag reads ok (:105-132). A group member that has never polled gets a silent row (:195-201). This is well built.
• Twin's view. primary_view (heartbeat.rs:243-264) — silent when the primary has not answered inside the bound. Surfaced on Bootstrap.Status (bootstrap.rs:313).

Failure mode after this morning's fix. Before 4d38319, start_follower called run() ONCE: any failure before the poll loop (no twin.json yet, an unreadable twin.key, a channel that would not build) killed the thread for the life of the process and nothing retried it. The commit's own measurement: the live twin at applied 0 of 19264, with an empty state word beside it, blocking every growth in the estate for ever because flushing refuses to unmount a set whose twin is behind. The fix retries every FOLLOWER_RETRY (5 s, no back-off ladder, on purpose) and writes the failure INTO the status with the attempt count, so the heartbeat carries unreachable instead of "". SnapshotNeeded stays a stop — it needs an operator's twin catchup and re-running would write the same refusal every five seconds. This is the right shape. It is on agent/stink-leaks and NOT on master (verified: git merge-base --is-ancestor 4d38319 master fails).

Residual: the registry is in-memory. Registry.rows is a Mutex<BTreeMap> (heartbeat.rs:139) with no persistence. A primary restart resets every row to "never polled since this server started". Correct-by-default (reads silent), but it means the primary can never tell you how long a twin has been dead ACROSS a restart. Rank: D9.

Can a stale twin be used for a restore without anybody noticing? YES. replay reads the twin's LogHead and, if it is not caught up, prints a warning on stderr and proceeds:

crates/gunnar-cli/src/control/twin.rs:99-101
    if head.twin && !head.caught_up {
        eprintln!("warning: the twin says `{}` ({}); restoring what it holds, {}..={}", …);
    }

There is no --force, no confirmation, no refusal. And it is worse than it looks in the case that matters: in the real disaster the primary is dead, so the twin's state is unreachable and caught_up is false ALWAYS — the warning fires on every genuine rescue and therefore carries no signal. Nor can the operator measure the staleness: LogHeadResponse has no last_answer_unix_ms field (mirror.proto:192-211), and primary_head_seq is itself the twin's stale memory from its last successful poll. A twin that stopped following three weeks ago and a twin that was level ten seconds before the primary died are indistinguishable to replay. Rank: D2 — this is the second most important finding in this audit.

5. Promotion

Promote-in-place is not supported, and it is refused by construction in three separate places.

1. twin.rs:2 — the module's first sentence: "applies them in order into stores of its own, and is never promoted." Repeated at twin.rs:220 as the reason a twin may keep a frame kind it cannot apply while a box being RESTORED may not.
2. restore_begin — anyhow::ensure!(!stores.is_twin(), "a twin is never restored into; restore into a fresh primary") (twin.rs:722).
3. become_twin — "a twin is never re-pointed" (twin.rs:647). There is no un-twin verb: nothing deletes twin.json, and is_twin_root (twin.rs:463) is read at every ServerStores::open (stores.rs:736), which sets writes_frozen and browse (stores.rs:852-853). A twin comes up frozen, for ever, by the presence of one file.

Mirror.SetRole(PRIMARY) (service/mirror.rs:350-366, control_plane.rs:5289-5343) belongs to the replication subsystem — standby / browse replica — which is a different and older mechanism. It does not promote a txlog twin, and a browse replica refuses its own promotion by name (control_plane.rs:5302-5313). The two mechanisms share the word "replica" and nothing else; that overlap is a documentation hazard in its own right.

Consequence, stated plainly. The twin's own hardware can never become main. Recovery requires a THIRD box — either main re-imaged, or a new VM. You cannot re-image the twin to make it main, because re-imaging it destroys the only copy you are rescuing from. Under the owner's words — "if the whole main dies, the twin has config + data" — this is correct as a RESTORE SOURCE and incomplete as a recovery plan: the plan silently requires that a fresh box is obtainable at the moment main dies. In an estate where Scaleway/UpCloud capacity has been out-of-stock (memory: scaleway-out-of-stock-2026-09-09), "obtain a fresh box" is not free. Rank: D5 — this is a design decision to affirm or change, not a bug.

The ruling of 2026-09-21 (one medium, role as an install parameter) makes promote-in-place cheap to add later if it is ever wanted — the binary on the twin is already the same binary — but it does not by itself create it. The refusals above are three explicit lines of code, not an accident of packaging.

6. What is untested

In-process coverage is genuinely good: crates/gunnar-server/src/twin/tests.rs holds 25 tests including late join, restart mid-range, a torn tail, the retention hole, restore rebuilding every object AND every document, "the same box not just the same repositories" (:623), the too-small-data-set refusal, the empty and short streams, an apply that failed being re-applied, LFS delete and repo purge not resurrecting, the KEK re-seal and the twin following the new key, snapshot catch-up then follow, unknown kinds in a log and in a snapshot, a broken snapshot stream not leaking into the next install, a crash mid-install, and a fresh box's own secret not bricking the box. Plus gunnar-server/tests/twin_heartbeat.rs, twin_snapshot_memory.rs, gunnar-cli/tests/twin_push_is_framed.rs, and the KVM drills twin_drill.rs / restore_drill.rs (H1, R8, R9, R10, R6).

What nothing exercises:

1. A twin following a DIFFERENT log at the same address (§2, D1). No test, no bench. The failure is silent by construction.
2. A restore from a demonstrably STALE twin (§4, D2). No test asserts what replay does when the twin is hours behind — because nothing refuses it.
3. The whole rescue as one ordered procedure. No verb (§3, D4), therefore no test of the verb. R6 proves the STEPS; nothing proves an operator following a documented order gets them in the right order.
4. The short-stream refusal under KVM. Named as unproven by the bench itself (restore_drill.rs:27-36) — the media predate restore_head (.nornir/twin-transaction-log.md:466-467). Proven only by pump_into unit tests.
5. A restore whose entitlement facts do not verify (§1, D3). Nothing drives EntitlementSlot::open with a restored fact under a different monetize key.
6. Bootstrap.Restore's unauthenticated door. tests/auth.rs asserts it is on the unauthenticated list; nothing drives a hostile restore into a fresh box.
7. Twin-key rotation (§2, D8). key_rotation.rs drives the PRIMARY-side principal churn; nothing rotates twin.key itself, because no verb does.
8. log/ and grow-journal absence after a restore (§1). No test asserts what a restored box has lost; R6 asserts only repos, users and catalog.
9. A twin re-image against a trimmed primary, i.e. the production freeze a full SnapshotFetch costs. R8 makes a hole deliberately with LogRetain; nothing measures the freeze duration on a real-sized set as an operational number.
10. Two twins. require_log_reader admits any member of group twin; the heartbeat registry is keyed per principal and would show both. Nothing exercises it, and nothing refuses it.

7. Against the one-medium ruling: where does the code assume the twin is a different KIND of box?

Answer: almost nowhere. The ruling is cheap to satisfy, and that is a genuinely good finding.

The role is already a runtime fact, not a build fact:

• GUNNAR_ROLE does not exist anywhere in this repo — zero hits across *.rs, *.toml, *.env. Nothing is keyed off it today, so introducing it costs one env read.
• The real source of truth is <data-root>/twin.json, written at runtime by Bootstrap.BecomeTwin, read by is_twin_root (twin.rs:463) at every ServerStores::open (stores.rs:736), which then sets twin, writes_frozen and browse (stores.rs:852-853). That file is in is_per_box_path (txlog.rs:1445), so it is never framed and never travels — correct.
• The secondary source is the boot argument --role twin (gunnar-common/src/cli.rs:286-291, gunnar-server/src/lib.rs:337). It does exactly two things: it binds the git listener to loopback:0 (serve.rs:994-999) and it makes Bootstrap.Status report role = "twin" (control_plane.rs:1707). It does NOT freeze writes, does NOT start the follower, does NOT set stores.is_twin(). So a box booted --role twin that has not yet joined is an ordinary fresh primary with no git listener — which is exactly what prepare_replay then requires.
• No runtime check reads a variant or medium name. Grep for GUNNAR_VARIANT, variant_name, build_variant finds only gunnar-recipe plumbing and a container-robot test. The server never asks what it was built from.
• Identity, keys, data layout and config paths are all derived from the data root and from boot params, never from the image. twin.key is minted at join, on the box, from gunnar_core::mint_ed25519() (twin.rs:653).

What actually encodes "a twin is a different image", and all of it is declaration, not code:

The two cache switches (ZNIPPY_GIT_EXPLODE=off, ZNIPPY_GIT_GRAPH_CACHE=off) are already redundant: a twin turns both off itself at runtime — mark_twin calls engine.hold_no_graph_memo() (stores.rs:1119-1123), objects.exploded* and objects.graph are is_cache_path so no frame can create them (txlog.rs:1428-1430), and forget_mirrors removes the mirror dir. So the one medium loses nothing by not carrying them.

Does anything in the join/pairing path break when both boxes are demonstrably the same build? No. become_twin requires only that the target is fresh; the follower needs only twin.json

• twin.key; require_log_reader checks only group membership. Nothing compares builds, versions or images in either direction. Two boxes off the same medium pair exactly as two boxes off two media do.

The one thing the ruling DOES require of the code, and it is small: --role twin must reach the server. Today it reaches it as korp.args=serve,…,--role,twin on a twin-only installer row. Under one medium it must come from an install parameter — --param role=twin → korp.env.GUNNAR_ROLE=twin — which means adding an env read that maps GUNNAR_ROLE=twin to ServeRole::Twin, defaulting to Primary when absent. That is one env lookup beside args.role at gunnar-server/src/lib.rs:337. Nothing else in the server changes.

Headline: nothing in this design fights the one-ISO ruling. The code was already role-at-runtime; only the build declarations and the bench's two-medium assumption are not.

VERDICT

The twin design is SOUND in its core and SALVAGEABLE at its edges. It is not wrong.

The core — one ordered transaction log, the gap rule, durable-before-apply, apply through the unlogged halves, the same apply serving follow and restore, the KEK as seq 1, the re-seal on restore — is a good design, carefully reasoned, and visibly built out of measured failures rather than out of a diagram. Several of the nastiest cases in this class of system (torn tail, half-installed snapshot, short stream read as complete, a fresh box's own secret bricking the restore, a frame kind a newer primary invented) are already found, fixed and regression-tested. The restore path the owner asked about exists, has a name, and has been driven under KVM to its last step.

What is wrong is not the mechanism. It is that the twin is trusted without being checked: nothing verifies it is following the log it thinks it is, nothing verifies it was current when you restored from it, and nothing walks the operator through the rescue in the right order. All three are additive fixes. None of them requires restructuring.

The right shape, concretely — four additions, no rewrites:

1. Give the log an identity. A per-box UUID minted at first open, framed as part of seq 1 beside the KEK (or carried as a new LogHeadResponse field). The twin records it at join in twin.json and refuses to follow a LogHead that reports a different one, by name. This closes D1 and costs one field on the wire plus one comparison in Follower::step.
2. Put a clock on LogHead. Add last_answer_unix_ms (the twin's own TwinStatus::last_answer_unix_ms, which already exists at twin.rs:65 and is simply not on the wire) to LogHeadResponse. Then make replay refuse a twin whose last successful poll is older than a bound, with --force-stale as the named override and the age printed in the refusal. This closes D2.
3. Write the rescue as one verb. cargo xtask restore main — the name restore_drill.rs:38 already uses. Order: prove the twin holds the watermark → re-image the dead main → prove it reads FRESH → twin replay → wait for the control plane (the 180 s the bench learned it needs) → verify a clone. The bench's R6 already IS this procedure in Rust; lifting it out of the bench into a shipped verb is refactoring, not new design. This closes D4 and makes D2's check unskippable.
4. Make the entitlement mismatch loud. EntitlementSlot::open should count refused facts and, when the count is non-zero on a box that has just been restored (reseal_pending/post-restore), refuse to start rather than run every tenant Free. This closes D3.

Leave promote-in-place refused (D5), but write the recovery precondition down where an operator meets it: the rescue needs a third box, and the twin is never it.

Defects, ranked by what it costs to leave them

On the one-medium ruling (§7): it is cheap. Nothing in the server code assumes the twin is a different kind of box. GUNNAR_ROLE does not exist yet, the role is already twin.json + --role twin, no runtime check reads a variant, and the two twin-only cache switches are already redundant at runtime. The work is: delete two declaration rows, map GUNNAR_ROLE=twin → ServeRole::Twin beside lib.rs:337, fix three refusal strings, and collapse the bench's --twin-iso into one --iso plus a role param. No part of this design fights the ruling.