nordisk/gunnar

The audit trail survives a restore — lane T11, 2026-09-21

Closes D6 from .nornir/TWIN-DESIGN-AUDIT-2026-09-21.md (lane T1): "log/ — the audit trail is silently gone." Rules on D7 (grow-journal) without changing it.

Branch agent/audit-survives-restore, based on master (3105cc0). It touches no file lane T4 is changing on agent/twin-liveness except one additive line in twin/tests.rs (see Coordination at the end).

1. What log/ and crash/ actually are — measured, not quoted

Read off the source, because the audit doc's summary is right in substance and loose in two places worth naming.

log/

gunnar_server::slow::LOG_DIR (slow.rs:121), resolved from dir.log.path in the installer's layout config — /data/log in the installer's own fixture, a symlink onto the other spindle; <data root>/log when there is no config at all (SlowRoot::resolve).

It holds exactly one file: audit-chain.jsonl (gunnar_core::AUDIT_FILE), written by ServerStores::audit_path() → gunnar_core::AuditChain → the gate's ChainLedger (grpc_gate.rs:107). Opened once on the startup path (serve.rs:1497).

The module's own doc calls it "the audit trail and the durable console log" (slow.rs:120, slow.rs:84). The console half does not exist. Nothing in the tree writes a console file into log/. The console goes to tracing, i.e. the journal. That sentence is stale and this lane did not make it true; it is recorded here rather than quietly corrected, because the estate has been bitten by stale claims twice today.

Format and guarantees. JSON Lines, one AuditEntry per line, deny_unknown_fields. It is a SHA-256 hash chain, not signed: each entry's hash covers a domain prefix, the previous entry's hash and every field, each length-prefixed; the first entry's prev is 64 zeros (audit.rs:1-78). Every append is write_all + sync_all before it returns. open reads and verifies the whole file before the first append, so a server refuses to extend a chain that has already been broken.

Relationship to "the signed append-logs" the txlog already carries — they are NOT the same thing, and the audit doc's phrasing invites the confusion. FRAMED_FILES held nine files: attestations.jsonl, transparency-heads.jsonl, builds.jsonl, build-heads.jsonl, forge.jsonl, forge-heads.jsonl, membership.jsonl, membership-heads.jsonl, plus replication.json/webhooks.json as whole documents. Those are the push/attestation/forge logs, they go through txlog::sink::append_line, and they are replicated. The control-plane trail — who was allowed and who was refused, per RPC — went through std::fs::OpenOptions directly, framed nothing, and was excluded a second time by is_per_box_path treating all of log/ as per box. It was the one append-only log in the tree that was not a transaction.

Size. One line per authorised or refused gRPC control-plane RPC, ~350–500 bytes. (Changed 2026-10-06, P3 of the load test: repeated plain READS — allowed or narrowed at level read — are aggregated: the first of a 60 s window per (method, principal, scope, decision, …) is an entry at once, and one more entry counts the repeats with the first and last time (grpc_gate::READ_WINDOW, reason = aggregated: <n> more calls, first <ms>, last <ms>, window 60 s). Every refusal, write, admin call, self-scope call and override is still one entry each. tests/audit_read_io.rs; see twin-transaction-log.md §"Group commit".) Git push/fetch go over SSH and are not audited. Mirror.LogHead — the twin's poll — writes NO entry: MirrorService::require_log_reader short-circuits on group twin membership and emits only a tracing::info!, never reaching the ledger (mirror.rs:36-54). That is the fact that makes this whole change safe: framing audit lines cannot feed back into the twin's polling and grow the log without bound. require's normal path (a non-twin reader) does audit.

crash/

slow::CRASH_DIR (slow.rs:119). One file per panic, panic-<unix-ms>.txt, written by record_crash from the panic hook (slow.rs:563-583). Best effort by construction — it runs inside a panic, a failure there has nowhere to be reported, and an unwrap would replace a readable panic with an unreadable double one. Registered from ServerStores::open (stores.rs:691). Unstructured text. Not chained, not signed, not indexed.

2. The shape chosen, and why

Chosen: replicate log/audit-chain.jsonl through the txlog like everything else. Rejected: a separate append-only stream a restore reassembles. Rejected: replicating crash/.

Why the txlog, not a second stream

1. The mechanism already exists and is proved. Nine append-only logs already ride Kind::Append frames; the twin applies them with append_line_once (idempotent under re-apply); snapshots carry each FRAMED_FILES entry whole and remove the ones a snapshot does not carry. A second stream would be a second ordering, a second retention policy, a second snapshot rule and a second way to be behind — for a file that is a line-appended JSONL log, i.e. exactly what Kind::Append is for.
2. A separate stream cannot be atomic with the thing it records. sink::append_line takes the log's mutation gate, so a snapshot cannot cut between the frame and the byte. A parallel stream would let the trail and the state it describes diverge across a snapshot — the worst possible property for an audit record.
3. "Best effort" is not available. The frame is taken before the line is written and the append fails if the frame fails. A line in the file with no frame behind it would restore as a chain with a hole, and verify would report the break as the primary's, months later, to somebody who cannot know it was ours. The gate's existing contract — one error!, never a failed RPC — is kept by the caller, which is where it belongs.
4. It gives a property a separate stream could not. Because the chain replicates, a restored box opens the file, finds the dead primary's tip, and appends its own first decision with prev = that tip. The trail is one unbroken, verifiable chain across the disaster. You can hand an auditor a single file.

The twin's own audit events — the decision

They are NOT in the same trail, and this is the load-bearing decision of the lane.

A twin serves a control plane of its own: Mirror.*, browse, its own refusals. Those decisions are audited by the same gate. If they went into log/audit-chain.jsonl, the twin would be a second writer into a hash chain whose entire value is that it has exactly one — the prev links would not hold from the splice onward, the restored copy would be refused at open, and the twin would in effect be forging entries into the primary's trail. That would break twin.rs:210's property (a twin originates nothing) at the one place where originating is indistinguishable from forgery.

So: ServerStores::audit_path() answers audit-chain.local.jsonl (txlog::AUDIT_LOCAL_REL) on a twin and on a box mid-restore. That file stays is_per_box_path, is never framed, and is never sent anywhere. The branch is taken on the same two facts that decide whether the log originates at all (ServerStores::open: a twin or a restoring box is a follower and registers no sink), so the trail and the log can never disagree about which box this is.

The twin's own record therefore lives and dies with the twin. That is correct: its decisions are facts about the twin, and a restore is not supposed to import them.

crash/ stays per box — deliberately, and this is a change of answer from the brief

The brief groups log/ and crash/. They are not the same problem.

A crash record is the panic text of one process on one box: a backtrace into that binary, at that build, on that hardware. It is not a record of a decision anyone made and it is not part of any chain. Replicating it would mean the twin's disk accumulates the primary's panics, and a restored box comes up holding crash files that describe a machine that no longer exists, in a crash/ an operator will read as this box's. That is a new way to be wrong, traded for nothing: the reason to want the crashes is diagnosis, and diagnosis is done from the console/ journal and the dead box's disk if it survives.

crash/ is also written from a panic hook with no error path, so it is the one writer in the tree that structurally cannot take the mutation gate or report a framing failure.

Verdict: crash/ is correctly per-box. D6 is about the audit trail, and only the audit trail.

3. grow-journal (D7) — verdict: stays unreplicated

It is is_cache_path today. It should stay so.

1. It describes hardware the restore does not land on. The journal is the state machine of an in-flight disk growth: device names, member counts, sizes, which step was reached. A restore deliberately lands on whatever machine is to hand — twin_drill T2 restores 4 devices onto 8, T3 onto fewer and smaller. Replaying another box's half-finished growth journal against different hardware is not a recovered fact, it is a false premise the box would act on.
2. The outcome is already carried by the thing that matters. The data set is laid for the machine the restore lands on by the installer, and the drill already has a row for it (ids.geometry).
3. A restored box answering "not growing" is TRUE of that box, which is the difference from D6: D6 destroyed a record and reported success; D7 loses a per-box work item that is no longer in flight.
4. And the D6 fix materially closes what was actually worrying about D7. Admin.GrowStart and Admin.GrowResume are audited RPCs. With the trail replicated, a restored box can now answer "a growth was started at T by principal P" — from the audit trail, which is where a question about what somebody did belongs, rather than from a journal describing dead disks.

4. The new drill class — for lane T3 to run

crates/gunnar-growbench/src/twin_drill.rs. One new verdict per scenario: T1v / T2v — "the control-plane audit trail back: the primary's own decisions, chained, at the same hashes." (T3's own scenario keeps its id space; add T3v the same way if wanted.)

Why it is not a control_state class

Every other class compares one string before and after. The audit trail grows while you look at it — reading it is itself an audited decision — so a tip-to-tip comparison could never pass. It is compared at a named sequence instead: because each entry's hash commits to its prev, one matching hash at the primary's last sequence proves the whole prefix came back byte for byte.

The scenario, precisely

Inside the existing T1/T2 drill, no new VM and no new boot:

1. Seeding is already done by the drill's own work. Every control user add, group add, group member add, namespace create, grant, repo set-visibility, repo set-default-branch, repo delete --purge, webhook add, … is an authorised RPC and therefore an audit entry. Nothing extra is seeded, which is the point: the class is about the record of what the drill did.
2. Before the disks die, last of all the "before" reads (after control_state and host_key), run on the primary: gunnar control audit -i <admin> --at 127.0.0.1:<grpc> and keep entries, tip, tip seq, verifies, replicated.
3. Destroy every primary disk, re-install the medium, gunnar control twin replay — unchanged, nothing in this lane touches the replay path.
4. After the restore, alongside the other "after" reads, run on the restored box: gunnar control audit -i <admin> --at 127.0.0.1:<grpc> --at-seq <the primary's tip seq>
5. PASS iff: the primary's trail was present, non-empty, verifies: yes and replicated: yes; the restored box's trail is present and verifies: yes; its hash at seq <N> is byte-identical to the primary's tip; and it holds at least as many entries as the primary did.
6. FAIL, each with its own sentence: the primary's trail not replicated ("this is D6"); the primary's chain already broken before the drill (nothing after it is a claim about the restore); the restored box has no trail at all; the restored chain does not verify; the restored box holds no entry at seq N (the primary's history did not come back); the hash at N differs (both hashes printed).
7. UNRUN, never a vacuous PASS, in the drill's own idiom: the medium's build answers no control audit (an ISO composed before this verb — unimplemented() already recognises the wording); the primary was serving without a durable trail; or the primary's trail was empty, which would mean the drill's own decisions never reached it.

The parser is audit_shape(), which reads the printed lines of the real CLI rather than opening a second gRPC client — so a verb that prints something other than what it means is a failure the drill sees.

What T3 needs

• A medium composed from this branch (the verb must exist in the ISO).
• No new flags, no new VM, no change to --twin-iso handling.
• Run T1 and T2. Report T1v / T2v verbatim, including the detail line.

Not run by lane T11: this lane cannot start KVM (T3 owns it on this box). The row is written and compiles; the KVM measurement is T3's to report, and until it does, the honest statement is that the drill class exists and has not been run.

5. What is proved natively, right now

• gunnar-core audit::tests::a_framed_chain_puts_every_entry_in_the_transaction_log — every entry is an Append frame naming log/audit-chain.jsonl; the file is rebuilt from the frames alone, opens, verifies, has the same tip, and the rebuilt chain's next entry has prev = the dead primary's tip.
• gunnar-core audit::tests::an_unreplicable_chain_refuses_to_be_the_replicated_trail
• gunnar-core audit::tests::a_twins_own_trail_is_never_a_transaction
• gunnar-core txlog::tests::the_audit_trail_is_framed_and_the_twins_own_trail_is_not
• gunnar-server slow_disk_placement::the_audit_trail_is_the_replicated_trail
• gunnar-server slow_disk_placement::a_twin_writes_its_own_audit_trail_and_not_the_primarys
• And, without a line of new test code: twin::tests::a_restore_from_the_twin_rebuilds_every_object_and_every_document and a_restored_box_comes_back_as_the_same_box_not_just_the_same_repositories now cover the audit trail too, because they iterate FRAMED_FILES.

Each new test's doc comment names the exact edit that reddens it.

6. Two things this lane did NOT fix, recorded rather than left

• slow.rs:120/:84 claim log/ holds "the durable console log". It does not. Nothing writes a console file there. The audit trail is the only occupant.
• become_twin does not call txlog.set_follower(true) on the LIVE log (twin.rs:644-664). A box conferred the twin role over the wire keeps a log that accepts originations until the next open. It is only reachable on a FRESH box (prepare_replay requires it) and the flag is set correctly at the next open, so nothing is known to break — but it is the same class of "role arrived over the wire, the door is a restart behind" that lane T4 is closing for the git door, and it belongs to twin.rs, which is T4's file this week.

Coordination with lane T4

T4 (agent/twin-liveness, on agent/stink-leaks) changes twin.rs, control/twin.rs, mirror.proto, bootstrap.proto, gunnar-ssh, xtask, and stores.rs (the twin flag → Arc<AtomicBool>, around stores.rs:522/846/1115/1496).

This branch changes stores.rs only at audit_path() (~line 898) — no textual overlap, and it is helped by T4's change: is_twin() becoming live means audit_path() answers correctly for a box conferred the twin role at run time.

twin.rs is not touched at all. The one shared file is twin/tests.rs, where this branch adds four lines to the fixture loop that writes every FRAMED_FILES entry (a framed file may now live in a subdirectory, and the appender does not create one). Additive; trivial to merge either order.

Merge chain: stink-leaks → twin-liveness → this, or this straight onto master — either works.

Addendum, same day: "config must be saved in twin too"

The owner widened this lane after the above was written. What follows answers the four gaps, the store-only question, and corrects two things stated to this lane that the source does not support.

7. Does any of this survive a STORE-ONLY twin? — yes, all of it

The owner is weighing a twin that receives and saves frames and never applies or indexes, with indexing deferred to recharge time on the rebuilt main. Stated plainly, for each piece:

• The audit trail: survives, unchanged. It rides as Kind::Append frames. A store-only twin stores those frames exactly as it stores a Pack; the restore streams the twin's log into the fresh box, which applies every frame including these, and the chain is rebuilt from the frames alone. That last sentence is not a hope — it is literally what audit::tests::a_framed_chain_puts_every_entry_in_the_transaction_log does: it discards the file, rebuilds it from the frames, and verifies it.
• The hub declaration: survives, unchanged, for the same reason — one Kind::File frame.
• What a store-only twin loses is the ability to ANSWER about the primary's trail, because it would hold frames and not a file. That is correct and was already the design here: a twin answers Admin.AuditTrail about its OWN local trail (log/audit-chain.local.jsonl), never about the primary's, because the primary's trail on a twin is a replica the twin must not speak for. Nothing in this lane ever asks a twin what the primary's config or trail says.
• The new drill class is store-only-safe: T1v/T2v read the PRIMARY before its disks die and the RESTORED box afterwards. The twin is never read.

So: nothing in this lane assumes the twin applies. The one place that would have — asking a twin to report the primary's trail — is refused by construction.

One consequence worth stating for whoever lands store-only: snapshot::install and snapshot::stream walk FRAMED_FILES as files on disk, which a store-only twin would not have. Snapshotting, not following, is where store-only bites. That is not this lane's call, but FRAMED_FILES is now the list to look at when it is made.

8. wg/hub.conf — and a correction

Correction to what this lane was told. The hub declaration is not "never into the data root". korp-installer stages it to <data root>/wg/hub.conf — on a FAT /data it writes it directly, on an xfs /data it rides the ESP under korp/data-stage/ and datahome::apply_data_stage copies it in on the first boot (esp.rs:420-438, install.rs:630-661). gunnar reads it from exactly there (gunnar_server::appliance: "<root>/wg/hub.conf, <root>/wg/identity.key, <root>/wg/identity.pub — the SAME names on P3, in a container volume, and under holger and korp").

So it was inside the data root and still not replicated — for a different and more interesting reason than being outside it: nothing in gunnar ever writes it, so it had no sink call site, and it was in neither FRAMED_FILES nor any write path. It was invisible to the replication machinery rather than excluded by it. That is the same failure mode as the audit trail and it is fixed the same way.

What must the twin hold so that a rebuilt primary comes up on the tunnel? The hub declaration, and nothing else from wg/:

Two reasons for the identity ruling, either sufficient: a twin holding it is a second place to steal the site's network identity from; and a restored main carrying it would come up claiming the dead machine's identity on a tunnel the hub registered to a key that no longer exists. The same ruling holger makes about its admin token and its role record, which the coordinator named — and gunnar's equivalents are exactly these two files plus twin.json/twin.key, which is_per_box_path has always refused.

Before this change the two identity files were merely un-framed by accident — nothing wrote them through the sink — which is not the same as being refused. A future writer, or a snapshot walk that took a directory rather than a list, would have carried a private key to the twin with nothing to stop it. They are refused now.

Not solved here, and not this lane's: [Interface] Address is per box, so the hub declaration the twin now holds is correct in four of its five values and names the DEAD box's address in the fifth. That is lane T9's reported blocker. What this lane changes is that the rebuilt main now has the file — four values right and one to be re-stated — instead of having nothing at all. Whoever closes T9's half should decide whether the restore rewrites Address from the medium or leaves it to the ceremony.

No drill class for it yet, deliberately. There is no control-plane read of the hub declaration and the drill's boxes are not on a tunnel, so a row would either be unseeded or vacuous — which the drill's own idiom forbids. It is proved natively by slow_disk_placement::the_hub_declaration_is_framed_once_per_change and by twin::tests::a_restore_from_the_twin_rebuilds_every_object_and_every_document, which now covers it because it walks FRAMED_FILES.

9. The kernel cmdline and korp.env.* — decision: the medium is sufficient

They should NOT be replicated, and this is a decision, not an oversight.

korp.env.GUNNAR_NET, GUNNAR_DATA_ROOT, GUNNAR_APPLIANCE, static_ip, gateway and the rest are install parameters baked into the medium by gunnar-recipe (reachability.rs, recipe.rs) and handed to init by the kernel's unknown_bootoption path. They describe how this artifact was composed and how this machine is wired — which NICs exist, where the data root is mounted, whether this binary is an appliance at all.

Three reasons they must come from the ISO and not from the twin:

1. They are properties of the medium, and a restore re-installs the medium. The rebuilt box boots its own ISO, which carries its own cmdline. Taking the dead box's would mean a box composed with GUNNAR_NET=eth0:dhcp,eth1:dhcp inheriting a declaration for interfaces it does not have.
2. A restore deliberately lands on different hardware (twin_drill T2: 4 devices → 8; T3: fewer and smaller). The cmdline is the one place the new hardware gets to speak.
3. They are read before there is a data root to read anything from. GUNNAR_DATA_ROOT is how gunnar finds the log the twin would have delivered them in.

The failure mode this leaves is real and is not silent: a rebuilt box composed from a different medium than the dead one comes up differently wired, and the operator finds out at the first boot console rather than three weeks later. That is the right place to find out.

10. grow-journal — unchanged verdict

See §3. Stays unreplicated, and the D6 fix materially closes what was worrying about it: with the trail replicated, Admin.GrowStart/GrowResume are answerable on the restored box from the audit trail.

11. One thing NOT fixed, and one flake, both reported rather than hidden

• prepare_replay now also drops this box's own audit trail (forget_own_audit_trail), for the same reason it drops its own secrets and its own log id: the incoming frames are the dead primary's chain from ITS seq 0, and a fresh box that had written one line of its own would take them onto the end of it and the result would not verify. Proved by twin::tests::a_fresh_boxs_own_audit_trail_goes_with_its_own_numbers. This was found by reading T4's new restore path after the merge — it did not exist as a bug before this lane, because the trail was not replicated at all.
• A flake, named because silence would be worse. twin::tests::a_restart_mid_range_resumes_at_the_twins_own_head failed once in four parallel runs on this branch (seq 13: refs into ns/app … would point at <oid>, which this repository does not have) and passed 3/3 with --test-threads=1 and on every re-run. Master (a01a8e7) was green 10/10. That comparison is confounded: this box was carrying several other lanes' builds, master's runs took 8–20 s and this branch's took 22–122 s for the same suite, so "master is stable" may be measuring a quieter machine rather than better code. It is not diagnosed. It is not a failure of any assertion this lane added.

12. Correction, same evening: T1v/T2v on KVM, and a diagnosis of mine that was wrong

T3 ran the audit cell on KVM (medium bench-metered-integ-ff2c7c9). Both rows FAILED: "DIFFERS at seq 40 … the restored box holds: (not held)".

My first diagnosis (624a62e) was wrong, and I am saying so here rather than leaving it in the commit history unanswered. I concluded that the restored box's copy of the trail, on the boot disk's P3 (/data/log → /serialdat/log), was lost in the hard stop between the replay and the restart, and I wrote a rebuild-from-the-log for it. The evidence I had supports a different cause, and T3 found it:

The restored box never left restore mode. restore_finish removed txlog-restoring with a bare remove_file and no directory fsync; the drill's restart is a QMP quit ~10 s later; the unlink was lost; the box reopened mid-restore — a follower, no re-seal, and its gate writing the per-box audit-chain.local.jsonl. Admin.AuditTrail on that box reads its own trail too, which held its own 16 decisions: hence "no entry at seq 40".

The evidence, all from T3's serial logs, and how I misread it:

Retracted, for T3/T4: I reported that "T1e passed on a stale answer — the twin never polled the restored box". That is false. The twin did poll it; T1e's "head 262" was a true answer from a box stuck in restore mode.

Is the rebuild still worth keeping?

It stays, described accurately: defence on a path where no loss has been demonstrated. The replay writes every line durably, so after T3's marker fix a restored box's trail should equal its log. The rebuild costs one header-and-meta scan per restore and makes "the file disagrees with the log at the one moment the whole trail was just rewritten" self-correcting and loud. Its test (a_restored_box_whose_off_stripe_trail_was_lost_rebuilds_it_from_its_own_log) is a correct test of that behaviour; its doc comment used to claim it reproduced the KVM failure, and that claim is removed.

The real window it does NOT cover — for the owner. On a running PRIMARY, AuditChain::append frames the line (the txlog append is sync_data'd) before writing it to the file. A crash between the two leaves the log one line ahead of the file. On the next boot the gate continues from the file's shorter tip and frames a second entry at the same seq; the twin, which applied both, then holds a chain that does not verify, and so does anything restored from it. Running the same rebuild on every primary open closes it, at the price of that meta scan on every boot. That is a scope decision and it is not taken here.

What the drill now says

The cell had two defects of its own, both fixed with tests that fail when the fix is removed:

1. It read the CLI's (not held) as a hash, so the row said "DIFFERS" instead of "holds no entry at seq 40 (it holds 16)".
2. It never checked replicated on the RESTORED box. That one field was no on T3's run and would have named the real cause on the first attempt. It now fails with: "the restored box answered from its OWN per-box trail … it is still mid-restore or believes it is a twin".

The proof is T3's next KVM run, on a medium carrying both T3's remove_durable and this branch.