crates/gunnar-core/src/audit.rs
//! **The control plane's audit trail, made tamper-evident** — an append-only
//! hash chain over the decisions the gRPC gate makes.
//!
//! # What was there, and why it was not evidence
//!
//! Every authorised RPC already emitted one `tracing` record on
//! `gunnar::audit` — method, principal, key fingerprint, scope, level, decision.
//! That is a good *log* and it is not a *trail*: it goes wherever the
//! subscriber points it, usually a file or a journal that the same box's root
//! can rewrite, and nothing in it says whether it has been rewritten. An
//! operator reading it cannot tell a complete record from one a line was cut
//! out of, which is exactly the question an audit trail exists to answer. The
//! security review recorded it as GUN-044.
//!
//! # The shape: each entry names the one before it
//!
//! ```text
//! entry.hash = SHA-256( "gunnar-audit-v1" ‖ prev ‖ len‖field … )
//! ```
//!
//! The first entry's `prev` is [`GENESIS`] (64 zeros). Every later entry's
//! `prev` is the previous entry's `hash`. So the tip hash commits to the whole
//! history: change one byte of one entry and that entry's hash changes, which
//! breaks the `prev` of the one after it, and so on to the end. Removing an
//! entry breaks the same link; so does reordering two. There is no way to edit
//! the middle of the file without rewriting every line after it, and a
//! recorded tip (an operator's note, a second box, a monitoring system that
//! reads the tip once a day) makes even that detectable.
//!
//! **This is not the transparency log.** [`crate::transparency`] is a Merkle
//! tree with signed heads and inclusion proofs, which is the right shape for a
//! log a *third party* audits without trusting the server. This is a linear
//! chain with no signature, which is the right shape for a trail an *operator*
//! checks on their own box, and it costs one hash per RPC and no key
//! management. Making the control-plane trail publicly verifiable is a
//! different, larger decision; this one closes the finding that was actually
//! reported — a trail that cannot say whether it is complete.
//!
//! # The fields are length-prefixed, and that is the load-bearing detail
//!
//! A chain whose fields were joined with a separator would let a value
//! containing that separator move the boundary: a `reason` of `"x\nallow"`
//! could hash identically to a different entry whose decision really was
//! `allow`. So each field goes in as its byte length (8 bytes, little-endian)
//! followed by its bytes, which is unambiguous whatever the field holds. The
//! same reasoning as [`crate::transparency`]'s domain-separating prefixes, one
//! layer down.
//!
//! # One file, appended, fsynced
//!
//! JSON Lines, one entry per line, beside the rest of the control state — the
//! same choice push attestation already made, for the same reason: a record
//! that outlives the process must be readable by something that is not gunnar.
//! Each append is durable before it returns. On the primary's REPLICATED trail
//! ([`AuditChain::open_framed`]) the durable copy is the `Append` frame in the
//! transaction log — group-committed with every frame staged beside it
//! (perf, 2026-10-06: concurrent decisions share one `fdatasync`, measured
//! 2.0 → 0.26 flushes per decision on eight request threads) — and the line in
//! this file follows it without an fsync of its own: the file is the view of the
//! log's frames that every start checks and rebuilds
//! (`gunnar_server::twin::rematerialise_audit_trail`). An unframed chain (a
//! twin's own trail) `sync_all`s each line as before.
//!
//! How decisions reach the chain is the gate's ledger's call
//! (`gunnar_server::grpc_gate`): every refusal, write, admin call, self-scope
//! call and override is one entry at once; repeated plain READS are aggregated —
//! the first of a 60 s window at once, then one entry with the count and the
//! first and last time (P3 of the load test, 2026-10-06, `READ_WINDOW`).
use std::io::Write as _;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use sha2::Digest as _;
/// The file the chain lives in, under the control directory.
pub const AUDIT_FILE: &str = "audit-chain.jsonl";
/// Domain separation, so a hash from this chain is not a hash from anything
/// else in this repository that also runs SHA-256 over records.
const DOMAIN: &str = "gunnar-audit-v1";
/// What the first entry's `prev` is: 64 zeros. A literal rather than an empty
/// string, so every entry in the file has the same shape and a truncated-to-
/// nothing `prev` cannot be mistaken for the start of the chain.
pub const GENESIS: &str = "0000000000000000000000000000000000000000000000000000000000000000";
/// What went wrong with the chain.
#[derive(Debug, thiserror::Error)]
pub enum AuditError {
/// The file could not be read, created or written.
#[error("the audit chain at {path}: {source}")]
Io {
/// Which file.
path: String,
/// The underlying failure.
source: std::io::Error,
},
/// A line in the file is not an entry.
#[error("the audit chain entry on line {line} does not parse: {source}")]
Malformed {
/// 1-based line number.
line: usize,
/// What serde said.
source: serde_json::Error,
},
/// The line could not be framed into the transaction log, so it was not
/// written down at all. **Deliberately not best-effort**: a trail whose
/// file and whose log disagree is worse than one missing entry, because
/// the missing entry is announced and the disagreement is not.
#[error("the audit chain at {path} could not be framed as a transaction: {detail}")]
Frame {
/// Which file.
path: String,
/// What the log said.
detail: String,
},
/// This chain was asked to be the REPLICATED trail and cannot be
/// ([`AuditChain::open_framed`]). Never silent: the caller says so out
/// loud and opens an unframed chain instead.
#[error("the audit chain at {path} cannot be the replicated trail: {why}")]
Unframed {
/// Which file.
path: String,
/// Why not, in a sentence an operator can act on.
why: String,
},
/// The box FOLLOWS (a twin, or a box being restored): it originates nothing into the
/// replicated trail it holds for its primary, so nothing was written ([`Recorded::NotOriginated`]
/// through [`AuditChain::append`]). Not a fault: the caller's log line is the record.
#[error("the audit chain at {path} is the primary's trail and this box follows it (a twin, or restoring): it originates nothing into it")]
NotOriginated {
/// Which file.
path: String,
},
/// The chain does not hold. **This is the finding, not a fault.**
#[error("the audit chain is broken at entry {seq} (line {line}): {what}")]
Broken {
/// The `seq` the entry claims.
seq: u64,
/// 1-based line number.
line: usize,
/// Which invariant failed, in a sentence an operator can act on.
what: String,
},
}
/// One decision, as the control plane hands it over.
///
/// Borrowed, because every field of it is already a `&str` at the call site and
/// an audit record must not make the decision path allocate.
#[derive(Debug, Clone, Copy)]
pub struct AuditEvent<'a> {
/// `Service.Method`, e.g. `Access.Grant`.
pub method: &'a str,
/// The ACL principal id the credential resolved to.
pub principal: &'a str,
/// `SHA256:…` of the key that was presented, empty under the development
/// escape hatch.
pub fingerprint: &'a str,
/// What was asked about: `server`, `namespace:x`, `repo:x/y`.
pub scope: &'a str,
/// The grant level the method needs.
pub level: &'a str,
/// `required` or `allow-all-insecure`.
pub auth_mode: &'a str,
/// `allow`, `deny` or `narrow` — the same three words the `tracing` record
/// carries, so the two can be read side by side.
pub decision: &'a str,
/// Why, for a decision that has a why. Empty otherwise.
pub reason: &'a str,
}
/// One entry, as it sits on disk.
///
/// Field order is the serialisation order and the hashing order. It is
/// `deny_unknown_fields` so a file written by a newer gunnar is refused loudly
/// rather than verified against fields this build cannot see.
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
#[serde(deny_unknown_fields)]
pub struct AuditEntry {
/// 0 for the first entry, then +1 each time. Redundant with the file's line
/// numbering on purpose: a chain whose only ordering was its line order
/// could be truncated to a shorter, still-valid chain without the entries
/// themselves saying anything was missing.
pub seq: u64,
/// When the decision was made.
pub at_unix_ms: i64,
/// `Service.Method`.
pub method: String,
/// The ACL principal id.
pub principal: String,
/// The key fingerprint.
pub fingerprint: String,
/// The scope asked about.
pub scope: String,
/// The grant level.
pub level: String,
/// The authentication mode the listener was in.
pub auth_mode: String,
/// `allow`, `deny`, `narrow`.
pub decision: String,
/// Why, or empty.
pub reason: String,
/// The previous entry's [`AuditEntry::hash`], or [`GENESIS`].
pub prev: String,
/// This entry's own hash. Not hashed into itself; see [`AuditEntry::compute_hash`].
pub hash: String,
}
impl AuditEntry {
/// The hash this entry **should** carry, computed from everything else in
/// it.
///
/// Verification is `entry.hash == entry.compute_hash()` plus
/// `entry.prev == previous.hash`. Both halves are needed: the first catches
/// an edited field, the second catches a removed, reordered or inserted
/// entry.
pub fn compute_hash(&self) -> String {
let mut hasher = sha2::Sha256::new();
let mut field = |bytes: &[u8]| {
// Length-prefixed, so no value can move a field boundary. See the
// module docs.
hasher.update((bytes.len() as u64).to_le_bytes());
hasher.update(bytes);
};
field(DOMAIN.as_bytes());
field(self.prev.as_bytes());
field(&self.seq.to_le_bytes());
field(&self.at_unix_ms.to_le_bytes());
field(self.method.as_bytes());
field(self.principal.as_bytes());
field(self.fingerprint.as_bytes());
field(self.scope.as_bytes());
field(self.level.as_bytes());
field(self.auth_mode.as_bytes());
field(self.decision.as_bytes());
field(self.reason.as_bytes());
let digest = hasher.finalize();
let mut out = String::with_capacity(64);
for byte in digest {
use std::fmt::Write as _;
let _ = write!(out, "{byte:02x}");
}
out
}
}
/// **The chain itself.** Open it once and append to it; it is `Send + Sync` and
/// holds its own lock, because one server writes one chain from many request
/// threads and two appends that interleaved would each claim the same `prev`.
#[derive(Debug)]
pub struct AuditChain {
path: PathBuf,
tip: Mutex<Tip>,
/// **Is every entry also a transaction?** ([`AuditChain::open_framed`]).
///
/// The primary's trail is: it goes into the box's transaction log as an
/// `Append` frame, so the twin holds it and a restore brings it back. The
/// twin's OWN trail is not, and neither is a test's — see
/// [`crate::txlog::AUDIT_LOCAL_REL`].
framed: bool,
}
/// What the next entry needs to know about the last one.
#[derive(Debug)]
struct Tip {
next_seq: u64,
hash: String,
/// Lines computed while the log was RELEASED (the data set being grown and unmounted),
/// oldest first, each already linked to the one before it: framed and written, in order,
/// once the log is taken back ([`AuditChain::flush`]).
queued: Vec<String>,
/// The box followed its primary since this tip was read: the file may hold the primary's
/// entries now, so the tip is read from the file again before the next write.
stale: bool,
/// **Lines framed and not yet in the file**, oldest first, each with its frame's sequence
/// number (perf, 2026-10-06): a framed chain stages its frame, waits for the group's fsync
/// with the tip unlocked — so concurrent decisions share one — and only then writes the line,
/// in chain order, every line whose frame is durable.
unfiled: Vec<(u64, String)>,
}
/// **What became of one decision** ([`AuditChain::record`]).
#[derive(Debug, Clone)]
pub enum Recorded {
/// In the file (and, for a framed chain, in the transaction log).
Written(AuditEntry),
/// The log is RELEASED (the data set is being grown): the entry is linked and held in
/// memory, in order, and is framed and written by the first [`AuditChain::record`] or
/// [`AuditChain::flush`] after the remount. `pending` counts the held entries.
Queued {
/// The entry, as it will be written.
entry: AuditEntry,
/// How many entries wait.
pending: usize,
},
/// The box FOLLOWS its primary (a twin, or restoring): a framed chain is the primary's
/// trail, and this box originates nothing into it. Written nowhere.
NotOriginated,
}
/// What one framed write came to.
enum Wrote {
Yes,
/// The log is released: nothing framed, nothing written.
Released,
/// The log follows: nothing framed, nothing written.
Follower,
}
impl AuditChain {
/// Open — or start — the chain at `path`.
///
/// The existing file is **read and verified** at open, not merely counted.
/// A server that appended to a chain it had not checked would extend
/// somebody's forgery with genuine entries and make the break look like it
/// happened later than it did.
pub fn open(path: impl Into<PathBuf>) -> Result<AuditChain, AuditError> {
let path = path.into();
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent).map_err(|source| AuditError::Io {
path: parent.display().to_string(),
source,
})?;
}
let entries = read_entries(&path)?;
verify_entries(&entries)?;
let tip = match entries.last() {
Some(last) => Tip {
next_seq: last.seq + 1,
hash: last.hash.clone(),
queued: Vec::new(),
stale: false,
unfiled: Vec::new(),
},
None => Tip {
next_seq: 0,
hash: GENESIS.to_owned(),
queued: Vec::new(),
stale: false,
unfiled: Vec::new(),
},
};
Ok(AuditChain {
path,
tip: Mutex::new(tip),
framed: false,
})
}
/// **Open the chain as the REPLICATED trail** — every append is an
/// `Append` transaction before it is a line in a file, so the twin holds
/// the trail and `gunnar control twin replay` brings it back.
///
/// This is the D6 fix (lane T11, 2026-09-21). Before it, `log/` was a
/// per-box path: the twin never held the audit trail and a restore came up
/// with none, reporting success. The restore happens on the worst day,
/// which is the day somebody has to reconstruct what happened.
///
/// **It refuses rather than falling back.** The chain must sit at
/// [`crate::txlog::AUDIT_REL`] under a data root the sink knows, or this
/// returns [`AuditError::Unframed`] naming why. A chain that quietly
/// wrote unframed lines would reproduce the exact bug — a trail the
/// restore drops without saying so. The caller's job is to say it out
/// loud and open an ordinary chain with [`AuditChain::open`].
pub fn open_framed(path: impl Into<PathBuf>) -> Result<AuditChain, AuditError> {
let path = path.into();
match crate::txlog::sink::log_for(&path) {
None => {
return Err(AuditError::Unframed {
path: path.display().to_string(),
why: "it is under no data root that has a transaction log — the layout put \
the `log` directory outside the data root, or this box originates \
nothing (a twin, a box being restored)"
.to_owned(),
})
}
Some((_, rel)) if rel != crate::txlog::AUDIT_REL => {
return Err(AuditError::Unframed {
path: path.display().to_string(),
why: format!(
"the log would name it {rel:?}, and a restore resolves only {:?}",
crate::txlog::AUDIT_REL
),
})
}
Some(_) => {}
}
let mut chain = AuditChain::open(path)?;
chain.framed = true;
Ok(chain)
}
/// Does every append of this chain also become a transaction? See
/// [`AuditChain::open_framed`].
pub fn is_framed(&self) -> bool {
self.framed
}
/// Where the chain lives.
pub fn path(&self) -> &Path {
&self.path
}
/// **Append one decision.** Returns the entry that was written (or, while the log is
/// released, queued — [`Recorded::Queued`]), whose `hash` is the new tip an operator can
/// record. A box that follows its primary answers [`AuditError::NotOriginated`].
pub fn append(&self, event: AuditEvent<'_>, at_unix_ms: i64) -> Result<AuditEntry, AuditError> {
match self.record(event, at_unix_ms)? {
Recorded::Written(entry) | Recorded::Queued { entry, .. } => Ok(entry),
Recorded::NotOriginated => Err(AuditError::NotOriginated {
path: self.path.display().to_string(),
}),
}
}
/// The log this chain frames into, when it is framed and the sink knows its root.
fn framing_log(&self) -> Option<std::sync::Arc<crate::txlog::Log>> {
if self.framed {
crate::txlog::sink::log_for(&self.path).map(|(log, _)| log)
} else {
None
}
}
/// **Record one decision** — [`Recorded`] says what became of it.
///
/// G + G2 (release-3 test2 mock 2026-09-29): a chain opened framed on a box that was
/// then told `BecomeTwin` at runtime, and a primary whose log was released for its own
/// growth, answered every decision with an error (`txlog: this box is a twin …`, `txlog:
/// released …`) that the gate logged at ERROR — 13 and 4 lines per mock — and the grow's
/// decisions were lost. A follower now originates nothing and says so without an error; a
/// released log holds the trail in memory, in order, and writes it after the remount.
pub fn record(&self, event: AuditEvent<'_>, at_unix_ms: i64) -> Result<Recorded, AuditError> {
// A framed decision is a mutation of the replicated trail, and its guard is taken BEFORE
// the chain's own lock and held until the file has the line: a snapshot's freeze waits
// for every guard in flight, and a thread that held the chain while it waited on the
// freeze would stop the very thread the freeze waits for (that one files its line under
// the chain's lock after its frame's fsync). The guards the frame write takes inside are
// nested and count once.
let _mutation = self.framing_log().map(|log| log.mutation());
// Poisoning is not a reason to stop writing the audit trail: a panic in
// some other request's append leaves the tip behind, and refusing every
// later record would turn one bug into a gap in the evidence.
let mut tip = self
.tip
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
let log = self.framing_log();
if log.as_ref().is_some_and(|l| l.is_follower()) {
// The file is the PRIMARY's trail now (the twin applies its frames into it): a
// local line would splice a second writer into a one-writer chain. Anything held
// from before is the primary role's, which ended.
tip.queued.clear();
tip.unfiled.clear();
tip.stale = true;
return Ok(Recorded::NotOriginated);
}
if tip.stale {
// Promoted again: continue from what the file holds NOW, not from the tip this
// box remembered before it followed.
let entries = read_entries(&self.path)?;
match entries.last() {
Some(last) => {
tip.next_seq = last.seq + 1;
tip.hash.clone_from(&last.hash);
}
None => {
tip.next_seq = 0;
tip.hash = GENESIS.to_owned();
}
}
tip.stale = false;
}
let released = log.as_ref().is_some_and(|l| l.is_released());
if !released {
self.flush_locked(&mut tip)?;
}
let mut entry = AuditEntry {
seq: tip.next_seq,
at_unix_ms,
method: event.method.to_owned(),
principal: event.principal.to_owned(),
fingerprint: event.fingerprint.to_owned(),
scope: event.scope.to_owned(),
level: event.level.to_owned(),
auth_mode: event.auth_mode.to_owned(),
decision: event.decision.to_owned(),
reason: event.reason.to_owned(),
prev: tip.hash.clone(),
hash: String::new(),
};
entry.hash = entry.compute_hash();
let line = serde_json::to_string(&entry).expect("an audit entry serialises");
debug_assert!(!line.contains('\n'), "a JSONL record holds no newline");
if released || !tip.queued.is_empty() {
return Ok(Self::hold(&mut tip, entry, line));
}
if self.framed {
// ── the group-committed path (perf, 2026-10-06) ──
match crate::txlog::sink::stage_line(&self.path, &line) {
Ok(Some(staged)) => {
tip.next_seq = entry.seq + 1;
tip.hash.clone_from(&entry.hash);
tip.unfiled.push((staged.seq(), line));
// The fsync is waited for with the tip UNLOCKED: decisions made meanwhile
// link onto this one and stage their frames behind it, and one leader's
// fdatasync makes them all durable.
drop(tip);
staged.durable().map_err(|e| AuditError::Frame {
path: self.path.display().to_string(),
detail: e.to_string(),
})?;
let mut tip = self
.tip
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
self.file_durable(&mut tip, staged.log())?;
drop(tip);
// The mutation guard goes only now: the file has the line.
drop(staged);
return Ok(Recorded::Written(entry));
}
Ok(None) => {} // under no registered root any more: the unframed write below
Err(crate::txlog::Error::Released(_)) => {
return Ok(Self::hold(&mut tip, entry, line))
}
Err(crate::txlog::Error::Follower) => {
tip.stale = true;
return Ok(Recorded::NotOriginated);
}
Err(source) => {
return Err(AuditError::Frame {
path: self.path.display().to_string(),
detail: source.to_string(),
});
}
}
}
match self.write_line(&line)? {
Wrote::Yes => {}
Wrote::Released => return Ok(Self::hold(&mut tip, entry, line)),
Wrote::Follower => {
tip.stale = true;
return Ok(Recorded::NotOriginated);
}
}
tip.next_seq = entry.seq + 1;
tip.hash.clone_from(&entry.hash);
Ok(Recorded::Written(entry))
}
/// Hold a linked entry until the log is taken back.
fn hold(tip: &mut Tip, entry: AuditEntry, line: String) -> Recorded {
tip.queued.push(line);
tip.next_seq = entry.seq + 1;
tip.hash.clone_from(&entry.hash);
Recorded::Queued {
entry,
pending: tip.queued.len(),
}
}
/// How many entries wait for the log to be taken back ([`Recorded::Queued`]).
pub fn queued(&self) -> usize {
self.tip
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.queued
.len()
}
/// **Write what waits** — every held entry, in order, once the log is taken back (the
/// remount after a growth). Answers how many were written; a log still released writes
/// none and keeps them.
pub fn flush(&self) -> Result<usize, AuditError> {
let mut tip = self
.tip
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
if self
.framing_log()
.is_some_and(|l| l.is_released() || l.is_follower())
{
return Ok(0);
}
self.flush_locked(&mut tip)
}
/// **Write every framed line whose frame is durable into the file**, oldest first, as ONE
/// write and WITHOUT an fsync: the frame is the durable copy, and the file is the view of the
/// log's frames that every start checks and rebuilds (`gunnar_server::twin::
/// rematerialise_audit_trail`, T17) — an fsync per line here bought nothing the log does not
/// already hold, at the cost of a second flush per decision. A line whose write fails stays
/// unfiled and goes with the next.
fn file_durable(&self, tip: &mut Tip, log: &crate::txlog::Log) -> Result<usize, AuditError> {
let head = log.head_seq();
let n = tip
.unfiled
.iter()
.take_while(|(seq, _)| *seq <= head)
.count();
if n == 0 {
return Ok(0);
}
let mut bytes = Vec::with_capacity(tip.unfiled[..n].iter().map(|(_, l)| l.len() + 1).sum());
for (_, l) in &tip.unfiled[..n] {
bytes.extend_from_slice(l.as_bytes());
bytes.push(b'\n');
}
let io = |source: std::io::Error| AuditError::Io {
path: self.path.display().to_string(),
source,
};
let mut file = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&self.path)
.map_err(io)?;
file.write_all(&bytes).map_err(io)?;
tip.unfiled.drain(..n);
Ok(n)
}
fn flush_locked(&self, tip: &mut Tip) -> Result<usize, AuditError> {
// Framed lines whose frames are durable go into the file first — they precede anything
// held. Their fsync is NOT forced here (that would hold the tip across an fsync and undo
// the group commit): the thread that staged each one files it. Only held lines waiting
// behind them (a release just ended) are worth an fsync of their own.
if !tip.unfiled.is_empty() {
if let Some(log) = self.framing_log() {
if !tip.queued.is_empty() {
let last = tip.unfiled.last().map(|(s, _)| *s).unwrap_or(0);
log.sync_through(last).map_err(|e| AuditError::Frame {
path: self.path.display().to_string(),
detail: e.to_string(),
})?;
}
self.file_durable(tip, &log)?;
}
if !tip.unfiled.is_empty() && !tip.queued.is_empty() {
return Ok(0);
}
}
let mut n = 0;
while let Some(line) = tip.queued.first().cloned() {
match self.write_line(&line)? {
Wrote::Yes => {
tip.queued.remove(0);
n += 1;
}
Wrote::Released | Wrote::Follower => break,
}
}
Ok(n)
}
/// Frame `line` (a framed chain) and append it to the file, durable.
fn write_line(&self, line: &str) -> Result<Wrote, AuditError> {
// **The trail is a transaction before it is a line** (lane T11). Framed FIRST and held
// until the file has the write, exactly as `attest.rs` frames an attestation: the
// mutation gate keeps a snapshot from cutting between the frame and the byte, so the
// twin can never hold a trail that is ahead of the file or behind it.
//
// NOT best effort. The gate's contract is that a failure here is one `error!` and a
// served RPC, and that contract is kept by the CALLER — what must not happen is a line
// in the file with no frame behind it, because the restore would then bring back a
// chain with a hole in the middle and `verify` would report the break as the
// primary's, months later, to somebody who cannot know it was ours.
let _framed = if self.framed {
match crate::txlog::sink::append_line(&self.path, line) {
Ok(f) => Some(f),
Err(crate::txlog::Error::Released(_)) => return Ok(Wrote::Released),
Err(crate::txlog::Error::Follower) => return Ok(Wrote::Follower),
Err(source) => {
return Err(AuditError::Frame {
path: self.path.display().to_string(),
detail: source.to_string(),
})
}
}
} else {
None
};
// G3: an UNFRAMED chain (the twin's own trail) has no log to tell it the set is off;
// the directory it lives in is gone while a growth has /data unmounted. That is the
// same state as a released log: hold the line, write it after the remount — and never
// create the directory on the root filesystem under the empty mountpoint.
if _framed.is_none() && self.path.parent().is_some_and(|p| !p.exists()) {
return Ok(Wrote::Released);
}
let io = |source: std::io::Error| AuditError::Io {
path: self.path.display().to_string(),
source,
};
let mut file = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&self.path)
.map_err(io)?;
file.write_all(format!("{line}\n").as_bytes()).map_err(io)?;
// Durable before the caller is told it landed; see the module docs.
file.sync_all().map_err(io)?;
Ok(Wrote::Yes)
}
/// Every entry, oldest first.
pub fn entries(&self) -> Result<Vec<AuditEntry>, AuditError> {
read_entries(&self.path)
}
/// **Check the chain end to end.** `Ok(n)` is "n entries, and they hold".
///
/// This is the function the finding asks for: an operator (or
/// `cargo test`) can ask the file whether anything in it has been changed
/// or removed, and get an answer that names the entry rather than a
/// shrug.
pub fn verify(&self) -> Result<usize, AuditError> {
let entries = read_entries(&self.path)?;
verify_entries(&entries)?;
Ok(entries.len())
}
/// The current tip hash — what a whole history commits to.
///
/// Worth writing down somewhere the box cannot reach: with a recorded tip,
/// even a rewrite of the entire file is detected.
pub fn tip(&self) -> String {
self.tip
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.hash
.clone()
}
}
/// Read the file, or `[]` if it is not there yet.
fn read_entries(path: &Path) -> Result<Vec<AuditEntry>, AuditError> {
let text = match std::fs::read_to_string(path) {
Ok(text) => text,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
Err(source) => {
return Err(AuditError::Io {
path: path.display().to_string(),
source,
})
}
};
let mut out = Vec::new();
for (index, line) in text.lines().enumerate() {
if line.trim().is_empty() {
continue;
}
let entry = serde_json::from_str(line).map_err(|source| AuditError::Malformed {
line: index + 1,
source,
})?;
out.push(entry);
}
Ok(out)
}
/// The arithmetic, over entries already parsed.
///
/// Separate from the file so a caller with entries in hand — a test, a future
/// reader that streams — checks them with the same code the server does (one
/// implementation).
pub fn verify_entries(entries: &[AuditEntry]) -> Result<(), AuditError> {
let mut expected_prev = GENESIS.to_owned();
for (index, entry) in entries.iter().enumerate() {
let line = index + 1;
let broken = |what: String| AuditError::Broken {
seq: entry.seq,
line,
what,
};
if entry.seq != index as u64 {
return Err(broken(format!(
"it is the {}th entry in the file and claims seq {}; an entry has been removed \
or reordered",
index, entry.seq
)));
}
if entry.prev != expected_prev {
return Err(broken(format!(
"it names {} as the entry before it and the entry before it hashes to {}; \
the chain has been cut",
short(&entry.prev),
short(&expected_prev)
)));
}
let computed = entry.compute_hash();
if entry.hash != computed {
return Err(broken(format!(
"its contents hash to {} and it carries {}; this entry has been edited",
short(&computed),
short(&entry.hash)
)));
}
expected_prev = entry.hash.clone();
}
Ok(())
}
/// A hash, shortened for a message a person reads. The full value is in the
/// file.
fn short(hash: &str) -> &str {
hash.get(..12).unwrap_or(hash)
}
#[cfg(test)]
mod tests {
use super::*;
fn event<'a>(method: &'a str, decision: &'a str) -> AuditEvent<'a> {
AuditEvent {
method,
principal: "alice",
fingerprint: "SHA256:abc",
scope: "repo:team/app",
level: "admin",
auth_mode: "required",
decision,
reason: "",
}
}
fn chain_of(path: &Path, count: usize) -> AuditChain {
let chain = AuditChain::open(path).expect("open");
for index in 0..count {
chain
.append(
event("Access.Grant", "allow"),
1_700_000_000_000 + index as i64,
)
.expect("append");
}
chain
}
/// The accept side, and the shape. Without it every tamper assertion below
/// would pass against a `verify` that refused everything.
#[test]
fn a_chain_links_each_entry_to_the_one_before_it_and_verifies() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(AUDIT_FILE);
let chain = chain_of(&path, 3);
assert_eq!(chain.verify().expect("a fresh chain verifies"), 3);
let entries = chain.entries().expect("read");
assert_eq!(entries[0].prev, GENESIS, "the first entry starts the chain");
assert_eq!(entries[1].prev, entries[0].hash);
assert_eq!(entries[2].prev, entries[1].hash);
assert_eq!(chain.tip(), entries[2].hash);
assert_eq!((entries[0].seq, entries[2].seq), (0, 2));
// Reopening continues the chain rather than starting a second one: the
// tip is read back off the file, which is what makes a restart
// invisible to a verifier.
let reopened = AuditChain::open(&path).expect("reopen");
assert_eq!(reopened.tip(), entries[2].hash);
reopened
.append(event("Repos.Delete", "deny"), 1_700_000_009_000)
.expect("append after reopen");
assert_eq!(reopened.verify().expect("still holds"), 4);
let entries = reopened.entries().expect("read");
assert_eq!(entries[3].prev, entries[2].hash);
}
/// **A modified entry is detected.** The whole point: `tracing` could not
/// say this, and this is the sentence the operator gets.
#[test]
fn an_edited_entry_is_named() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(AUDIT_FILE);
let _chain = chain_of(&path, 3);
// Rewrite the middle record's decision, the edit somebody covering a
// refusal would make, leaving everything else — the hash included —
// exactly as gunnar wrote it.
let text = std::fs::read_to_string(&path).expect("read");
let mut lines: Vec<String> = text.lines().map(str::to_owned).collect();
lines[1] = lines[1].replace("\"decision\":\"allow\"", "\"decision\":\"deny\"");
std::fs::write(&path, lines.join("\n") + "\n").expect("rewrite");
let err = AuditChain::open(&path).expect_err("a rewritten chain must not open");
match err {
AuditError::Broken {
seq,
line,
ref what,
} => {
assert_eq!((seq, line), (1, 2), "the break must be named: {what}");
assert!(what.contains("edited"), "{what}");
}
other => panic!("the edit was not detected: {other}"),
}
}
/// **A removed entry is detected** — the tamper `tracing` is least able to
/// show, because a deleted line leaves nothing behind.
#[test]
fn a_removed_entry_is_named() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(AUDIT_FILE);
let chain = chain_of(&path, 4);
assert_eq!(chain.verify().expect("green first"), 4);
let text = std::fs::read_to_string(&path).expect("read");
let lines: Vec<&str> = text.lines().collect();
// Drop the second of four.
let cut: Vec<&str> = lines
.iter()
.enumerate()
.filter(|(index, _)| *index != 1)
.map(|(_, line)| *line)
.collect();
std::fs::write(&path, cut.join("\n") + "\n").expect("rewrite");
let err = AuditChain::open(&path).expect_err("a cut chain must not open");
assert!(
matches!(
err,
AuditError::Broken {
seq: 2,
line: 2,
..
}
),
"the removal was not detected where it happened: {err}"
);
}
/// **Truncating the tail is detected too — but only against a recorded
/// tip**, and the difference is stated rather than assumed.
///
/// Cutting the LAST entries leaves a chain that is internally consistent:
/// that is what a hash chain is, and claiming otherwise would be the kind
/// of overclaim this file is here to replace. What catches it is the tip an
/// operator wrote down, so the test asserts exactly that — the shortened
/// file verifies, and its tip is not the one that was recorded.
#[test]
fn a_truncated_tail_verifies_but_does_not_match_the_recorded_tip() {
let dir = tempfile::tempdir().expect("tempdir");
let path = dir.path().join(AUDIT_FILE);
let chain = chain_of(&path, 4);
let recorded = chain.tip();
let text = std::fs::read_to_string(&path).expect("read");
let kept: Vec<&str> = text.lines().take(2).collect();
std::fs::write(&path, kept.join("\n") + "\n").expect("rewrite");
let shortened = AuditChain::open(&path).expect("a truncated chain is self-consistent");
assert_eq!(shortened.verify().expect("and verifies"), 2);
assert_ne!(
shortened.tip(),
recorded,
"a truncated chain reproduced the tip of the full one"
);
}
/// A field cannot be moved across a boundary into another field, because
/// each one is length-prefixed. Without the prefixes these two entries hash
/// the same.
#[test]
fn a_field_holding_a_separator_cannot_impersonate_two_fields() {
let base = AuditEntry {
seq: 0,
at_unix_ms: 1,
method: "Access.Grant".into(),
principal: "alice".into(),
fingerprint: "SHA256:abc".into(),
scope: "server".into(),
level: "admin".into(),
auth_mode: "required".into(),
decision: "deny".into(),
reason: String::new(),
prev: GENESIS.into(),
hash: String::new(),
};
// The pair that COLLIDES under a newline-joined concatenation: both
// sides render as `alice\nSHA256:abc\n`, and only the length prefixes
// tell them apart.
let mut shifted = base.clone();
shifted.principal = "alice\nSHA256:abc".into();
shifted.fingerprint = String::new();
// The naive concatenation of the two differs only in where the
// boundary is drawn.
assert_ne!(
base.compute_hash(),
shifted.compute_hash(),
"two entries with different field boundaries hash the same"
);
}
// ── the trail survives a restore (lane T11, 2026-09-21; the D6 finding) ──
/// A data root with a transaction log registered on it, and the audit chain where the
/// layout puts it: `<root>/log/audit-chain.jsonl`.
fn framed_root() -> (
tempfile::TempDir,
std::sync::Arc<crate::txlog::Log>,
PathBuf,
) {
let dir = tempfile::tempdir().expect("tempdir");
let log =
std::sync::Arc::new(crate::txlog::Log::open(dir.path().join("txlog")).expect("a log"));
crate::txlog::sink::register(dir.path(), &log);
let path = dir.path().join(crate::txlog::AUDIT_REL);
(dir, log, path)
}
fn a_decision(method: &str) -> AuditEvent<'_> {
AuditEvent {
method,
principal: "alice",
fingerprint: "SHA256:abc",
scope: "server",
level: "admin",
auth_mode: "required",
decision: "allow",
reason: "",
}
}
/// **Every entry of a framed chain is a transaction**, so the twin holds the audit trail
/// and `gunnar control twin replay` brings it back. That is the whole D6 fix: before it,
/// `log/` was a per-box path and a restore came up with no audit history, silently.
///
/// Proven the way a restore would see it: the frames are read back out of the log and the
/// file is rebuilt from them alone, then opened and verified.
///
/// **Fails before the fix**: make `AuditChain::open_framed` return the chain that
/// `AuditChain::open` returns (drop the `chain.framed = true;` line) and the log holds
/// nothing — `head_seq()` is 0 and the rebuilt file is empty.
#[test]
fn a_framed_chain_puts_every_entry_in_the_transaction_log() {
let (_dir, log, path) = framed_root();
let chain = AuditChain::open_framed(&path).expect("the trail is framable where it lives");
assert!(chain.is_framed());
for m in ["Repos.Create", "Access.Grant", "Policy.SetProtection"] {
chain.append(a_decision(m), 1).expect("append");
}
assert_eq!(chain.verify().expect("verify"), 3);
// What the twin holds: three `Append` frames, all naming the ONE canonical path, in
// the order the chain wrote them.
assert_eq!(log.head_seq(), 3, "one frame per decision");
let mut rebuilt = String::new();
for seq in 1..=3 {
let e = log.read(seq).expect("a frame");
assert_eq!(e.kind(), Some(crate::txlog::Kind::Append));
match e.meta().expect("meta") {
crate::txlog::Meta::Append { log: rel } => assert_eq!(rel, crate::txlog::AUDIT_REL),
other => panic!("seq {seq} is {other:?}, not an Append of the audit trail"),
}
rebuilt.push_str(std::str::from_utf8(&e.blob).expect("utf8"));
rebuilt.push('\n');
}
// A restored box rebuilds the file from the frames alone — and it verifies, with the
// same tip. That is the property the owner asked for in one assertion.
let elsewhere = tempfile::tempdir().expect("tempdir");
let restored_path = elsewhere.path().join("audit-chain.jsonl");
std::fs::write(&restored_path, rebuilt).expect("write");
let restored = AuditChain::open(&restored_path).expect("a restored trail opens");
assert_eq!(restored.verify().expect("verify"), 3);
assert_eq!(restored.tip(), chain.tip());
// And the restored box CONTINUES the chain rather than starting a new one: its next
// entry's `prev` is the dead primary's tip, so the history is unbroken across the
// disaster.
let next = restored
.append(a_decision("Repos.List"), 2)
.expect("append");
assert_eq!(next.prev, chain.tip());
assert_eq!(next.seq, 3);
}
/// **A chain that cannot be replicated REFUSES to pretend it is**, rather than quietly
/// writing unframed lines — which is the exact shape of the bug being fixed: a record the
/// restore drops without saying so.
///
/// Two ways it can happen: the `log` directory resolved outside any data root (no sink),
/// and the chain sitting somewhere other than the one path a restore resolves.
///
/// **Fails before the fix**: delete the `match crate::txlog::sink::log_for(&path)` block
/// from `AuditChain::open_framed` and both halves return `Ok`.
#[test]
fn an_unreplicable_chain_refuses_to_be_the_replicated_trail() {
let outside = tempfile::tempdir().expect("tempdir");
let err = AuditChain::open_framed(outside.path().join("audit-chain.jsonl"))
.expect_err("a chain under no data root is not the replicated trail");
assert!(matches!(err, AuditError::Unframed { .. }), "{err}");
assert!(format!("{err}").contains("under no data root"), "{err}");
let (dir, _log, _path) = framed_root();
let misplaced = dir.path().join("somewhere-else.jsonl");
let err = AuditChain::open_framed(&misplaced).expect_err("a misplaced chain is refused");
assert!(matches!(err, AuditError::Unframed { .. }), "{err}");
assert!(format!("{err}").contains(crate::txlog::AUDIT_REL), "{err}");
}
/// **An unframed chain frames nothing** — the twin's own trail
/// ([`crate::txlog::AUDIT_LOCAL_REL`]) and every test harness. A twin originates nothing,
/// and a twin that put its own decisions into the log it is holding for the primary would
/// be forging the primary's trail.
///
/// **Fails before the fix**: make `AuditChain::append` frame unconditionally (drop the
/// `if self.framed`) and the log holds the twin's own decision at seq 1.
#[test]
fn a_twins_own_trail_is_never_a_transaction() {
let (dir, log, _path) = framed_root();
let local = dir.path().join(crate::txlog::AUDIT_LOCAL_REL);
let chain = AuditChain::open(&local).expect("the twin's own trail opens");
assert!(!chain.is_framed());
chain
.append(a_decision("Mirror.LogHead"), 1)
.expect("append");
assert_eq!(
log.head_seq(),
0,
"a twin originates nothing into the primary's log"
);
assert_eq!(chain.verify().expect("verify"), 1);
let _ = log;
}
/// **G (release-3 test2 mock 2026-09-29): a box that FOLLOWS originates nothing into the
/// replicated trail, and says so without an error.** The twin opened its chain framed (it
/// started as a primary) and was then told `Bootstrap.BecomeTwin` at runtime: its log turned
/// follower, and every later decision (`Admin.GrowStatus` polls from monetize, 13 of them)
/// was an `ERROR txlog: this box is a twin (or restoring); it originates no transactions`.
///
/// A follower's decision is NOT written — not into the log (the primary's numbers), and not
/// into the file (the primary's chain, which the twin holds) — and the answer is a
/// [`Recorded::NotOriginated`], not an error. A box promoted again continues from the tip the
/// file holds THEN, not from the one it remembered before it followed.
#[test]
fn a_framed_chain_on_a_box_that_follows_originates_nothing_and_is_no_error() {
let (_dir, log, path) = framed_root();
let chain = AuditChain::open_framed(&path).expect("framed");
chain
.append(a_decision("Repos.Create"), 1)
.expect("the primary's own decision");
log.set_follower(true);
match chain
.record(a_decision("Admin.GrowStatus"), 2)
.expect("a follower's decision is not an error")
{
Recorded::NotOriginated => {}
other => panic!("a follower originates nothing: {other:?}"),
}
assert!(matches!(
chain.append(a_decision("Admin.GrowStatus"), 3),
Err(AuditError::NotOriginated { .. })
));
assert_eq!(
log.head_seq(),
1,
"nothing framed into the primary's numbers"
);
assert_eq!(
chain.verify().expect("verify"),
1,
"nothing written into the primary's chain"
);
// The primary's own frames reach the file while it follows (the twin applies them) …
let other = AuditChain::open(&path).expect("the applier's view");
let applied = other
.append(a_decision("Access.Grant"), 4)
.expect("applied");
// … and a promoted box continues from THAT tip.
log.set_follower(false);
let next = chain
.append(a_decision("Repos.List"), 5)
.expect("originates again");
assert_eq!(
(next.seq, next.prev.clone()),
(applied.seq + 1, applied.hash.clone())
);
assert_eq!(chain.verify().expect("verify"), 3);
}
/// **G2: while the data set is grown the trail WAITS, in order, and is written after the
/// remount — never an error.** MEASURED main on the same mock: 4 `ERROR txlog: released — the
/// files under /data/txlog are given up while the data set is grown` on `Admin.Capacity` /
/// `Admin.GrowStatus` during its growth, and those decisions were never in the trail.
#[test]
fn a_released_log_queues_the_trail_and_the_first_append_after_the_remount_writes_it_in_order() {
let (_dir, log, path) = framed_root();
let chain = AuditChain::open_framed(&path).expect("framed");
chain.append(a_decision("Repos.Create"), 1).expect("append");
log.release().expect("the grow releases the log");
for (i, m) in ["Admin.Capacity", "Admin.GrowStatus"]
.into_iter()
.enumerate()
{
match chain
.record(a_decision(m), 2 + i as i64)
.expect("a decision during the grow is not an error")
{
Recorded::Queued { pending, .. } => assert_eq!(pending, i + 1),
other => panic!("released: queued, not {other:?}"),
}
}
assert_eq!(chain.queued(), 2);
assert_eq!(
log.head_seq(),
1,
"nothing framed while the files are given up"
);
assert_eq!(
chain.verify().expect("verify"),
1,
"nothing written under the unmounted set"
);
log.reacquire().expect("the remount");
match chain
.record(a_decision("Admin.GrowStatus"), 9)
.expect("after the remount")
{
Recorded::Written(e) => assert_eq!(e.seq, 3),
other => panic!("written after the remount: {other:?}"),
}
assert_eq!(chain.queued(), 0);
assert_eq!(chain.verify().expect("the chain holds across the grow"), 4);
assert_eq!(log.head_seq(), 4, "every decision framed, in order");
let methods: Vec<String> = chain
.entries()
.unwrap()
.into_iter()
.map(|e| e.method)
.collect();
assert_eq!(
methods,
[
"Repos.Create",
"Admin.Capacity",
"Admin.GrowStatus",
"Admin.GrowStatus"
]
);
// `flush` writes a queue without waiting for the next decision.
log.release().unwrap();
chain.record(a_decision("Admin.Capacity"), 10).unwrap();
log.reacquire().unwrap();
assert_eq!(chain.flush().expect("flush"), 1);
assert_eq!(
(chain.queued(), chain.verify().unwrap(), log.head_seq()),
(0, 5, 5)
);
}
/// **G3 (d-upg3 2026-10-02, the KVM mock of the full release upgrade at ops fa23a53): an
/// UNFRAMED chain whose set is unmounted for a growth WAITS too — never an error.** With
/// `[machines.twin]` fed to vm1, monetize grows the TWIN; the twin's own trail
/// ([`crate::txlog::AUDIT_LOCAL_REL`]) is not framed, so nothing told it the set was off, and
/// monetize's own `Admin.GrowStatus` poll during the growth answered `ERROR … the audit chain
/// at /data/log/audit-chain.local.jsonl: No such file or directory` — the log gate's FAIL.
/// The directory the chain lives in is GONE while /data is unmounted: the line is held, in
/// order, and written by the first decision after the remount (or `flush`).
#[test]
fn an_unframed_chain_whose_set_is_unmounted_queues_and_writes_after_the_remount() {
let dir = tempfile::tempdir().unwrap();
let data = dir.path().join("data");
let local = data.join(crate::txlog::AUDIT_LOCAL_REL);
let chain = AuditChain::open(&local).expect("the twin's own trail opens");
chain
.append(a_decision("Mirror.LogHead"), 1)
.expect("before the grow");
// The growth unmounts /data: the directory the chain is in is not there.
let off = dir.path().join("data-off");
std::fs::rename(&data, &off).unwrap();
match chain
.record(a_decision("Admin.GrowStatus"), 2)
.expect("a decision while the set is off is not an error")
{
Recorded::Queued { pending, .. } => assert_eq!(pending, 1),
other => panic!("the set is off: queued, not {other:?}"),
}
assert!(
!data.exists(),
"nothing is created where the set is not mounted"
);
std::fs::rename(&off, &data).unwrap();
match chain
.record(a_decision("Admin.GrowStatus"), 3)
.expect("after the remount")
{
Recorded::Written(e) => assert_eq!(e.seq, 2),
other => panic!("written after the remount: {other:?}"),
}
assert_eq!(
(
chain.queued(),
chain.verify().expect("the chain holds across the grow")
),
(0, 3)
);
}
/// **Decisions group-committed from many threads, with a snapshot's freeze cutting in,
/// never deadlock and never fork the chain** (perf, 2026-10-06). Eight threads record while
/// another freezes the log over and over: every decision lands once, the chain verifies, the
/// file holds exactly the lines the log's frames carry, in the same order — and the whole run
/// ends (a thread that held the chain's lock while it waited on the freeze would stop the
/// thread the freeze waits for; the guard is taken before the lock).
#[test]
fn concurrent_framed_decisions_with_freezes_land_once_in_order_and_never_deadlock() {
let (_dir, log, path) = framed_root();
let chain = std::sync::Arc::new(AuditChain::open_framed(&path).expect("framed"));
let (tx, rx) = std::sync::mpsc::channel();
{
let (chain, log) = (std::sync::Arc::clone(&chain), std::sync::Arc::clone(&log));
std::thread::spawn(move || {
let stop = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false));
let freezer = {
let (log, stop) = (std::sync::Arc::clone(&log), std::sync::Arc::clone(&stop));
std::thread::spawn(move || {
let mut n = 0u64;
while !stop.load(std::sync::atomic::Ordering::SeqCst) {
let g = log.freeze();
std::thread::sleep(std::time::Duration::from_micros(200));
drop(g);
n += 1;
}
n
})
};
std::thread::scope(|s| {
for t in 0..8 {
let chain = std::sync::Arc::clone(&chain);
s.spawn(move || {
for i in 0..40 {
chain
.append(
a_decision(if (t + i) % 2 == 0 {
"Repos.Create"
} else {
"Access.Grant"
}),
i,
)
.expect("a decision lands");
}
});
}
});
stop.store(true, std::sync::atomic::Ordering::SeqCst);
let _ = tx.send(freezer.join().unwrap());
});
}
let freezes = rx
.recv_timeout(std::time::Duration::from_secs(120))
.expect("320 decisions under freezes end — no deadlock");
assert!(freezes > 0, "the freezer cut in");
assert_eq!(chain.verify().expect("the chain holds"), 320);
let file: Vec<String> = std::fs::read_to_string(&path)
.unwrap()
.lines()
.map(str::to_owned)
.collect();
let framed: Vec<String> = (1..=log.head_seq())
.map(|s| String::from_utf8(log.read(s).unwrap().blob).unwrap())
.collect();
assert_eq!(
file, framed,
"the file is exactly the log's frames, in the log's order"
);
}
}