crates/gunnar-server/src/replication_view.rs
//! **What replication looks like from outside it** — the one shape the console
//! band, `Admin.Stats` and the log all read.
//!
//! # Why this exists rather than a direct reference to the replicator
//!
//! Two problems, one answer.
//!
//! 1. **Order.** The gRPC control plane is built before the replicator is: the
//! replicator subscribes to the event bus the server already has, and it is
//! started last on purpose so that nothing above it can be slowed or failed
//! by it. A surface that held an `Arc<Replicator>` could not be constructed.
//! 2. **The feature.** `mirror.rs` is behind `feature = "replication"`, and a
//! build without it must still be able to say *"no standby is
//! configured"*. A field that vanishes with a feature makes that sentence
//! vanish too, and the whole point of this module is that silence is the
//! bug.
//!
//! So the view holds a **closure**, filled once by `serve` when the replicator
//! exists. Nothing here names `Replicator`, and a build with no replication
//! answers [`ReplicationReport::default`] — `configured: false`, which is a
//! fact, not an absence.
//!
//! # The number that matters is the one that is missing
//!
//! MEASURED: every configured standby received **nothing** from
//! `e31ab8f4` until the fix on 2026-08-30, and it said so five times out
//! of five in a log line nobody read. The field that would have caught it on
//! day one is [`StandbyRow::last_success_unix_ms`] being **zero**, and the
//! sentence is *"last successful replication: never"* — which is why
//! [`StandbyRow::never_synced`] is its own predicate and
//! [`StandbyRow::describe`] gives it its own message rather than folding it
//! into "stale".
use std::sync::Arc;
/// Everything a surface needs to draw replication, in one snapshot.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ReplicationReport {
/// **Is any standby configured at all?**
///
/// False and empty is "nothing to replicate to", which is the ordinary
/// state of a single box and is not an alarm. True and empty would be a
/// bug, and the band says so rather than drawing nothing.
pub configured: bool,
/// Jobs that entered a queue.
pub enqueued: u64,
/// Attempts a standby accepted.
pub synced: u64,
/// Attempts that failed. Never resets.
pub failed: u64,
/// Times the event-stream subscription fell behind and lost events.
pub bus_gaps: u64,
/// One row per configured standby.
pub standbys: Vec<StandbyRow>,
/// **Refs on this server that no standby carries**, and the total they are
/// counted against. See [`RefCoverage`].
pub refs: RefCoverage,
}
impl ReplicationReport {
/// Any standby whose [`StandbyRow::verdict`] is an alarm.
///
/// `behind` is in here because of the sixty seconds described on
/// [`BEHIND_AFTER_RETRY_MAX`]: a standby that has been powered off is not
/// stale for the better part of an hour, and an alarm that only watches
/// `stale` sees nothing wrong in the meantime.
pub fn alarming(&self) -> impl Iterator<Item = &StandbyRow> {
self.standbys.iter().filter(|row| row.verdict().alarming())
}
}
/// ★ **How many refs a mirror does not carry** — the number behind
/// `copied alice/korp 50 refs (94 not mirrored)`.
///
/// # The green row that was 56% of the truth
///
/// 🔴 MEASURED on the live primary 2026-08-31, during the first restore drill:
/// across all 24 replicated repositories, **1292 refs exist and 567 of them are
/// not mirrored** — `alice/korp` alone has 144 refs of which 94 never leave
/// this box. `gunnar_client::mirror::is_mirrored_ref` carries `refs/heads/*`
/// and `refs/tags/*` and nothing else, which is a deliberate choice and is not
/// changed here; what was missing is that **nothing said so**. `replication
/// list` printed `copied alice/korp 50 refs` and the console drew a green
/// row: both true, and together they give an operator the impression that the
/// standby holds the repository. It holds every branch and tag of it.
///
/// Most of the difference is `refs/remotes/origin/*` left by pushes from mirror
/// clones. Not all of it: `refs/superseded/wip-pre-reset-…`,
/// `refs/salvage/*` and `refs/original/refs` are places somebody parked
/// work **because it mattered**, and for some of them the primary is the only
/// copy.
///
/// Counted from the PRIMARY's own refs rather than from what an attempt did, so
/// a repository that has never replicated still has a number, and a repository
/// whose refs are all branches and tags reports **0** rather than nothing at
/// all. A counter that only prints when it is non-zero is not a counter.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct RefCoverage {
/// Every ref in the repository — the denominator.
pub total: u64,
/// Refs outside `refs/heads/` and `refs/tags/`: on this box only.
pub unmirrored: u64,
}
impl RefCoverage {
/// Refs a standby can hold: `total - unmirrored`.
pub fn mirrorable(&self) -> u64 {
self.total.saturating_sub(self.unmirrored)
}
/// Add another repository's count into a server-wide total.
pub fn add(&mut self, other: RefCoverage) {
self.total = self.total.saturating_add(other.total);
self.unmirrored = self.unmirrored.saturating_add(other.unmirrored);
}
}
/// ★ **What one standby's row SAYS, in one word** — decided here and nowhere
/// else (one implementation), so the CLI row, the console badge, the appliance band and the
/// alarm cannot disagree about the same standby.
///
/// # The alarm that lied
///
/// 🔴 MEASURED on the reference appliance 2026-08-31: `gunnar control replication list`
/// printed `pending 0 · failures 0 · last success <ts> · STALE` on a box with
/// **nothing wrong with it**. Staleness was `now - last_success > 10x the retry
/// ceiling` and nothing else, and the last *success* is a clock that stops the
/// moment there is nothing to replicate — so every standby on a quiet weekend
/// crossed the threshold and shouted about it. An alarm that fires on a Sunday
/// is an alarm an operator learns to ignore, which is worse than none.
///
/// The distinction the word has to carry is **fallen behind** versus **nothing
/// to do**, and it is not a question about time since a success. It is a
/// question about outstanding work:
///
/// * [`Verdict::Idle`] — nothing queued, nothing in flight, nothing failing.
/// There is no work, so the age of the last success measures the weekend and
/// not the standby.
/// * [`Verdict::Working`] — work in hand, inside the retry ceiling. An ordinary
/// push on its way.
/// * [`Verdict::Behind`] — work outstanding for longer than the retry ceiling
/// ([`BEHIND_AFTER_RETRY_MAX`]). Five minutes with the shipped defaults.
/// * [`Verdict::Stale`] — **and** nothing has landed for ten times the ceiling.
/// Fifty minutes with the shipped defaults, and it is now a statement about
/// work that is not moving rather than about a quiet box.
/// * [`Verdict::Never`] — not one byte, ever. Its own word because it needs its
/// own action: `resync` after fixing the address or the key, not "wait".
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum Verdict {
/// Not one byte has ever reached this standby.
Never,
/// Work is stuck AND nothing has landed for [`STALENESS_FACTOR`]x the
/// retry ceiling.
Stale,
/// Work outstanding longer than the retry ceiling.
Behind,
/// Work in hand, inside the ceiling.
Working,
/// Nothing outstanding and nothing failing: there is nothing to do.
#[default]
Idle,
}
impl Verdict {
/// The word every surface prints. Upper case is an alarm; lower case is
/// not, and that is deliberate — the shape of the row says which it is
/// before the word is read.
pub const fn word(self) -> &'static str {
match self {
Verdict::Never => "NEVER",
Verdict::Stale => "STALE",
Verdict::Behind => "BEHIND",
Verdict::Working => "syncing",
Verdict::Idle => "idle",
}
}
/// Does this word deserve somebody's attention tonight?
pub const fn alarming(self) -> bool {
matches!(self, Verdict::Never | Verdict::Stale | Verdict::Behind)
}
}
/// **Is a standby that HAS synced now stale?** The one computation behind
/// [`StandbyRow::stale`], as a pure function so it can be seen red without a
/// live standby (measured, not asserted).
///
/// Two conditions, and the second is the fix: nothing has landed for
/// `stale_after_ms`, **and** there is evidence the standby is not keeping up —
/// work stuck past the retry ceiling, or an attempt that has already failed.
/// Without the second condition an idle box reads STALE, which is what it did.
///
/// It deliberately does NOT fire on work that has only just arrived. A push
/// after a quiet weekend is outstanding for a second or two against a last
/// success that is hours old; calling that stale would flap an alarm on every
/// ordinary push, and `behind` is the number that separates the two.
pub fn is_stale(
now_unix_ms: i64,
last_success_unix_ms: i64,
consecutive_failures: u32,
behind: bool,
stale_after_ms: i64,
) -> bool {
if last_success_unix_ms <= 0 {
// NEVER is its own verdict and its own sentence. Saying "stale" about
// it too would fold the two back together.
return false;
}
let struggling = behind || consecutive_failures > 0;
struggling && now_unix_ms.saturating_sub(last_success_unix_ms) > stale_after_ms
}
/// One standby's state, flattened.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct StandbyRow {
/// The operator's name for it.
pub name: String,
/// Where it is.
pub remote: String,
/// Repositories with work outstanding.
pub pending: u64,
/// Consecutive failures for the job at the head of the queue.
pub consecutive_failures: u32,
/// The last failure since the last success.
pub last_error: String,
/// **Zero means NEVER**, and that is the value this whole module exists
/// for. Not "unknown", not "just now".
pub last_success_unix_ms: i64,
/// A hole retrying will not close; `gunnar control replication resync` is
/// the fix.
pub needs_full_resync: bool,
/// **Work is stuck and nothing has landed for [`STALENESS_FACTOR`]x the
/// retry ceiling.** See [`is_stale`] — this is NOT "the last success is
/// old", which is what it used to be and what made an idle box shout.
/// *Never* is not folded in here; it is [`Verdict::Never`].
pub stale: bool,
/// **The repository an attempt is in flight for**, empty when none.
pub in_flight: String,
/// **How long the oldest outstanding repository has been outstanding**, ms.
/// Zero when there is none.
pub outstanding_for_ms: i64,
/// Outstanding work older than the retry ceiling. See [`BEHIND_AFTER_RETRY_MAX`].
pub behind: bool,
}
impl StandbyRow {
/// **Has a byte ever reached this standby?**
///
/// The distinction is not cosmetic. "It has not synced for six hours" is a
/// standby that was working and stopped; "it has never synced" is a
/// standby that has never worked, and the second one means the durability
/// promise on this box does not exist. They need different sentences
/// because they need different actions.
pub fn never_synced(&self) -> bool {
self.last_success_unix_ms <= 0
}
/// **The one word this row says.** See [`Verdict`].
///
/// Every surface asks this rather than reading the booleans itself: the
/// console badge, the CLI row, the band and the alarm all print what this
/// returns, so "is it stale?" has exactly one answer per snapshot.
pub fn verdict(&self) -> Verdict {
if self.never_synced() {
Verdict::Never
} else if self.stale {
Verdict::Stale
} else if self.behind {
Verdict::Behind
} else if self.pending > 0 {
Verdict::Working
} else {
Verdict::Idle
}
}
/// Seconds since the last success, or `None` when there has never been
/// one.
pub fn since_last_success(&self, now_unix_ms: i64) -> Option<i64> {
if self.never_synced() {
return None;
}
Some(((now_unix_ms - self.last_success_unix_ms).max(0)) / 1000)
}
/// The band's row and the log line, one writer (one implementation).
///
/// ```text
/// standby synced 105 failed 40 last 00:00:03 ago idle
/// standby synced 0 failed 8 last successful replication: NEVER
/// ```
///
/// The word comes from [`Self::verdict`], so `idle` is printed rather than
/// left blank: a row with no word on it is a row an operator has to guess
/// about, and guessing is what `STALE` on a healthy box taught them to do.
/// `BEHIND` is not repeated here — [`Self::outstanding`] carries it beside
/// the work it is about.
pub fn describe(&self, synced: u64, failed: u64, now_unix_ms: i64) -> String {
let word = match self.verdict() {
Verdict::Behind | Verdict::Working => String::new(),
verdict => format!(" {}", verdict.word()),
};
let head = match self.since_last_success(now_unix_ms) {
Some(secs) => format!(
"{} synced {synced} failed {failed} last {} ago{word}",
self.name,
hms(secs),
),
None => format!(
"{} synced {synced} failed {failed} last successful replication: NEVER",
self.name
),
};
format!("{head}{}", self.outstanding())
}
/// **The clock that keeps ticking while a standby is dead**, as the tail of
/// a row: ` pending 3 in flight alice/x outstanding 00:01:03 BEHIND`.
///
/// Empty when nothing is outstanding, so a healthy idle standby's row is
/// unchanged.
pub fn outstanding(&self) -> String {
if self.pending == 0 && self.outstanding_for_ms <= 0 {
return String::new();
}
let mut tail = format!(" pending {}", self.pending);
if !self.in_flight.is_empty() {
tail.push_str(&format!(" in flight {}", self.in_flight));
}
if self.outstanding_for_ms > 0 {
tail.push_str(&format!(
" outstanding {}",
hms(self.outstanding_for_ms / 1000)
));
}
if self.behind {
tail.push_str(" BEHIND");
}
tail
}
}
/// `HH:MM:SS` from a count of seconds, saturating at three digits of hours.
pub fn hms(secs: i64) -> String {
let secs = secs.max(0);
format!(
"{:02}:{:02}:{:02}",
secs / 3600,
(secs % 3600) / 60,
secs % 60
)
}
/// **How long WITH WORK STUCK ON IT and no success before a standby is STALE.**
///
/// Ten times the retry ceiling. A multiple of the retry rather than a
/// wall-clock constant, because that is the number that separates *"this has
/// stopped working"* from *"this is between attempts"*: at the cap the worker
/// tries every `DEFAULT_RETRY_MAX`, so ten missed attempts in a row is not a
/// slow standby, it is a broken one. With the shipped defaults, 50 minutes.
///
/// 🔴 The threshold on its own was the whole test until 2026-08-31, and on its
/// own it is a clock that stops: with nothing to replicate, every healthy
/// standby crosses it and the row shouts `STALE` at an operator who has done
/// nothing wrong. [`is_stale`] is the other half — see [`Verdict`].
pub const STALENESS_FACTOR: u32 = 10;
/// **How long outstanding work may sit before the standby is BEHIND**, as a
/// multiple of the retry ceiling.
///
/// One. At the ceiling the worker retries every `retry_max`, so a repository
/// still outstanding after a full ceiling has had at least one whole attempt
/// cycle fail to converge. With the shipped defaults, **five minutes**.
///
/// # Why this exists at all — the fifty minutes that said nothing
///
/// 🔴 MEASURED on the reference appliance 2026-08-31: the standby was **hard
/// powered-off** with a push outstanding, and sixty seconds later its row read
/// `pending 0 failures 0 last success <a minute ago>`. Every number on it
/// said healthy. `stale` could not help — it is `STALENESS_FACTOR` × the retry
/// ceiling, **fifty minutes** with the shipped defaults, and it is measured
/// from the last *success*, which is a clock that has stopped.
///
/// `behind` is measured from the oldest **outstanding work**, which is a clock
/// that keeps ticking. It is a different fact from `stale` and is reported
/// separately rather than folded in: *"nothing has landed for fifty minutes"*
/// and *"something has been trying to land for five"* need different sentences
/// because a standby can be either without being the other.
///
/// It will also fire on a repository that legitimately takes longer than the
/// ceiling to ship — and that is the right trade. A repository that has been
/// replicating for five minutes is something an operator should be able to see;
/// the row names it (`in flight <repo>`) so which of the two it is can be read
/// off the same line.
pub const BEHIND_AFTER_RETRY_MAX: u32 = 1;
/// **The one snapshot every surface draws replication from** — the console
/// band, `Admin.Stats`, and the staleness alarm (one implementation).
///
/// A standby that has **never** synced alarms from the moment the replicator
/// starts, and it is the case this whole module exists for: from `e31ab8f4` to
/// 2026-08-30 every configured standby on this project received nothing, and
/// the only thing that said so was a log line nobody read. Waiting out a
/// threshold before saying so would be the same silence, only shorter. It is
/// [`Verdict::Never`] rather than [`Verdict::Stale`], because the two need
/// different actions.
///
/// # The threshold uses the CONFIGURED ceiling, not the clamped one
///
/// `retry_max_ms` below its `retry_base_ms` is a misconfiguration the worker
/// clamps for itself. This does not repeat that clamp — a second copy of it
/// would be a second thing to keep in step (one implementation) — and it does not need to:
/// a threshold computed from a too-small ceiling makes the alarm fire
/// *earlier*, which is the safe direction for an alarm.
///
/// # It takes a `Replicator`, not a slot or a server
///
/// The smallest thing that has the numbers. Replication's lifecycle around it has
/// been in motion (a live, reconfigurable slot is being built beside it), and
/// a reporting function pinned to whichever wrapper is current would have to
/// move every time that does. `Replicator` is the object that holds the
/// counters and it is not going anywhere.
#[cfg(feature = "replication")]
pub fn report(replicator: &crate::mirror::Replicator) -> ReplicationReport {
let counters = replicator.counters();
let now = gunnar_grpc::bus::now_unix_ms();
let stale_after_ms =
(replicator.config().retry_max_ms as i64).saturating_mul(i64::from(STALENESS_FACTOR));
let behind_after_ms =
(replicator.config().retry_max_ms as i64).saturating_mul(i64::from(BEHIND_AFTER_RETRY_MAX));
ReplicationReport {
configured: true,
enqueued: counters.enqueued,
synced: counters.succeeded,
failed: counters.failed,
bus_gaps: counters.bus_gaps,
// **No walk here.** This function is the cheap operational report —
// counters, verdicts, the "never synced" zero — and every one of its
// callers is a POLLED surface (`Admin.Stats`, the appliance band,
// `Mirror.ReplicationStatus`). Counting the refs a mirror does not
// carry opens every repository's reference database, and a polled
// surface paying that on a timer is how a status page became a load.
// So `refs` starts at zeros; the surface that wants the number
// fills it — `Mirror.ReplicationStatus` with `include_refs` takes a
// FRESH walk, and the polled surfaces peek the last one via
// [`crate::mirror::Replicator::ref_coverage_cached`] and never walk.
refs: RefCoverage::default(),
standbys: replicator
.status()
.into_iter()
.map(|status| {
let last = status.last_success_unix_ms.unwrap_or(0);
let outstanding_for_ms = status
.outstanding_for
.map(|d| d.as_millis().min(i64::MAX as u128) as i64)
.unwrap_or(0);
let behind = outstanding_for_ms > behind_after_ms;
StandbyRow {
stale: is_stale(
now,
last,
status.consecutive_failures,
behind,
stale_after_ms,
),
behind,
in_flight: status.in_flight.unwrap_or_default(),
outstanding_for_ms,
name: status.name,
remote: status.remote,
pending: status.pending as u64,
consecutive_failures: status.consecutive_failures,
last_error: status.last_error.unwrap_or_default(),
last_success_unix_ms: last,
needs_full_resync: status.needs_full_resync,
}
})
.collect(),
}
}
/// A slot `serve` fills once the replicator exists.
///
/// Cloneable and cheap: every surface holds the same `Arc`, and the closure is
/// set exactly once. Reading before it is set answers "nothing configured",
/// which is what a server with no replication *is*.
#[derive(Clone, Default)]
pub struct ReplicationView {
source: Arc<std::sync::OnceLock<Box<dyn Fn() -> ReplicationReport + Send + Sync>>>,
}
impl ReplicationView {
/// An empty view. Reads as "nothing configured" until [`Self::fill`].
pub fn new() -> Self {
Self::default()
}
/// Fill it. The second call is ignored: one server has one replicator, and
/// silently replacing the source would make which one is being read depend
/// on start-up order.
pub fn fill(&self, source: impl Fn() -> ReplicationReport + Send + Sync + 'static) {
let _ = self.source.set(Box::new(source));
}
/// The snapshot, now.
pub fn report(&self) -> ReplicationReport {
match self.source.get() {
Some(source) => source(),
None => ReplicationReport::default(),
}
}
}
impl std::fmt::Debug for ReplicationView {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("ReplicationView")
.field("filled", &self.source.get().is_some())
.finish()
}
}
/// How often the staleness watch looks. Thirty seconds: the threshold is
/// measured in tens of minutes, so the poll period is noise against it, and it
/// costs a few atomic loads.
pub const STALENESS_POLL: std::time::Duration = std::time::Duration::from_secs(30);
/// **The staleness alarm.** One writer for the log line and the event-stream event
/// (one implementation), so the console, `journalctl`, the UDP narration and every
/// `Events.Watch`/SSE subscriber say the same thing about the same standby.
///
/// **Edge-triggered.** A warning emitted every poll is the log line nobody
/// read, with more of it: this fires once when a standby goes stale and once
/// when it comes back. The band draws the state continuously; the alarm marks
/// the transition.
///
/// The event is a `ReplicationFailed` on the existing stream rather than a new
/// event kind, because that is exactly what it is — a replication that is not
/// happening — and because a subscriber written before this change already
/// knows how to render one. `attempt` carries the consecutive-failure count and
/// `reason` carries the sentence.
///
/// `needs_full_resync` is **true for a standby that has never synced**: there
/// is no partial state to catch up from, and a resync is the operator's next
/// step once the address or the key is fixed.
pub fn spawn_staleness_watch(
view: ReplicationView,
bus: std::sync::Arc<gunnar_grpc::EventBus>,
) -> tokio::task::JoinHandle<()> {
// A32-14, stays a task: its handle is kept by serve and aborted at the server's stop.
tokio::spawn(async move {
let mut alarmed: Vec<String> = Vec::new();
loop {
let report = view.report();
for row in &report.standbys {
let verdict = row.verdict();
let alarming = verdict.alarming();
let was = alarmed.iter().any(|name| name == &row.name);
if alarming && !was {
alarmed.push(row.name.clone());
let reason = if verdict == Verdict::Behind {
// **The sixty seconds that read as healthy.** A standby
// that has just been powered off is not stale for the
// better part of an hour, and the only number moving in
// the meantime is how long its work has been waiting.
tracing::warn!(
target: "gunnar::replication",
event = "replication.behind",
standby = %row.name,
remote = %row.remote,
pending = row.pending,
in_flight = %row.in_flight,
outstanding_for_ms = row.outstanding_for_ms,
"this standby has had work outstanding for longer than the retry \
ceiling. It may be unreachable, or one repository may simply be \
large; the row names which."
);
format!(
"{} repository(ies) outstanding for {}, longer than the retry \
ceiling{}",
row.pending,
hms(row.outstanding_for_ms / 1000),
if row.in_flight.is_empty() {
String::new()
} else {
format!("; in flight: {}", row.in_flight)
}
)
} else if verdict == Verdict::Never {
tracing::error!(
target: "gunnar::replication",
event = "replication.never_synced",
standby = %row.name,
remote = %row.remote,
failed = report.failed,
"LAST SUCCESSFUL REPLICATION: NEVER. This standby has never taken a \
single byte, so there is no second copy of anything on it. Check its \
address, its host key and the deploy key in replication.json."
);
"last successful replication: NEVER — this standby has never taken a byte"
.to_owned()
} else {
tracing::warn!(
target: "gunnar::replication",
event = "replication.stale",
standby = %row.name,
remote = %row.remote,
last_success_unix_ms = row.last_success_unix_ms,
"this standby has not taken a byte for more than {STALENESS_FACTOR}x \
the retry ceiling; its copy is behind by an unknown amount"
);
format!(
"no successful replication for more than {STALENESS_FACTOR}x the \
retry ceiling"
)
};
bus.publish(gunnar_proto::event::Payload::ReplicationFailed(
gunnar_proto::ReplicationFailed {
repo: None,
target: row.name.clone(),
remote_url: row.remote.clone(),
attempt: row.consecutive_failures,
retry_in_ms: 0,
reason,
needs_full_resync: row.needs_full_resync || row.never_synced(),
},
));
} else if !alarming && was {
alarmed.retain(|name| name != &row.name);
tracing::info!(
target: "gunnar::replication",
event = "replication.recovered",
standby = %row.name,
"this standby is taking bytes again"
);
}
}
// A standby that was removed from the configuration stops being
// alarming; keeping its name would make a re-added one silent.
alarmed.retain(|name| report.standbys.iter().any(|row| &row.name == name));
tokio::time::sleep(STALENESS_POLL).await;
}
})
}
#[cfg(test)]
mod tests {
use super::*;
/// **"never" is its own sentence.** The line that would have caught weeks
/// of silence on day one, asserted as a string because a string is what an
/// operator reads.
#[test]
fn a_standby_that_has_never_synced_says_so_in_those_words() {
let never = StandbyRow {
name: "standby".into(),
last_success_unix_ms: 0,
stale: true,
..StandbyRow::default()
};
let line = never.describe(0, 8, 1_000_000);
assert!(
line.contains("last successful replication: NEVER"),
"a standby that has never synced must say so: {line}"
);
assert!(never.never_synced());
assert_eq!(never.since_last_success(1_000_000), None);
// …and one that HAS synced must not say it, or the sentence is noise.
let worked = StandbyRow {
name: "standby".into(),
last_success_unix_ms: 997_000,
..StandbyRow::default()
};
let line = worked.describe(105, 40, 1_000_000);
assert!(!line.contains("NEVER"), "{line}");
assert!(line.contains("synced 105"), "{line}");
assert!(line.contains("failed 40"), "{line}");
assert!(line.contains("last 00:00:03 ago"), "{line}");
}
/// ★ **The sixty seconds that read as healthy.**
///
/// 🔴 MEASURED on the reference appliance 2026-08-31: the standby was hard
/// powered-off with a push outstanding, and a minute later its row read
/// `pending 0 failures 0 last success <a minute ago>` — every number said
/// healthy. `stale` could not help: it is `STALENESS_FACTOR` × the retry
/// ceiling, **fifty minutes**, measured from the last *success*, which is a
/// clock that has stopped.
///
/// Both directions are asserted. A row that said BEHIND whenever anything
/// was outstanding would flag every ordinary push, so the healthy-but-busy
/// case is named by value too.
#[test]
fn a_standby_with_work_stuck_on_it_says_so_long_before_it_is_stale() {
let now = 1_000_000_000_000i64;
// A minute after a power-off: the last success is recent, so `stale` is
// false and must stay false — this is the state the measurement caught.
let dead = StandbyRow {
name: "standby-fi-hel1".into(),
pending: 1,
in_flight: "alice/linux-mirror".into(),
outstanding_for_ms: 60_000,
last_success_unix_ms: now - 61_000,
stale: false,
behind: false,
..StandbyRow::default()
};
let line = dead.describe(105, 0, now);
assert!(
line.contains("pending 1") && line.contains("in flight alice/linux-mirror"),
"the row must name the outstanding work, or `pending 0 failures 0` is all an \
operator sees: {line}"
);
assert!(
line.contains("outstanding 00:01:00"),
"the row must carry the clock that is still ticking: {line}"
);
// Past the ceiling, it is BEHIND — and that is an alarm, at five
// minutes rather than fifty.
let behind = StandbyRow {
outstanding_for_ms: 400_000,
behind: true,
..dead.clone()
};
assert!(behind.describe(105, 0, now).contains("BEHIND"));
let report = ReplicationReport {
configured: true,
standbys: vec![dead.clone(), behind.clone()],
..ReplicationReport::default()
};
let alarming: Vec<&str> = report.alarming().map(|r| r.in_flight.as_str()).collect();
assert_eq!(
alarming.len(),
1,
"exactly the BEHIND row must alarm; a row that alarmed on any outstanding work \
would fire on every ordinary push"
);
// A standby with nothing outstanding says nothing extra at all, or the
// tail is noise on every healthy row.
let idle = StandbyRow {
name: "standby".into(),
last_success_unix_ms: now - 3_000,
..StandbyRow::default()
};
assert_eq!(idle.outstanding(), "");
assert_eq!(
idle.describe(105, 0, now),
"standby synced 105 failed 0 last 00:00:03 ago idle"
);
}
/// ★ **The alarm that lied: `STALE` on a healthy idle box.**
///
/// 🔴 MEASURED on the reference appliance 2026-08-31 —
/// `pending 0 · failures 0 · last success <ts> · STALE`, with nothing wrong
/// at all. Staleness was `now - last_success > 10x the retry ceiling` and
/// nothing else, and the last *success* is a clock that stops when there is
/// nothing left to replicate: a weekend with no pushes takes every standby
/// past the threshold.
///
/// Seen RED by restoring that one line
/// (`stale: last <= 0 || now - last > stale_after_ms`): the first assertion
/// below fails, and the two after it — the standby that really IS behind —
/// keep passing, which is the half that makes this a fix rather than a
/// deleted alarm.
#[test]
fn an_idle_standby_is_not_stale_and_a_stuck_one_still_is() {
let hour = 3_600_000i64;
let now = 1_000_000_000_000i64;
let stale_after = 50 * 60 * 1000; // the shipped default: 10 x 5 minutes
// A quiet Sunday: nothing queued, nothing in flight, nothing failing,
// and the last success is hours old BECAUSE THERE WAS NOTHING TO DO.
assert!(
!is_stale(now, now - 6 * hour, 0, false, stale_after),
"a standby with nothing to do read STALE — this is the measured lie"
);
let quiet = StandbyRow {
name: "standby".into(),
last_success_unix_ms: now - 6 * hour,
..StandbyRow::default()
};
assert_eq!(quiet.verdict(), Verdict::Idle);
assert!(!quiet.verdict().alarming());
assert!(quiet.describe(105, 0, now).contains(" idle"));
// …and the alarm still fires on a standby that has fallen behind:
// work stuck past the retry ceiling, nothing landing for ten times it.
assert!(
is_stale(now, now - 6 * hour, 0, true, stale_after),
"a standby with work stuck on it for hours must still be STALE"
);
// A failed attempt is the other evidence: the queue may be short, but
// an attempt that failed is not a quiet Sunday.
assert!(is_stale(now, now - 6 * hour, 3, false, stale_after));
// Inside the window it is not stale yet, however stuck it is — that is
// what `behind` is for, and it fires ten times sooner.
assert!(!is_stale(now, now - 60_000, 3, true, stale_after));
// And a push that has only just arrived after a quiet weekend must NOT
// trip the alarm on its way through: outstanding, but not yet behind.
assert!(!is_stale(now, now - 6 * hour, 0, false, stale_after));
let stuck = StandbyRow {
name: "standby".into(),
pending: 1,
in_flight: "alice/korp".into(),
outstanding_for_ms: 6 * hour,
last_success_unix_ms: now - 6 * hour,
stale: true,
behind: true,
..StandbyRow::default()
};
assert_eq!(stuck.verdict(), Verdict::Stale);
assert!(stuck.verdict().alarming());
assert!(stuck.describe(105, 4, now).contains("STALE"));
// NEVER is not folded into stale: it needs a different sentence and a
// different action.
let never = StandbyRow {
name: "standby".into(),
..StandbyRow::default()
};
assert_eq!(never.verdict(), Verdict::Never);
assert!(never.verdict().alarming());
assert!(!is_stale(now, 0, 0, false, stale_after));
// An ordinary push on its way is neither an alarm nor "idle".
let working = StandbyRow {
name: "standby".into(),
pending: 1,
outstanding_for_ms: 900,
last_success_unix_ms: now - 3_000,
..StandbyRow::default()
};
assert_eq!(working.verdict(), Verdict::Working);
assert!(!working.verdict().alarming());
}
/// The shortfall counter reports **zero**, not a blank, for a repository
/// whose refs a mirror carries entirely.
#[test]
fn ref_coverage_counts_zero_as_a_number() {
let all_carried = RefCoverage {
total: 20,
unmirrored: 0,
};
assert_eq!(all_carried.mirrorable(), 20);
let korp = RefCoverage {
total: 144,
unmirrored: 94,
};
assert_eq!(
korp.mirrorable(),
50,
"the number `replication list` prints"
);
let mut server = RefCoverage::default();
server.add(all_carried);
server.add(korp);
assert_eq!((server.total, server.unmirrored), (164, 94));
}
/// An unfilled view is "nothing configured", not a panic and not an empty
/// success that looks like a healthy standby.
#[test]
fn an_unfilled_view_reports_nothing_configured() {
let view = ReplicationView::new();
let report = view.report();
assert!(!report.configured);
assert!(report.standbys.is_empty());
}
#[test]
fn a_filled_view_reports_what_the_source_says() {
let view = ReplicationView::new();
view.fill(|| ReplicationReport {
configured: true,
synced: 105,
failed: 40,
..ReplicationReport::default()
});
let report = view.report();
assert!(report.configured);
assert_eq!(report.synced, 105);
// A second fill is ignored; one server has one replicator.
view.fill(ReplicationReport::default);
assert_eq!(view.report().synced, 105);
}
}