crates/gunnar-server/src/capacity.rs
//! **"If a tenant's cap rose to N bytes, could you serve it?"** — `Admin.Capacity`.
//!
//! The question a seller must ask before the money moves. monetize grows an
//! "increase my quota" button: a tenant on 10 GB asks for 100 GB and something
//! upstream is about to buy 90 GB from a cloud vendor and charge for it. gunnar
//! is the only component that knows whether those bytes can ever reach a
//! repository, and on the appliance the honest answer is frequently **no**.
//!
//! # Two facts, not one number
//!
//! A single free-bytes answer is the dangerous one: a caller reads a small
//! number as "buy another disk", and on the appliance that is money converted
//! into nothing. So [`answer`] returns a `verdict` — can this request be served
//! out of what exists right now — **and**, independently, a `growth` — can this
//! data set ever hold more than it does, and by whose hand. `growth` is
//! reported on the yes arm as well, because "yes, and this is the last capacity
//! this box will ever have" is a different sale from "yes, and there is more
//! where that came from".
//!
//! ## When the appliance is SEALED — and when it is NOT
//!
//! SEALED was once the only answer, on three facts:
//!
//! * `/data` is an md set laid once by `korp-installer`'s `lay_set`
//! (`edda/crates/korp-installer/src/install.rs`), which runs **only** on
//! `InstallMode::Fresh`.
//! * gunnar had no data-set growth of any kind.
//! * gunnar is **PID 1 with no shell**, so nothing on the machine can
//! `xfs_growfs` a newly attached disk either.
//!
//! The second is no longer true, and [`decide_growth`] must not say it is.
//! `korp-start` carries the boot-time grow engine
//! (`korp_installer::grow`, `DATA-SET-GROWTH-DESIGN.md`) and, under the
//! `grow-runtime` feature, THIS server links the same engine's runtime half
//! (`korp_installer::grow::runtime`, driven by [`crate::grow`]). So the
//! verdict is a ladder over MEASURED facts — the journal on the box
//! ([`GrowthFacts::grow`]), this build's engine ([`GrowthFacts::runtime_engine`])
//! and the kernel's PCI hot-plug ([`GrowthFacts::hotplug`]) — and SEALED is
//! only its bottom rung, reached when **this box keeps no grow journal**.
//! Nothing here may claim "no such path in any build": that is a statement
//! about the binary, and the binary is measured one field away.
//!
//! On that bottom rung a disk bought and attached to the box is **billed and
//! invisible**. That is what [`SetGrowth::GrowthSealed`] says, in those words,
//! in the reason line — so a caller that renders nothing but the sentence
//! still cannot mistake it for "not enough space right now".
//!
//! # Measured, never declared, and measured ONCE (one implementation)
//!
//! The bytes are [`gunnar_ssh::DiskHeadroomGate::headroom`] — the same
//! `statvfs` the push door refuses under, so the capacity a seller is quoted
//! and the floor a push dies at cannot disagree. The set's shape is
//! [`crate::dashboard::probe::disk_kind`] — the same `/proc/mounts` + md sysfs
//! read the console's header band prints `4×75G raid0 xfs` from. This module
//! opens no third spelling of either question; it decides over their output.
//!
//! [`answer`] is pure over [`Question`] precisely so the SEALED arm can be
//! driven red on a developer box, where gunnar is never PID 1 and `/data` is
//! never an md set.
//!
//! # The box: cores and RAM, measured (product scope)
//!
//! Disk is a tenant's: a cap is bytes in one namespace. Cores and RAM are the
//! **box's**: no tenant owns a core, so the ceiling a seller may sell either
//! up to is whatever this machine has — and that must be read off the
//! machine, not declared in a config that was true of the plan somebody
//! ordered. [`MachineFacts`] reads them the way the rest of the server
//! already does (one implementation): `available_parallelism` is what
//! [`crate::budget::Machine`] sizes the thread pools from, `/proc/meminfo`
//! is [`crate::dashboard::probe::meminfo`] behind the header band's `ram`
//! cell, the cgroup clamp is [`crate::budget::cgroup_memory`]'s, and the
//! load is the band's `load1`.
//!
//! Every box field on the wire is `optional`, and **absent means unmeasured**
//! — never `0`. A `0` is a number a caller will divide by, sell against, or
//! read as "this box has no RAM"; an absence with `machine_reason` naming
//! the OS error is the same UNMEASURED discipline the disk half has.
use std::path::{Path, PathBuf};
use gunnar_proto::{CapacityResponse, CapacityVerdict, SetGrowth};
use gunnar_ssh::{headroom::human, Headroom};
use crate::dashboard::probe::{self, DiskKind};
/// **What the machine says about the set under the data root.** Two measured
/// facts and nothing inferred; [`decide_growth`] turns them into a verdict.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct GrowthFacts {
/// The mount and the array under it, as
/// [`crate::dashboard::probe::disk_kind`] reads them. `None` when the data
/// root is not a mount point of its own, or `/proc/mounts` is unreadable —
/// which is the honest UNKNOWN, not a growable answer.
pub kind: Option<DiskKind>,
/// **This process is PID 1.** On the appliance gunnar IS init: there is no
/// shell beside it, so nothing on the machine can grow a filesystem
/// out of band. On a developer box or in a container beside an init this
/// is false and an operator's hands still exist.
pub is_pid1: bool,
/// **The boot stack's grow journal** (`/serialdat/grow-journal`,
/// `DATA-SET-GROWTH-DESIGN.md`), when this box's `korp-start` keeps one —
/// which is the measured fact that the boot stack CAN grow the set at a
/// restart. `None` on an image older than the engine, where the
/// appliance is as sealed as it ever was.
pub grow: Option<gunnar_grpc::grow_journal::Summary>,
/// **This gunnar-server links the runtime grow engine** (`crate::grow`,
/// feature `grow-runtime`): with the journal on the box, the set grows
/// WITHOUT a restart through `Admin.GrowStart`. A build fact, read once.
pub runtime_engine: bool,
/// **The running kernel has PCI hot-plug**
/// ([`crate::dashboard::probe::pci_hotplug`]). UpCloud resizes only
/// DETACHED volumes, so without it a member detached and re-attached while
/// gunnar runs is invisible: the engine is there, the growth is not.
pub hotplug: bool,
}
impl GrowthFacts {
/// Read them off the running machine.
pub fn measure(root: &Path) -> GrowthFacts {
GrowthFacts {
kind: crate::dashboard::probe::disk_kind(root),
is_pid1: std::process::id() == 1,
grow: gunnar_grpc::grow_journal::Summary::measure(),
runtime_engine: cfg!(feature = "grow-runtime"),
hotplug: crate::dashboard::probe::pci_hotplug().present,
}
}
}
/// The growth verdict and the clause the reason line carries for it.
pub fn decide_growth(facts: &GrowthFacts, root: &Path) -> (SetGrowth, String) {
let root = root.display();
match (&facts.kind, facts.is_pid1) {
(None, _) => (
SetGrowth::GrowthUnknown,
format!(
"gunnar cannot tell what {root} sits on — it is not a mount point of its own, \
so whether this set can grow is UNKNOWN, which is not a yes"
),
),
// The boot stack carries the grow engine (its journal is on this
// box) AND this server links it: the set grows WITHOUT a restart
// (`Admin.GrowStart`: drain, unmount, wait for larger members, grow,
// mount, reopen — DATA-SET-GROWTH-FLOW.md §3). The restart path
// below still holds as the fallback for a crash mid-way.
(Some(kind), true) if facts.grow.is_some() && facts.runtime_engine && facts.hotplug => {
let journal = facts.grow.as_ref().expect("guarded");
let n = kind.set.as_ref().map(|s| s.members).unwrap_or(1);
(
SetGrowth::GrowthAtRuntime,
format!(
"AND THE SET GROWS WITHOUT A RESTART: {root} is a {kind} set, this gunnar-server \
links the grow engine, the kernel has PCI hot-plug and the journal is on this box — `Admin.GrowStart` drains \
the box, unmounts, waits for ALL {n} members to come back larger at one equal \
size, grows in place and serves again in minutes; a crash mid-way is finished \
at the next boot. Grow journal: {}",
journal.detail()
),
)
}
// The boot stack carries the grow engine (its journal is on this
// box): the set grows at the NEXT RESTART into enlarged member
// volumes. Still no hot-plug, still no shell — an attached disk is
// as wasted as under SEALED, and that is said in the same breath.
(Some(kind), true) if facts.grow.is_some() => {
let journal = facts.grow.as_ref().expect("guarded");
let n = kind.set.as_ref().map(|s| s.members).unwrap_or(1);
(
SetGrowth::GrowthAtRestart,
format!(
"AND THE SET GROWS AT RESTART: {root} is a {kind} set and this box's boot stack \
(korp-start) grows it into enlarged member volumes at the next boot — enlarge \
ALL {n} members to one equal size, restart, and the next measurement reads the \
larger set. It does NOT grow while gunnar runs{}, and a disk ATTACHED to this \
machine is still invisible to it. Grow journal: {}",
if facts.runtime_engine && !facts.hotplug {
" (the grow engine is linked, but the running kernel has NO PCI hot-plug: a member detached and re-attached while gunnar runs is invisible, so a runtime growth is refused by name)"
} else {
""
},
journal.detail()
),
)
}
(Some(kind), true) => {
// The `lay_set` clause is claimed only where an md set was
// actually measured. A single-disk PID-1 box is just as sealed —
// no growth path, no shell — but it was not laid by
// `korp-installer`'s array pass, and saying so would be inventing
// a fact about a machine nobody looked at.
let laid = if kind.set.is_some() {
" laid once at install (korp-installer `lay_set`, InstallMode::Fresh only)"
} else {
""
};
// The MEASURED reason this box cannot grow is that it keeps no
// grow journal — its boot stack predates the engine. Whether
// THIS binary links the runtime half is a separate fact, and it
// is measured (`runtime_engine`), so the line names it instead
// of asserting "no such path in any build" over a binary that
// may well carry one.
//
// "so it refuses by name" below was a CLAIM until 2026-09-18 and
// is a fact since: an audit found `GrowMachine::start` had no
// journal precondition, so a `GrowStart` on exactly the box this
// arm describes was accepted and drained the gate before failing
// at `note(Step::Drain)`. The refusal now sits in
// `grow::GrowMachine::start`, on the SAME file `facts.grow`
// measures here, so this sentence and that door cannot disagree.
let engine = if facts.runtime_engine {
" (this server links the runtime grow engine, but there is no grow journal on this box for it to continue, so it refuses by name)"
} else {
""
};
(
SetGrowth::GrowthSealed,
format!(
"AND THE SET CANNOT GROW WHILE GUNNAR RUNS: {root} is a {kind} set{laid}, \
this box's boot stack keeps no grow journal{engine}, and gunnar is PID 1 \
with no shell — so a disk attached to this machine would be UNUSED AND \
INVISIBLE. Serving more needs a fresh install onto a larger set, not \
another disk"
),
)
}
(Some(kind), false) => (
SetGrowth::GrowthOperatorOnly,
format!(
"gunnar will not grow the set itself here — the grow engine runs only where \
gunnar is PID 1 over a set its own boot stack laid, and this process is not \
PID 1 — but that is also why an operator with a shell can extend {root} \
({kind}) out of band, and the next measurement will see it"
),
),
}
}
/// **What the box has: cores, RAM, and how busy it is.** Product-scope facts,
/// each either measured or carrying the reason it was not; [`machine_fields`]
/// turns them into the wire's `optional` fields and the `machine_reason`.
#[derive(Debug, Clone, PartialEq)]
pub struct MachineFacts {
/// CPUs this process may **run on**: the affinity mask, further limited by
/// a cgroup v2 `cpu.max` quota when one is set
/// (`std::thread::available_parallelism`, the spelling
/// [`crate::budget::Machine`] sizes the pools from). `Err` is the OS's
/// reason the kernel would not say.
pub cpus: Result<usize, String>,
/// `(MemTotal, MemAvailable)` in bytes as `/proc/meminfo` reports them,
/// then clamped by `cgroup_memory_max` when a limit is set. `Err` names why
/// `/proc/meminfo` could not be read or parsed.
pub memory: Result<(u64, u64), String>,
/// The cgroup v2 `memory.max` that clamped `memory`, when one did and was
/// smaller than the box — reported so the reason line can say the quoted
/// RAM is the container's, not the host's.
pub cgroup_memory_max: Option<u64>,
/// The 1-minute load average, or why `/proc/loadavg` could not be read.
pub load1: Result<f64, String>,
/// **What the data root's disk did at start-up**: the last outcome of
/// [`crate::disktest`], or `None` when the test has not finished one (the
/// gate was closed, or the thread is still running). A LOOKUP — reading
/// this never starts a disk test, because the answer wanted is what the
/// disk did at start-up on an idle box, not what it does under whatever
/// load is on it now.
pub disk: Option<crate::disktest::Outcome>,
}
impl MachineFacts {
/// Read them off the running machine. Every reader is one the server
/// already has (one implementation); this only keeps their failures instead of
/// flattening them to a dash or a zero.
pub fn measure() -> MachineFacts {
let cpus = std::thread::available_parallelism()
.map(|n| n.get())
.map_err(|err| format!("the affinity mask could not be read ({err})"));
let mut memory = probe::meminfo();
let mut cgroup_memory_max = None;
// The clamp `budget::Machine::read` applies, for the reason it gives:
// inside a cgroup `/proc/meminfo` is the HOST's memory, and the limit
// is the smaller truth. The appliance has no limit; there this is a
// no-op.
if let (Ok((total, available)), Some((cap, used))) =
(&memory, crate::budget::cgroup_memory())
{
if cap < *total {
cgroup_memory_max = Some(cap);
memory = Ok((cap, (*available).min(cap.saturating_sub(used))));
}
}
MachineFacts {
cpus,
memory,
cgroup_memory_max,
load1: probe::load1_value(),
disk: crate::disktest::last(),
}
}
}
/// The wire's `disk_*` fields out of one [`crate::disktest::Outcome`], and
/// nothing else set — the same shape and the same contract as
/// [`machine_fields`], so both arms of [`answer`] fill them the same way.
///
/// **Absent, never zero, and never a fast disk.** Every arm below either
/// fills a number that was really measured or leaves the field `None` and
/// says why in `disk_reason`. The one subtle case is `cached`: the read leg
/// came back over [`crate::disktest::CACHED_BPS`], which means the page cache
/// answered and the device was never touched, so `disk_seq_read_bps` is
/// ABSENT while the write and sync legs of the same run stand. A caller
/// comparing two boxes must be able to tell "this disk is slow" from "this
/// disk was not measured", and that is the whole reason this function exists
/// instead of a `unwrap_or(0)`.
pub fn disk_fields(disk: &Option<crate::disktest::Outcome>) -> CapacityResponse {
use crate::disktest::{Outcome, CACHED_BPS};
use gunnar_proto::DiskTest;
let bps = |v: f64| -> Option<u64> {
// A negative or non-finite rate is not a measurement.
(v.is_finite() && v > 0.0).then(|| v.round() as u64)
};
match disk {
None => CapacityResponse {
disk_test: DiskTest::DisktestNotRun as i32,
disk_reason: "The start-up disk test has not finished a run in this process: its \
gate is closed (not the appliance's PID 1 and GUNNAR_DISKTEST unset) \
or it is still running. Every disk_* speed field is absent — this is \
NOT a report of a slow disk."
.to_owned(),
..Default::default()
},
Some(Outcome::Skipped { free, need }) => CapacityResponse {
disk_test: DiskTest::DisktestSkipped as i32,
disk_reason: format!(
"The start-up disk test SKIPPED: {} free on the data root, below the {} it \
needs, and the test must never be the thing that fills the disk. Every disk_* \
speed field is absent — nothing was measured.",
human(*free),
human(*need)
),
..Default::default()
},
Some(Outcome::Failed(why)) => CapacityResponse {
disk_test: DiskTest::DisktestFailed as i32,
disk_reason: format!(
"The start-up disk test FAILED ({why}). Every disk_* speed field is absent — \
the run did not produce numbers."
),
..Default::default()
},
Some(Outcome::Done(r)) => {
let cached = r.seq_read_bps > CACHED_BPS;
let mut clauses = Vec::with_capacity(3);
let disk_seq_write_bps = bps(r.seq_write_bps);
match disk_seq_write_bps {
Some(_) => clauses.push(format!(
"sequential write {}",
crate::disktest::fmt_rate(r.seq_write_bps)
)),
None => clauses
.push("sequential write UNMEASURED (the leg reported no rate)".to_owned()),
}
let disk_seq_read_bps = if cached {
clauses.push(format!(
"sequential read CACHED and therefore ABSENT: the leg came back at {} \
({}), over the {} bar, so the page cache answered and the device was never \
touched — this is not a fast disk, it is no reading of the disk at all",
crate::disktest::fmt_rate(r.seq_read_bps),
r.read_how.as_str(),
crate::disktest::fmt_rate(CACHED_BPS)
));
None
} else {
let v = bps(r.seq_read_bps);
match v {
Some(_) => clauses.push(format!(
"sequential read {} ({})",
crate::disktest::fmt_rate(r.seq_read_bps),
r.read_how.as_str()
)),
None => clauses
.push("sequential read UNMEASURED (the leg reported no rate)".to_owned()),
}
v
};
let disk_sync_milliops_per_s = (r.sync_ops_per_s.is_finite() && r.sync_ops_per_s > 0.0)
.then(|| (r.sync_ops_per_s * 1000.0).round() as u64);
match disk_sync_milliops_per_s {
Some(_) => clauses.push(format!(
"4 KiB overwrite + fsync {:.0} ops/s",
r.sync_ops_per_s
)),
None => {
clauses.push("4 KiB fsync UNMEASURED (the leg reported no rate)".to_owned())
}
}
let us =
|d: std::time::Duration| Some(u64::try_from(d.as_micros()).unwrap_or(u64::MAX));
CapacityResponse {
disk_test: DiskTest::DisktestMeasured as i32,
disk_seq_write_bps,
disk_seq_read_bps,
disk_sync_milliops_per_s,
disk_read_how: r.read_how.as_str().to_owned(),
// What the three rates are made of, raw: the same run, its
// bytes and each leg's own clock. Present even when the read
// came back cached — the TIME is still what happened; it is
// the rate derived from it that is not a disk's.
disk_test_bytes: Some(r.file_bytes),
disk_write_us: us(r.write_elapsed),
disk_read_us: us(r.read_elapsed),
disk_sync_ops: Some(r.sync_ops),
disk_sync_us: us(r.sync_elapsed),
disk_test_us: us(r.elapsed),
disk_reason: format!(
"This disk, at start-up, on an idle box: {}. Taken in {:.1} s.",
clauses.join("; "),
r.elapsed.as_secs_f64()
),
..Default::default()
}
}
}
}
/// The wire's box fields out of [`MachineFacts`] — and nothing else set, so a
/// caller builds a response as `CapacityResponse { ..., ..machine_fields(&m) }`
/// and both arms of [`answer`] fill these the same way (one implementation).
///
/// **Absent, never zero.** An `Err` fact leaves its fields `None` and puts the
/// reason in `machine_reason`; `machine_reason` is never empty, because a
/// caller that renders only the strings must still learn what was measured.
pub fn machine_fields(m: &MachineFacts) -> CapacityResponse {
let mut clauses = Vec::with_capacity(3);
let cpu_millicores_total = match &m.cpus {
Ok(n) => {
clauses.push(format!(
"{n} CPUs = {} millicores gunnar may run on (affinity mask, cgroup cpu.max limit \
applied)",
n * 1000
));
Some((*n as u64) * 1000)
}
Err(why) => {
clauses.push(format!(
"cores UNMEASURED ({why}), so cpu_millicores_total is absent"
));
None
}
};
let (ram_bytes_total, ram_bytes_available) = match &m.memory {
Ok((total, available)) => {
let clamp = match m.cgroup_memory_max {
Some(cap) => format!(
" — clamped to the cgroup memory.max of {}, which is what this process \
may use; /proc/meminfo reports the host",
human(cap)
),
None => String::new(),
};
clauses.push(format!(
"RAM {} total, {} available (/proc/meminfo MemTotal/MemAvailable{clamp})",
human(*total),
human(*available)
));
(Some(*total), Some(*available))
}
Err(why) => {
clauses.push(format!(
"RAM UNMEASURED ({why}), so ram_bytes_total and ram_bytes_available are absent"
));
(None, None)
}
};
let cpu_millicores_busy = match &m.load1 {
Ok(load) => {
let busy = (load * 1000.0).round().max(0.0) as u64;
clauses.push(format!(
"1-min load {load:.2} ≈ {busy} millicores busy (/proc/loadavg × 1000: an \
APPROXIMATION of the whole box's runnable tasks, not gunnar's own CPU time)"
));
Some(busy)
}
Err(why) => {
clauses.push(format!(
"load UNMEASURED ({why}), so cpu_millicores_busy is absent"
));
None
}
};
// The disk's own fields ride along, because every caller builds its
// answer as `CapacityResponse { ..., ..machine_fields(&m) }` and two
// struct updates cannot compose — one seam, one implementation, and the
// disk can never be filled on one arm of `answer` and not the other.
let d = disk_fields(&m.disk);
CapacityResponse {
cpu_millicores_total,
ram_bytes_total,
ram_bytes_available,
cpu_millicores_busy,
machine_reason: format!("This box: {}.", clauses.join("; ")),
..d
}
}
/// **Every disk on the box, for `Admin.Capacity { disks: true }`** (2026-09-28).
///
/// edda's product-neutral `disk-snapshot` does the reading — the mount table,
/// one statvfs per storage mount, `/sys/block` — and this names gunnar's own
/// paths on it: `system` is `/`, `data` the store root (the explode read cache
/// lives inside each repository there, so it is ON this mount and not a disk of
/// its own), `slow` the off-stripe root. The floor is the `--disk-floor` in
/// force, applied per mount, so the `data` row's floor is the number the push
/// door refuses at.
pub fn measure_disks(
data_root: &Path,
slow_root: Option<&Path>,
floor: gunnar_ssh::DiskFloor,
) -> gunnar_proto::DiskSnapshot {
let mut volumes = vec![
disk_snapshot::Volume::new("system", "/"),
disk_snapshot::Volume::new("data", data_root),
];
if let Some(slow) = slow_root {
volumes.push(disk_snapshot::Volume::new("serialdat", slow));
}
let snap = disk_snapshot::take(
&volumes,
disk_snapshot::Floor {
percent: floor.percent,
min_bytes: floor.bytes,
},
);
disks_to_wire(snap, gunnar_core::now_ms())
}
/// **What is on /serialdat, thing by thing** — the known paths gunnar keeps
/// there, each measured as the blocks it holds (a directory summed), plus
/// `other`: the mount's used bytes no known path accounts for.
///
/// `known` is `(label, path)`; a path that does not exist holds 0 bytes and is
/// measured as such (nothing is there). A path whose walk failed is
/// `measured: false` with the error, and is left out of the sum, so `other`
/// is never made smaller by a number nobody read. `used` is the mount's own
/// statvfs used bytes; `None` (unmeasured mount) → no `other` row.
pub fn serialdat_breakdown(
known: &[(&str, PathBuf)],
used: Option<u64>,
) -> Vec<gunnar_proto::PathUse> {
let mut rows: Vec<gunnar_proto::PathUse> = known
.iter()
.map(|(label, path)| {
let mut row = gunnar_proto::PathUse {
label: (*label).to_owned(),
path: path.display().to_string(),
..Default::default()
};
match allocated_bytes(path) {
Ok(bytes) => {
row.measured = true;
row.bytes = bytes;
}
Err(why) => row.error = why,
}
row
})
.collect();
// A known path INSIDE another (the audit trail inside the log directory)
// is counted once: the outer row keeps only what the inner ones do not.
let inner: Vec<(PathBuf, u64)> = known
.iter()
.zip(rows.iter())
.filter(|(_, r)| r.measured)
.map(|((_, p), r)| (p.clone(), r.bytes))
.collect();
for ((_, outer), row) in known.iter().zip(rows.iter_mut()) {
if !row.measured {
continue;
}
let nested: u64 = inner
.iter()
.filter(|(p, _)| p != outer && p.starts_with(outer))
.map(|(_, b)| *b)
.sum();
row.bytes = row.bytes.saturating_sub(nested);
}
if let Some(used) = used {
let known_sum: u64 = rows.iter().filter(|r| r.measured).map(|r| r.bytes).sum();
rows.push(gunnar_proto::PathUse {
label: "other".to_owned(),
path: String::new(),
measured: true,
bytes: used.saturating_sub(known_sum),
error: String::new(),
});
}
rows
}
/// The blocks a path holds on disk (`st_blocks × 512`), a directory summed
/// without following symlinks. A missing path is 0: nothing is there.
fn allocated_bytes(path: &Path) -> Result<u64, String> {
use std::os::unix::fs::MetadataExt as _;
let meta = match std::fs::symlink_metadata(path) {
Ok(m) => m,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(0),
Err(e) => return Err(format!("{}: {e}", path.display())),
};
let mut total = meta.blocks() * 512;
if meta.is_dir() {
let entries = std::fs::read_dir(path).map_err(|e| format!("{}: {e}", path.display()))?;
for entry in entries {
let entry = entry.map_err(|e| format!("{}: {e}", path.display()))?;
total += allocated_bytes(&entry.path())?;
}
}
Ok(total)
}
/// The snapshot, as the wire carries it. Pure, so a recorded snapshot can be
/// checked field by field: an unmeasured mount keeps `measured = false` and its
/// reason, and its numbers stay 0 rather than being invented.
pub fn disks_to_wire(snap: disk_snapshot::Snapshot, at_unix_ms: i64) -> gunnar_proto::DiskSnapshot {
let mounts = snap
.disks
.into_iter()
.map(|d| {
let mut row = gunnar_proto::DiskMount {
source: d.mount.source,
mount_point: d.mount.mount_point.display().to_string(),
fs_type: d.mount.fstype,
read_only: d.mount.read_only,
labels: d.labels,
floor_bytes: d.floor_bytes,
servable_bytes: d.servable_bytes,
..Default::default()
};
match d.figures {
Ok(f) => {
row.measured = true;
row.total_bytes = f.total_bytes;
row.free_bytes = f.free_bytes;
row.available_bytes = f.available_bytes;
row.used_bytes = f.used_bytes;
row.reserved_bytes = f.reserved_bytes;
row.inodes_total = f.inodes_total;
row.inodes_free = f.inodes_free;
}
Err(why) => row.error = why,
}
row
})
.collect();
let devices = snap
.block_devices
.into_iter()
.map(|b| gunnar_proto::BlockDevice {
name: b.name,
size_bytes: b.size_bytes,
rotational: b.rotational,
removable: b.removable,
read_only: b.read_only,
partitions: b
.partitions
.into_iter()
.map(|(name, size_bytes)| gunnar_proto::BlockPartition { name, size_bytes })
.collect(),
holders: b.holders,
members: b.members,
raid_level: b.raid_level,
})
.collect();
let unresolved = snap
.unresolved
.into_iter()
.map(|u| gunnar_proto::UnresolvedDisk {
label: u.label,
path: u.path.display().to_string(),
why: u.why,
})
.collect();
gunnar_proto::DiskSnapshot {
measured_at_unix_ms: at_unix_ms,
mounts,
devices,
unresolved,
problems: snap.problems,
// Filled by the caller, which knows gunnar's own paths on /serialdat.
serialdat: Vec::new(),
}
}
/// Everything [`answer`] decides from. Built by
/// [`crate::control_plane::CoreControlPlane`] out of the disk gate and the
/// namespace's meter; built by hand in the tests, which is the point.
#[derive(Debug, Clone)]
pub struct Question {
/// The store root the numbers are of.
pub root: PathBuf,
/// The one `statvfs`, or the OS's reason it failed. `Err` is
/// [`CapacityVerdict::CapacityUnmeasured`] — never a yes.
pub room: Result<Headroom, String>,
/// `--disk-floor` as configured, for the reason line.
pub floor_spec: String,
/// The cap being contemplated.
pub requested_bytes: u64,
/// What the namespace already holds (`Usage.Read`'s `pack_bytes`).
pub held_bytes: u64,
/// Whose cap is rising; empty asks about bytes that are not here yet.
pub namespace: String,
pub growth: GrowthFacts,
/// The box, for the product-scope fields. Independent of the question:
/// the same cores and RAM are reported whatever cap was asked about.
pub machine: MachineFacts,
}
/// **The answer.** Pure: every field below is arithmetic over [`Question`].
pub fn answer(q: &Question) -> CapacityResponse {
let (growth, growth_clause) = decide_growth(&q.growth, &q.root);
// The box fields, once for both arms: what the machine has does not
// depend on whether the disk could be measured.
let mut machine = machine_fields(&q.machine);
// The journal rides both arms: what the boot stack did about the set is
// a fact whether or not the filesystem could be measured now.
machine.grow_state = q.growth.grow.as_ref().map(|s| s.to_wire());
machine.hotplug = q.growth.hotplug;
let data_set = q
.growth
.kind
.as_ref()
.map(|k| k.to_string())
.unwrap_or_default();
let root = q.root.display();
// Bytes the raise actually needs: what a tenant already holds is already
// spent out of free space, so a 10 GB tenant asking for 100 GB needs 90,
// not 100. An unnamed namespace has held 0 and needs the whole cap.
let additional = q.requested_bytes.saturating_sub(q.held_bytes);
let asked = if q.namespace.is_empty() {
format!("a cap of {}", human(q.requested_bytes))
} else {
format!(
"raising {} to {} (it holds {})",
q.namespace,
human(q.requested_bytes),
human(q.held_bytes)
)
};
let room = match &q.room {
Err(err) => {
return CapacityResponse {
verdict: CapacityVerdict::CapacityUnmeasured as i32,
growth: growth as i32,
root: q.root.display().to_string(),
requested_bytes: q.requested_bytes,
held_bytes: q.held_bytes,
additional_bytes: additional,
data_set,
reason: format!(
"{root} could not be measured ({err}), so gunnar will not answer that {asked} \
can be served — an unmeasured box has not said yes. {growth_clause}"
),
// Nothing was measured, so nothing is reported. A plausible
// number here would be a blind guard, asserting nothing.
total_bytes: 0,
free_bytes: 0,
floor_bytes: 0,
servable_bytes: 0,
shortfall_bytes: 0,
..machine
};
}
Ok(room) => *room,
};
// The floor is space that EXISTS and may never be sold: it is what lets the
// box compact a pack and rewrite its control documents after it has stopped
// accepting pushes. Quoting against `free` rather than against this is how
// a seller sells the reserve.
let servable = room.free.saturating_sub(room.floor);
let measured = format!(
"{root} has {} free of {}, {} of it reserved by the disk floor (--disk-floor {}), so {} \
may be served",
human(room.free),
human(room.total),
human(room.floor),
q.floor_spec,
human(servable),
);
// One constructor for both arms (one implementation): the verdict, the shortfall and
// the clause differ; the twelve measured fields do not, and writing them
// twice is how the two arms drift into reporting different things.
let (verdict, shortfall, outcome) = if additional <= servable {
(
CapacityVerdict::CapacityAvailable,
0,
format!("needs {} more: YES, it can be served.", human(additional)),
)
} else {
let shortfall = additional - servable;
(
CapacityVerdict::CapacityExhausted,
shortfall,
format!(
"needs {} more, which is {} beyond that: NO.",
human(additional),
human(shortfall)
),
)
};
CapacityResponse {
verdict: verdict as i32,
growth: growth as i32,
root: q.root.display().to_string(),
total_bytes: room.total,
free_bytes: room.free,
floor_bytes: room.floor,
servable_bytes: servable,
requested_bytes: q.requested_bytes,
held_bytes: q.held_bytes,
additional_bytes: additional,
shortfall_bytes: shortfall,
data_set,
reason: format!("{measured}. {asked} {outcome} {growth_clause}"),
..machine
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::dashboard::probe::SetFacts;
const GIB: u64 = 1024 * 1024 * 1024;
/// The live gunnar.rs appliance, as `/proc/mounts` and the md sysfs read
/// on it: `/data` is a 4×75 GiB raid0 xfs set and gunnar is PID 1.
fn appliance() -> GrowthFacts {
GrowthFacts {
kind: Some(DiskKind {
fs: "xfs".into(),
set: Some(SetFacts {
level: "raid0".into(),
members: 4,
member_bytes: 75 * GIB,
chunk_bytes: 512 * 1024,
}),
}),
is_pid1: true,
grow: None,
runtime_engine: false,
hotplug: false,
}
}
/// The same disk under an init, with a shell beside gunnar.
fn developer_box() -> GrowthFacts {
GrowthFacts {
is_pid1: false,
..appliance()
}
}
/// The appliance re-imaged with the grow engine in its boot stack: the
/// journal exists, with `text` in it.
fn growing_appliance(text: &str) -> GrowthFacts {
GrowthFacts {
grow: Some(gunnar_grpc::grow_journal::Summary::of(
text,
"/serialdat/grow-journal",
)),
..appliance()
}
}
/// **The runtime arm** (`DATA-SET-GROWTH-FLOW.md` ruling 1): the journal
/// on the box AND this server linking the engine is `GROWTH_AT_RUNTIME`;
/// either fact alone is not — the journal without the engine is the
/// restart arm, the engine without the journal is SEALED (measured, not
/// declared). The clause names the verb and "ALL 4 members".
#[test]
fn the_engine_in_the_server_with_the_journal_on_the_box_grows_at_runtime() {
let both = GrowthFacts {
runtime_engine: true,
hotplug: true,
..growing_appliance("")
};
let (v, line) = decide_growth(&both, Path::new("/data"));
assert_eq!(v, SetGrowth::GrowthAtRuntime);
assert!(
line.contains("WITHOUT A RESTART")
&& line.contains("Admin.GrowStart")
&& line.contains("ALL 4 members"),
"{line}"
);
let engine_only = GrowthFacts {
runtime_engine: true,
hotplug: true,
..appliance()
};
assert_eq!(
decide_growth(&engine_only, Path::new("/data")).0,
SetGrowth::GrowthSealed,
"no journal: the engine alone proves nothing"
);
assert_eq!(
decide_growth(&growing_appliance(""), Path::new("/data")).0,
SetGrowth::GrowthAtRestart,
"journal alone: the restart arm"
);
}
/// **No hot-plug, no runtime growth** (owner's rule 2026-09-15): the engine
/// linked and the journal on the box, but a kernel without PCI hot-plug, is
/// `GROWTH_AT_RESTART` with the reason named — and `hotplug` rides the wire
/// on every arm so monetize stops guessing.
#[test]
fn the_engine_without_a_hotplug_kernel_grows_only_at_restart_and_says_why() {
let no_hp = GrowthFacts {
runtime_engine: true,
hotplug: false,
..growing_appliance("")
};
let (v, line) = decide_growth(&no_hp, Path::new("/data"));
assert_eq!(v, SetGrowth::GrowthAtRestart);
assert!(
line.contains("NO PCI hot-plug") && line.contains("refused by name"),
"{line}"
);
let mut q = question(gunnar_rs_today(), no_hp.clone());
q.requested_bytes = 1;
assert!(!answer(&q).hotplug);
q.growth.hotplug = true;
let a = answer(&q);
assert!(
a.hotplug && a.growth == SetGrowth::GrowthAtRuntime as i32,
"{a:?}"
);
q.room = Err("EIO".into());
assert!(answer(&q).hotplug, "the unmeasured arm carries it too");
}
/// **The verdict the grow engine adds** (`DATA-SET-GROWTH-DESIGN.md`
/// §3, §5): a PID-1 md-set box whose boot stack keeps the journal is
/// `GROWTH_AT_RESTART`, never SEALED — and the clause still says an
/// attached disk is wasted, names "ALL 4 members" and carries the
/// journal's own sentence (idle / in progress / done / PARTIAL). The
/// journal rides the wire on the yes arm, the no arm and the unmeasured
/// arm alike; a box with no journal has no `grow_state` and stays SEALED.
#[test]
fn a_boot_stack_with_the_grow_engine_grows_at_restart_and_reports_the_journal() {
let (v, line) = decide_growth(&growing_appliance(""), Path::new("/data"));
assert_eq!(v, SetGrowth::GrowthAtRestart);
assert!(
line.contains("GROWS AT RESTART")
&& line.contains("ALL 4 members")
&& line.contains("invisible to it"),
"{line}"
);
assert!(line.contains("no grow has run"), "{line}");
assert_eq!(
decide_growth(&appliance(), Path::new("/data")).0,
SetGrowth::GrowthSealed,
"no journal: sealed as before"
);
// partial resize: the last boot refused, and the clause says which members grew
let partial = "epoch=0 target_sectors=134215680 member=0 step=PARTIAL at=1\nepoch=0 target_sectors=134215680 member=1 step=PARTIAL at=1\n";
let (v, line) = decide_growth(&growing_appliance(partial), Path::new("/data"));
assert_eq!(v, SetGrowth::GrowthAtRestart);
assert!(
line.contains("PARTIAL RESIZE: 2 member(s) [0, 1]"),
"{line}"
);
// on the wire, on every arm
let done = "epoch=1 target_sectors=134215680 member=- step=G1 at=1\nepoch=1 target_sectors=134215680 member=- step=G6 at=2\n";
let mut q = question(gunnar_rs_today(), growing_appliance(done));
q.requested_bytes = 1;
let a = answer(&q);
assert_eq!(a.growth, SetGrowth::GrowthAtRestart as i32);
let g = a.grow_state.expect("the journal is on the wire");
assert!(
g.engine_present
&& g.phase == "done"
&& g.epoch == 1
&& g.target_sectors == 134_215_680
&& g.journal_path == "/serialdat/grow-journal",
"{g:?}"
);
assert!(a.reason.contains("epoch 1 done"), "{}", a.reason);
q.requested_bytes = 1000 * GIB;
assert!(answer(&q).grow_state.is_some(), "the no arm carries it too");
q.room = Err("EIO".into());
assert!(answer(&q).grow_state.is_some(), "and the unmeasured arm");
let q = question(gunnar_rs_today(), appliance());
assert!(
answer(&q).grow_state.is_none(),
"no journal, no grow_state — an absence, never an empty one"
);
}
/// The live gunnar.rs appliance's box: the UpCloud STARTER plan, 4 vCPU
/// and 16 GiB, no cgroup (gunnar is PID 1), a quiet minute.
fn appliance_box() -> MachineFacts {
MachineFacts {
cpus: Ok(4),
memory: Ok((16 * GIB, 12 * GIB)),
cgroup_memory_max: None,
load1: Ok(0.42),
disk: Some(crate::disktest::Outcome::Done(crate::disktest::Report {
seq_write_bps: 812e6,
seq_read_bps: 1.9e9,
read_how: crate::disktest::ReadHow::Direct,
sync_ops_per_s: 3100.0,
elapsed: std::time::Duration::from_millis(1600),
file_bytes: 256 << 20,
write_elapsed: std::time::Duration::from_millis(330),
read_elapsed: std::time::Duration::from_millis(141),
sync_ops: 1550,
sync_elapsed: std::time::Duration::from_millis(500),
})),
}
}
fn question(room: Headroom, growth: GrowthFacts) -> Question {
Question {
root: PathBuf::from("/data"),
room: Ok(room),
floor_spec: "5%,2.0 GiB".into(),
requested_bytes: 0,
held_bytes: 0,
namespace: String::new(),
growth,
machine: appliance_box(),
}
}
/// gunnar.rs today: 300 GB of set, ~230 GB used, the 5 % floor in force.
fn gunnar_rs_today() -> Headroom {
let total = 300 * GIB;
Headroom {
total,
free: 70 * GIB,
floor: (total / 100) * 5,
}
}
#[test]
fn the_growth_verdict_separates_a_sealed_appliance_from_a_box_with_a_shell() {
let (sealed, sealed_line) = decide_growth(&appliance(), Path::new("/data"));
let (operator, operator_line) = decide_growth(&developer_box(), Path::new("/data"));
let (unknown, unknown_line) = decide_growth(
&GrowthFacts {
kind: None,
is_pid1: false,
grow: None,
runtime_engine: false,
hotplug: false,
},
Path::new("/tmp/x"),
);
// Three answers, never two: an unmeasured set is not a growable one,
// and a sealed set is not merely "gunnar will not do it".
assert_eq!(sealed, SetGrowth::GrowthSealed);
assert_eq!(operator, SetGrowth::GrowthOperatorOnly);
assert_eq!(unknown, SetGrowth::GrowthUnknown);
assert!(
sealed != operator && operator != unknown && sealed != unknown,
"the same facts must not collapse to one verdict"
);
// The sealed clause says the thing that stops a purchase, in words.
assert!(
sealed_line.contains("UNUSED AND INVISIBLE"),
"the sealed clause must say an attached disk is wasted: {sealed_line}"
);
assert!(
sealed_line.contains("fresh install") && sealed_line.contains("not another disk"),
"and must name what WOULD work: {sealed_line}"
);
assert!(
sealed_line.contains("4\u{d7}75G raid0 xfs"),
"and must name the measured set, not a generic one: {sealed_line}"
);
// The difference is real, not a wording variant: the operator arm
// promises the opposite thing.
assert!(
!operator_line.contains("UNUSED AND INVISIBLE")
&& operator_line.contains("an operator with a shell"),
"{operator_line}"
);
assert!(
unknown_line.contains("UNKNOWN, which is not a yes"),
"{unknown_line}"
);
}
/// **The row this whole RPC exists for.** gunnar.rs, 300 GiB of raid0,
/// 70 GiB free, a tenant holding 10 GiB asking to be raised to 100 GiB —
/// 90 GiB of new bytes, against 55 GiB servable.
#[test]
fn ninety_more_gigabytes_on_the_sealed_appliance_is_a_no_that_names_the_reinstall() {
let mut q = question(gunnar_rs_today(), appliance());
q.namespace = "tenant".into();
q.held_bytes = 10 * GIB;
q.requested_bytes = 100 * GIB;
let a = answer(&q);
assert_eq!(
a.verdict,
CapacityVerdict::CapacityExhausted as i32,
"{}",
a.reason
);
assert_eq!(a.growth, SetGrowth::GrowthSealed as i32);
// The arithmetic, every term of it.
assert_eq!(a.free_bytes, 70 * GIB);
assert_eq!(a.floor_bytes, 15 * GIB, "5 % of 300 GiB");
assert_eq!(a.servable_bytes, 55 * GIB, "free minus the floor, not free");
assert_eq!(
a.additional_bytes,
90 * GIB,
"the raise needs the DELTA, not the cap"
);
assert_eq!(a.shortfall_bytes, 35 * GIB);
assert_eq!(a.data_set, "4\u{d7}75G raid0 xfs");
// And the sentence carries both halves.
assert!(a.reason.contains("NO."), "{}", a.reason);
assert!(a.reason.contains("UNUSED AND INVISIBLE"), "{}", a.reason);
}
/// The SAME numbers on a box with a shell answer NO too — but for a
/// different reason, and that difference is the whole point of the second
/// field. A caller reading only `verdict` would buy a disk here and be
/// right; on the appliance it would buy one and be robbed.
#[test]
fn the_same_shortfall_on_a_box_with_a_shell_does_not_say_a_disk_is_wasted() {
let mut sealed = question(gunnar_rs_today(), appliance());
sealed.requested_bytes = 90 * GIB;
let mut shelled = sealed.clone();
shelled.growth = developer_box();
let a = answer(&sealed);
let b = answer(&shelled);
assert_eq!(a.verdict, b.verdict, "the shortfall is identical");
assert_eq!(a.shortfall_bytes, b.shortfall_bytes);
assert_ne!(a.growth, b.growth, "the growth answer is NOT the verdict");
assert_ne!(
a.reason, b.reason,
"and it changes what the operator is told"
);
assert!(b.reason.contains("NO.") && !b.reason.contains("UNUSED AND INVISIBLE"));
}
/// A yes still reports the growth fact — "yes, and this is the last
/// capacity this box will ever have" is a different sale.
#[test]
fn a_request_that_fits_is_a_yes_that_still_says_the_set_is_sealed() {
let mut q = question(gunnar_rs_today(), appliance());
q.requested_bytes = 20 * GIB;
let a = answer(&q);
assert_eq!(a.verdict, CapacityVerdict::CapacityAvailable as i32);
assert_eq!(a.shortfall_bytes, 0);
assert!(a.reason.contains("YES"), "{}", a.reason);
assert_eq!(
a.growth,
SetGrowth::GrowthSealed as i32,
"the growth fact is reported on the YES arm too"
);
assert!(a.reason.contains("UNUSED AND INVISIBLE"), "{}", a.reason);
// The boundary is `servable`, not `free`: 60 GiB is under the 70 GiB
// free and over the 55 GiB servable, and must be refused.
q.requested_bytes = 60 * GIB;
assert_eq!(
answer(&q).verdict,
CapacityVerdict::CapacityExhausted as i32,
"a request under free but over free-minus-floor must be refused: the floor is not \
for sale"
);
q.requested_bytes = 55 * GIB;
assert_eq!(
answer(&q).verdict,
CapacityVerdict::CapacityAvailable as i32
);
}
#[test]
fn a_filesystem_that_cannot_be_measured_is_not_a_yes_and_reports_no_numbers() {
let mut q = question(gunnar_rs_today(), appliance());
q.room = Err("No such file or directory (os error 2)".into());
q.requested_bytes = 1;
let a = answer(&q);
assert_eq!(a.verdict, CapacityVerdict::CapacityUnmeasured as i32);
assert_ne!(
a.verdict,
CapacityVerdict::CapacityAvailable as i32,
"one byte would fit on any real disk; an unmeasured one must still not say yes"
);
assert_eq!(
(a.total_bytes, a.free_bytes, a.floor_bytes, a.servable_bytes),
(0, 0, 0, 0),
"nothing was measured, so nothing may be reported"
);
assert!(a.reason.contains("os error 2"), "{}", a.reason);
assert!(a.reason.contains("has not said yes"), "{}", a.reason);
}
/// **The box, as the live appliance would answer it**: 4 vCPU = 4000
/// millicores, 16 GiB / 12 GiB, load 0.42 ≈ 420 millicores — and the
/// reason line names each source, calls the load an approximation, and
/// is carried on the UNMEASURED-disk arm too.
#[test]
fn the_appliance_box_is_reported_in_millicores_and_bytes_with_its_sources_named() {
let q = question(gunnar_rs_today(), appliance());
let a = answer(&q);
assert_eq!(a.cpu_millicores_total, Some(4000));
assert_eq!(a.ram_bytes_total, Some(16 * GIB));
assert_eq!(a.ram_bytes_available, Some(12 * GIB));
assert_eq!(a.cpu_millicores_busy, Some(420), "0.42 × 1000, rounded");
for word in [
"affinity mask",
"/proc/meminfo",
"/proc/loadavg",
"APPROXIMATION",
] {
assert!(
a.machine_reason.contains(word),
"{word}: {}",
a.machine_reason
);
}
assert!(
!a.machine_reason.contains("clamped"),
"no cgroup clamped the appliance: {}",
a.machine_reason
);
// The same box on the disk-unmeasured arm: the fields do not depend
// on whether statvfs worked, so one constructor serves both.
let mut unmeasured = q.clone();
unmeasured.room = Err("No such file or directory (os error 2)".into());
let u = answer(&unmeasured);
assert_eq!(
(
u.cpu_millicores_total,
u.ram_bytes_total,
u.ram_bytes_available,
u.cpu_millicores_busy
),
(
a.cpu_millicores_total,
a.ram_bytes_total,
a.ram_bytes_available,
a.cpu_millicores_busy
)
);
assert_eq!(u.machine_reason, a.machine_reason);
}
/// **Absent, never zero.** A box whose `/proc/meminfo` cannot be read
/// reports NO RAM fields — `None` on the wire, not `Some(0)` — and the
/// reason names the OS error; the cores it DID measure are still there.
#[test]
fn an_unreadable_meminfo_leaves_the_ram_fields_absent_and_names_why() {
let mut q = question(gunnar_rs_today(), appliance());
q.machine.memory =
Err("/proc/meminfo could not be read (No such file or directory (os error 2))".into());
q.machine.load1 =
Err("/proc/loadavg could not be read (Permission denied (os error 13))".into());
let a = answer(&q);
assert_eq!(a.ram_bytes_total, None, "absent, not 0: {a:?}");
assert_eq!(a.ram_bytes_available, None);
assert_eq!(a.cpu_millicores_busy, None);
assert_ne!(
a.ram_bytes_total,
Some(0),
"a zero here would be a box with no RAM"
);
assert_eq!(
a.cpu_millicores_total,
Some(4000),
"the cores were measured and stay"
);
assert!(
a.machine_reason.contains("RAM UNMEASURED"),
"{}",
a.machine_reason
);
assert!(
a.machine_reason.contains("os error 2"),
"{}",
a.machine_reason
);
assert!(
a.machine_reason.contains("load UNMEASURED"),
"{}",
a.machine_reason
);
assert!(
a.machine_reason.contains("os error 13"),
"{}",
a.machine_reason
);
// And the whole box unmeasured is a reason line, not an empty one.
q.machine.cpus = Err("the affinity mask could not be read (os error 22)".into());
let none = answer(&q);
assert_eq!(
(
none.cpu_millicores_total,
none.ram_bytes_total,
none.ram_bytes_available,
none.cpu_millicores_busy
),
(None, None, None, None)
);
assert!(
none.machine_reason.contains("cores UNMEASURED"),
"{}",
none.machine_reason
);
// The disk answer is untouched by any of it.
assert_eq!(none.verdict, a.verdict);
assert_eq!(none.reason, a.reason);
}
/// A cgroup limit smaller than the host is the RAM gunnar may use, and
/// the reason says the number is the container's, not the host's.
#[test]
fn a_cgroup_memory_clamp_is_reported_as_the_ram_and_named_in_the_reason() {
let mut q = question(gunnar_rs_today(), appliance());
q.machine.memory = Ok((8 * GIB, 6 * GIB));
q.machine.cgroup_memory_max = Some(8 * GIB);
let a = answer(&q);
assert_eq!(a.ram_bytes_total, Some(8 * GIB));
assert!(
a.machine_reason
.contains("clamped to the cgroup memory.max of 8.0 GiB"),
"{}",
a.machine_reason
);
}
/// **The thirteen disk fields are independent of the box.** The same
/// question with a measured box and with an unmeasured one yields
/// byte-identical disk fields — and the wire proves it in
/// `tests/capacity.rs`, where the bytes are decoded by an OLDER client.
#[test]
fn the_disk_answer_does_not_change_with_the_box() {
let q = question(gunnar_rs_today(), appliance());
let mut blind = q.clone();
blind.machine = MachineFacts {
cpus: Err("x".into()),
memory: Err("y".into()),
cgroup_memory_max: None,
load1: Err("z".into()),
disk: None,
};
let (a, b) = (answer(&q), answer(&blind));
assert_ne!(a.machine_reason, b.machine_reason, "the fixtures differ");
assert_ne!(a.ram_bytes_total, b.ram_bytes_total);
// Mask the BOX's own fields and the start-up disk TEST's — both are
// facts about the machine rather than about the store's free space,
// and this test is about the `statvfs` answer alone. The disk test
// joined the mask 2026-09-21 when it reached the wire: a box that has
// measured its disk and one that has not must still give the same
// capacity verdict, which is exactly what the mask asserts.
let disk = |r: &CapacityResponse| {
CapacityResponse {
cpu_millicores_total: None,
ram_bytes_total: None,
ram_bytes_available: None,
cpu_millicores_busy: None,
machine_reason: String::new(),
disk_test: 0,
disk_seq_write_bps: None,
disk_seq_read_bps: None,
disk_sync_milliops_per_s: None,
disk_read_how: String::new(),
disk_reason: String::new(),
// The run's raw bytes and clocks (2026-09-28) are the same
// disk test's, so they are masked with it.
disk_test_bytes: None,
disk_write_us: None,
disk_read_us: None,
disk_sync_ops: None,
disk_sync_us: None,
disk_test_us: None,
..r.clone()
}
};
assert_eq!(disk(&a), disk(&b));
// …and the two fixtures really did differ on the disk test, or the
// mask above would be asserting nothing.
assert_ne!(a.disk_test, b.disk_test);
assert_ne!(a.disk_reason, b.disk_reason);
}
}
/// **The disk on the wire** — added 2026-09-21 because the twin election has
/// to compare two boxes' disks and `Admin.Capacity` did not carry them.
///
/// The property under test is one sentence: *a box that could not measure
/// must be distinguishable from a box that measured badly, and neither may
/// look like a fast disk.* Every arm below drives one way of not measuring
/// and asserts that it is named, and that no speed field carries a zero.
#[cfg(test)]
mod disk_on_the_wire {
use super::*;
use crate::disktest::{Outcome, ReadHow, Report};
use gunnar_proto::DiskTest;
use std::time::Duration;
fn done(seq_read_bps: f64) -> Outcome {
Outcome::Done(Report {
seq_write_bps: 812e6,
seq_read_bps,
read_how: ReadHow::Direct,
sync_ops_per_s: 3100.4,
elapsed: Duration::from_millis(1600),
file_bytes: 256 << 20,
write_elapsed: Duration::from_millis(330),
read_elapsed: Duration::from_millis(141),
sync_ops: 1550,
sync_elapsed: Duration::from_millis(500),
})
}
/// A real run puts three numbers on the wire, in the wire's units.
#[test]
fn a_measured_disk_carries_its_three_numbers_and_says_how_it_read() {
let a = disk_fields(&Some(done(1.9e9)));
assert_eq!(a.disk_test, DiskTest::DisktestMeasured as i32);
assert_eq!(a.disk_seq_write_bps, Some(812_000_000));
assert_eq!(a.disk_seq_read_bps, Some(1_900_000_000));
// ops/s × 1000, so the wire stays integral the way millicores do.
assert_eq!(a.disk_sync_milliops_per_s, Some(3_100_400));
assert_eq!(a.disk_read_how, "direct");
assert!(a.disk_reason.contains("812 MB/s"), "{}", a.disk_reason);
assert!(a.disk_reason.contains("1.9 GB/s"), "{}", a.disk_reason);
// …and what they are made of: the bytes and each leg's clock.
assert_eq!(a.disk_test_bytes, Some(256 << 20));
assert_eq!(
(a.disk_write_us, a.disk_read_us),
(Some(330_000), Some(141_000))
);
assert_eq!(
(a.disk_sync_ops, a.disk_sync_us),
(Some(1550), Some(500_000))
);
assert_eq!(a.disk_test_us, Some(1_600_000));
}
/// **The cached read is the subtle one.** Over 5 GB/s the page cache
/// answered and the device was never touched — so the read is ABSENT,
/// not fast; and the write and fsync legs of the same run still stand,
/// because they were really measured.
#[test]
fn a_cached_read_is_absent_from_the_wire_and_is_never_reported_as_a_fast_disk() {
let a = disk_fields(&Some(done(7.2e9)));
assert_eq!(
a.disk_test,
DiskTest::DisktestMeasured as i32,
"the run DID measure"
);
assert_eq!(a.disk_seq_read_bps, None, "not 7.2 GB/s and not 0 — absent");
assert_eq!(
a.disk_seq_write_bps,
Some(812_000_000),
"the other legs stand"
);
assert_eq!(a.disk_sync_milliops_per_s, Some(3_100_400));
assert!(a.disk_reason.contains("CACHED"), "{}", a.disk_reason);
assert!(
a.disk_reason.contains("not a fast disk"),
"the reason says what the absence MEANS: {}",
a.disk_reason
);
}
/// The four silences are four different facts, each named, and none of
/// them puts a number anywhere.
#[test]
fn a_disk_that_was_not_measured_is_named_and_carries_no_number_at_all() {
let cases = [
(None, DiskTest::DisktestNotRun, "gate"),
(
Some(Outcome::Skipped {
free: 800 << 20,
need: 1 << 30,
}),
DiskTest::DisktestSkipped,
"SKIPPED",
),
(
Some(Outcome::Failed("write: No space left on device".into())),
DiskTest::DisktestFailed,
"FAILED",
),
];
for (outcome, want, word) in cases {
let a = disk_fields(&outcome);
assert_eq!(a.disk_test, want as i32, "{outcome:?}");
assert_eq!(a.disk_seq_write_bps, None, "{outcome:?}");
assert_eq!(a.disk_seq_read_bps, None, "{outcome:?}");
assert_eq!(a.disk_sync_milliops_per_s, None, "{outcome:?}");
assert!(a.disk_read_how.is_empty(), "{outcome:?}");
assert_eq!(
(
a.disk_test_bytes,
a.disk_write_us,
a.disk_read_us,
a.disk_sync_ops,
a.disk_sync_us,
a.disk_test_us
),
(None, None, None, None, None, None),
"no run, no bytes and no times: {outcome:?}"
);
assert!(!a.disk_reason.is_empty(), "never silent: {outcome:?}");
assert!(a.disk_reason.contains(word), "{}", a.disk_reason);
}
// Skipped names BOTH figures, so an operator can act on it.
let s = disk_fields(&Some(Outcome::Skipped {
free: 800 << 20,
need: 1 << 30,
}));
assert!(
s.disk_reason.contains("800") && s.disk_reason.contains("1.0"),
"{}",
s.disk_reason
);
// And the three silences do not say the same thing as each other.
let reasons: std::collections::BTreeSet<String> = [
disk_fields(&None).disk_reason,
disk_fields(&Some(Outcome::Skipped { free: 0, need: 1 })).disk_reason,
disk_fields(&Some(Outcome::Failed("x".into()))).disk_reason,
]
.into_iter()
.collect();
assert_eq!(reasons.len(), 3);
}
/// The disk rides on the SAME seam as the box's cores and RAM, so it can
/// never be filled on one arm of `answer` and forgotten on the other.
#[test]
fn the_box_fields_and_the_disk_fields_come_out_of_one_call() {
let m = MachineFacts {
cpus: Ok(4),
memory: Ok((16 << 30, 12 << 30)),
cgroup_memory_max: None,
load1: Ok(0.42),
disk: Some(done(1.9e9)),
};
let a = machine_fields(&m);
assert_eq!(a.cpu_millicores_total, Some(4000));
assert_eq!(a.disk_seq_write_bps, Some(812_000_000));
assert!(!a.machine_reason.is_empty() && !a.disk_reason.is_empty());
// A box with no disk reading still reports its cores.
let blind = machine_fields(&MachineFacts { disk: None, ..m });
assert_eq!(blind.cpu_millicores_total, Some(4000));
assert_eq!(blind.disk_test, DiskTest::DisktestNotRun as i32);
}
/// The latch is a LOOKUP: reading it never starts a disk test, and an
/// outcome recorded is the outcome read back.
#[test]
fn the_latch_hands_back_what_was_recorded_and_runs_nothing() {
let before = crate::disktest::last();
crate::disktest::record(&done(1.9e9));
assert_eq!(crate::disktest::last(), Some(done(1.9e9)));
crate::disktest::record(&Outcome::Failed("boom".into()));
assert_eq!(
crate::disktest::last(),
Some(Outcome::Failed("boom".into()))
);
// Put back whatever another test in this binary may have left.
if let Some(b) = before {
crate::disktest::record(&b);
}
}
}
#[cfg(test)]
mod disk_tests {
use super::*;
/// A recorded snapshot reaches the wire field by field, and an unmeasured
/// mount stays unmeasured: `measured = false`, the reason, and zeroes that
/// the flag says are not readings.
#[test]
fn a_snapshot_reaches_the_wire_and_an_unmeasured_mount_is_not_a_number() {
let mount = |src: &str, at: &str| disk_snapshot::Mount {
source: src.to_owned(),
mount_point: PathBuf::from(at),
fstype: "xfs".to_owned(),
read_only: false,
};
let snap = disk_snapshot::Snapshot {
disks: vec![
disk_snapshot::DiskEntry {
mount: mount("/dev/md0", "/data"),
labels: vec!["data".to_owned()],
paths: vec![PathBuf::from("/data")],
figures: disk_snapshot::Figures::from_counts(1000, 400, 350, 4096, 10, 5),
floor_bytes: 4096 * 50,
servable_bytes: 4096 * 300,
},
disk_snapshot::DiskEntry {
mount: mount("/dev/vda2", "/"),
labels: vec!["system".to_owned()],
paths: vec![PathBuf::from("/")],
figures: Err("statvfs(/): EIO".to_owned()),
floor_bytes: 0,
servable_bytes: 0,
},
],
unresolved: vec![],
block_devices: vec![disk_snapshot::BlockDevice {
name: "md0".to_owned(),
size_bytes: 4096 * 1000,
members: vec!["vdb1".to_owned(), "vdc1".to_owned()],
raid_level: "raid0".to_owned(),
..Default::default()
}],
problems: vec![],
};
let wire = disks_to_wire(snap, 42);
assert_eq!(wire.measured_at_unix_ms, 42);
let data = &wire.mounts[0];
assert!(data.measured && data.labels == ["data"]);
assert_eq!(
(
data.total_bytes,
data.free_bytes,
data.available_bytes,
data.used_bytes,
data.reserved_bytes
),
(4096 * 1000, 4096 * 400, 4096 * 350, 4096 * 600, 4096 * 50)
);
assert_eq!(
(data.floor_bytes, data.servable_bytes),
(4096 * 50, 4096 * 300)
);
let system = &wire.mounts[1];
assert!(!system.measured && system.error.contains("EIO") && system.total_bytes == 0);
assert_eq!(wire.devices[0].members, ["vdb1", "vdc1"]);
assert_eq!(wire.devices[0].raid_level, "raid0");
}
/// **The /serialdat breakdown sums**: each known path measured by its
/// blocks, the audit trail counted once though it sits inside the log
/// directory, a missing path 0, and `other` = the mount's used bytes minus
/// what the known paths hold — so the rows add up to the mount's use.
#[test]
fn the_serialdat_breakdown_sums_to_the_mounts_use() {
use std::io::Write as _;
let dir = tempfile::tempdir().expect("a temp mount");
let root = dir.path();
let write = |rel: &str, n: usize| {
let p = root.join(rel);
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
let mut f = std::fs::File::create(&p).unwrap();
f.write_all(&vec![7u8; n]).unwrap();
f.sync_all().unwrap();
};
write("meta/principals.json", 10_000);
write("log/audit-chain.jsonl", 70_000);
write("log/console.log", 30_000);
write("crash/last.txt", 5_000);
let known = vec![
("metadata copy", root.join("meta")),
("crash record", root.join("crash")),
("audit trail", root.join("log/audit-chain.jsonl")),
("logs", root.join("log")),
("grow journal", root.join("grow-journal")),
];
let used = 10 << 20;
let rows = serialdat_breakdown(&known, Some(used));
let get = |label: &str| {
rows.iter()
.find(|r| r.label == label)
.unwrap_or_else(|| panic!("{label}: {rows:?}"))
};
assert!(rows.iter().all(|r| r.measured), "{rows:?}");
assert!(get("audit trail").bytes >= 70_000, "{rows:?}");
let logs = get("logs").bytes;
assert!(
(30_000..70_000).contains(&logs),
"the audit trail was counted inside the logs too: {rows:?}"
);
assert_eq!(
get("grow journal").bytes,
0,
"nothing there is 0, not an error"
);
let sum: u64 = rows.iter().map(|r| r.bytes).sum();
assert_eq!(
sum, used,
"the rows must add up to the mount's used bytes: {rows:?}"
);
// An unmeasured mount: no `other`, nothing invented.
assert!(serialdat_breakdown(&known, None)
.iter()
.all(|r| r.label != "other"));
}
/// The live road on this machine: a data root in a temp dir is placed on
/// the mount it sits on and measured, and `/` is labelled `system`.
#[test]
fn this_machines_disks_carry_gunnars_labels() {
let dir = tempfile::tempdir().expect("a temp dir");
let wire = measure_disks(dir.path(), None, gunnar_ssh::DiskFloor::default());
assert!(wire.problems.is_empty(), "{:?}", wire.problems);
let data = wire
.mounts
.iter()
.find(|m| m.labels.iter().any(|l| l == "data"))
.expect("the data root is on some mount");
assert!(data.measured && data.total_bytes > 0, "{data:?}");
assert!(
wire.mounts
.iter()
.any(|m| m.labels.iter().any(|l| l == "system")),
"{:?}",
wire.mounts
);
}
}