crates/skidbladnir-systemd/src/running.rs
//! **Is the RUNNING process actually the binary on disk?**
//!
//! # The false green this exists to kill
//!
//! `<binary> --version` reads the binary **on disk**. It says nothing whatever
//! about the process that is currently serving. Those two drift apart constantly:
//! you rebuild, the new bytes land at the same path, and the old process keeps
//! running the *deleted* inode it was started from. The version line then reports
//! the NEW hash while the OLD code answers every request.
//!
//! Observed on `oden` (2026-07-31), and it is the reason this module exists:
//!
//! ```text
//! $ korp-ui --version → korp v0.1.0 (fda8c87) ← the NEW build
//! $ readlink /proc/<pid>/exe → …/korp-ui (deleted) ← the OLD inode
//! process started 09:56:42, binary written 11:01:14 ← an hour stale
//! ```
//!
//! Every "verified live, `--version` says so" claim made that way is unsound.
//! `restart != rebuild` is already a standing LAW; this is the *check* that makes
//! the LAW enforceable instead of a thing people are asked to remember.
//!
//! # The two questions, both answered from `/proc` (pure `std::fs`, no shell)
//!
//! 1. **Is the running inode still the file on disk?** Linux appends the literal
//! marker `" (deleted)"` to `/proc/<pid>/exe` when the executable it was started
//! from has been unlinked. A deleted exe is conclusive: the process CANNOT be
//! running the current bytes.
//! 2. **Did the process start after the binary was written?** If the binary's
//! mtime is newer than the process start, the process predates the build — the
//! classic "you rebuilt but forgot to restart".
//!
//! Start time comes from `/proc/<pid>/stat` field 22 (`starttime`, in clock
//! ticks since boot) added to `btime` from `/proc/stat`. The tempting shortcut —
//! the mtime of the `/proc/<pid>` **directory** — is WRONG and was measured to
//! be wrong here: on this kernel that mtime tracks the moment you stat it, so a
//! freshly-written binary and a long-running process compare equal to the
//! nanosecond and every stale process reads as current. The shortcut would have
//! produced exactly the false green this module exists to prevent.
//!
//! That sum is **coarser than it looks**, and getting this wrong produced a
//! reproducible false STALE on this very test binary. `starttime` carries 10 ms
//! resolution and must be added as milliseconds — dividing the ticks by `USER_HZ`
//! in integer arithmetic throws away up to 0.99 s and always rounds the start
//! time DOWN, i.e. always toward "stale". `btime` is then itself only a whole
//! second (the kernel derives it as now − uptime), worth another ±1 s. So the
//! computed start can sit up to ~1 s before the true one even with the ticks
//! handled exactly, which is why the comparison carries an explicit
//! [`START_TIME_SLACK`](crate::running::START_TIME_SLACK) instead of pretending
//! to second precision it does not
//! have. The slack costs nothing real: it can only miss a rebuild that landed
//! within two seconds of the process starting, while the drift this module
//! exists to catch is minutes to hours.
//!
//! Both are best-effort by design: anything we cannot determine is
//! [`Freshness::Unknown`](crate::running::Freshness::Unknown), never a silent
//! "fine". An unanswerable question must not
//! read as a pass — that is the very defect this module is about.
//!
//! # The third question, and the only CONCLUSIVE one: the content hash
//!
//! The two questions above are *proxies*. A timestamp is not an identity, and the
//! `(deleted)` marker only fires for the replace-by-unlink shape. Both can be
//! evaded by an ordinary install:
//!
//! - a unit whose `ExecStart=` was pointed at a **different path** runs a binary
//! that is neither deleted nor older than the file the installer just wrote;
//! - `StartUnit` against an already-`active` unit is a **no-op**, so the old
//! process survives a `daemon-reload` with a start time that predates nothing.
//!
//! That is the defect this module was extended for (gunnar, 2026-08-06: a wrong
//! `--exec` produced three greens plus exit 0 with the previous build still
//! serving). The answer that cannot be argued with is to **hash the bytes the
//! kernel is actually executing**: `/proc/<pid>/exe` is not merely a symlink, it is
//! openable and readable — including when the on-disk path has been unlinked, in
//! which case it is the *only* way to read those bytes back. Comparing
//! `sha256(/proc/<pid>/exe)` against `sha256(<the artifact just installed>)` asks
//! the question directly, with no clock, no inode and no marker in between.
//!
//! So [`running_binary_is_current`] leads with the hash and keeps the timestamp
//! reasoning only as the fallback for when a hash cannot be taken (the on-disk file
//! is gone, `/proc` is unreadable, not Linux). A mismatch is
//! [`Freshness::ExecMismatch`] and names both digests, because "it does not match"
//! without the two numbers is not evidence anyone can check.
use std::path::{Path, PathBuf};
use std::time::SystemTime;
/// The marker Linux appends to `/proc/<pid>/exe` once the executable is unlinked.
const DELETED_SUFFIX: &str = " (deleted)";
/// How much earlier than the truth `process_start_time` may land, and therefore
/// how much newer than the process start a binary's mtime may be before we are
/// willing to call the process stale.
///
/// `btime` in `/proc/stat` is a whole second, so the reconstructed start time
/// inherits up to a second of error even once the `starttime` ticks are added at
/// their full 10 ms resolution. Erring in the other direction is what matters:
/// under-reporting the start time accuses a healthy process, and a freshly built
/// binary running its own tests hit exactly that. Two seconds covers the granularity
/// with room to spare while remaining far below any real rebuild-without-restart
/// window.
pub const START_TIME_SLACK: std::time::Duration = std::time::Duration::from_secs(2);
/// Whether a running process is actually running the binary that is on disk now.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Freshness {
/// The process runs the on-disk binary, and started at or after it was written.
Current,
/// `/proc/<pid>/exe` is marked `(deleted)` — the process runs an unlinked inode.
/// Conclusive staleness: a restart is required, no timestamp reasoning needed.
DeletedBinary {
/// The exe link as reported, marker included.
exe: PathBuf,
},
/// The process started BEFORE the binary was last written — rebuilt, not
/// restarted.
OlderThanBinary {
/// Process start (`/proc/<pid>/stat` field 22 + `/proc/stat` `btime`),
/// second-granular.
started: SystemTime,
/// Last write to the binary on disk.
binary_written: SystemTime,
},
/// **The conclusive red.** The bytes the process is executing
/// (`sha256(/proc/<pid>/exe)`) are NOT the bytes at the expected path. No
/// timestamp, marker or `ActiveState` can talk this one down: a different digest
/// is a different program.
///
/// This is the variant that catches the shape the other two miss — a unit that
/// was `active` when the installer ran, so `StartUnit` was a no-op and the
/// previous build simply kept serving.
ExecMismatch {
/// `sha256` of `/proc/<pid>/exe` — what is REALLY executing.
running_sha256: String,
/// `sha256` of the binary at the expected path — what SHOULD be executing.
expected_sha256: String,
/// The expected path, as resolved by the caller.
binary: PathBuf,
},
/// **The manager has not finished handing the process over — a stale HANDLE,
/// not a verdict about any binary.**
///
/// Since systemd 254 the manager does not `execve` a service binary itself: it
/// forks **`systemd-executor`**, and THAT process execs the service. For the
/// window in between, `MainPID` names a live process whose `/proc/<pid>/exe` is
/// the helper — readable, hashable, and not the service's artifact. So the
/// conclusive hash compares systemd's helper against the product's binary and
/// answers [`ExecMismatch`](Freshness::ExecMismatch) on a service that is
/// perfectly healthy.
///
/// MEASURED on the gunnar.rs front vm (Ubuntu 24.04, systemd 255) 2026-09-04:
/// `monetize-server --install` FAILED with *"a DIFFERENT program is serving"*
/// naming sha256 `b8424efa…`, twice, deterministically — while
/// `sha256sum /proc/<pid>/exe`, `cat /proc/<pid>/exe | sha256sum` and the file
/// itself all agreed on `acc2ccfb…`. Scanning every live process for the
/// reported digest found it: `/usr/lib/systemd/systemd-executor`, the `(sd-pam)`
/// process. The unit was correct, enabled and serving the right bytes; only the
/// verdict was wrong.
///
/// This is the same lesson as [`PidGone`](Freshness::PidGone) arriving through a
/// third door — the pid is real but not yet the service's — so it is answered
/// the same way: UNDETERMINED, re-read `MainPID`, ask again. It is deliberately
/// NOT [`is_proven_stale`](Freshness::is_proven_stale): an install that is
/// correct must not be failed.
StillHandingOver {
/// The pid the manager reported mid-handover.
pid: u32,
/// The helper `/proc/<pid>/exe` resolved to.
helper: PathBuf,
},
/// **The pid is GONE — a stale HANDLE, not a verdict about any binary.**
///
/// Split out of [`Unknown`](Freshness::Unknown) because the two are opposite kinds
/// of news and only one of them is news at all. "I read a live process's exe and
/// it does not match" is a finding. "The pid I was handed no longer exists" is a
/// statement about the QUESTION, not the answer: the caller asked about the wrong
/// process and should ask the manager again.
///
/// MEASURED, 2026-08-25 on t14s: `RestartUnit` over D-Bus returns the moment the
/// job is ENQUEUED, and for as long as the outgoing process takes to drain the
/// manager keeps reporting the OLD `MainPID` — non-zero, alive, and doomed. With a
/// service that takes 3s to stop, `MainPID` stayed on the pre-restart pid for 2.8s
/// after the call returned. A verifier that reads it once, on the
/// update-an-already-running-service path, is therefore holding a pid that is
/// about to die by construction. That is not an unreadable file; it is the wrong
/// question, and [`verify_current_binary`] answers it by asking again.
PidGone {
/// The pid that no longer exists.
pid: u32,
},
/// Could not be determined (no `/proc`, permissions, non-Linux).
/// **Never** treat this as current.
///
/// A pid that has simply EXITED is [`PidGone`](Freshness::PidGone), not this — see
/// there for why the split matters.
Unknown(String),
}
impl Freshness {
/// **Is this a PROVEN wrong-binary finding — the only kind that may fail an
/// install?**
///
/// Named, and asked as a question rather than left to a match arm's position,
/// because getting it wrong has happened twice in this file's history and both
/// times the same way: an UNDETERMINED answer folded into the failing arm and
/// refused a correct install. The first was `Unknown` (fixed, and the fix is
/// commented at the call site); the second would have been
/// [`PidGone`](Freshness::PidGone), which arrives on the ordinary
/// update-an-already-running-service path. The THIRD did happen, on
/// 2026-09-04: [`StillHandingOver`](Freshness::StillHandingOver), where the pid
/// is alive and its exe hashes cleanly — to systemd's own `systemd-executor`.
///
/// `true` for the three findings that are evidence about a binary:
/// [`ExecMismatch`](Freshness::ExecMismatch),
/// [`DeletedBinary`](Freshness::DeletedBinary),
/// [`OlderThanBinary`](Freshness::OlderThanBinary). `false` for `Current` and for
/// both undetermined shapes — an unanswered question is not a failed one.
pub fn is_proven_stale(&self) -> bool {
matches!(
self,
Freshness::ExecMismatch { .. }
| Freshness::DeletedBinary { .. }
| Freshness::OlderThanBinary { .. }
)
}
/// `true` only for [`Freshness::Current`]. `Unknown` is NOT current — an
/// undetermined answer must not read as a pass.
pub fn is_current(&self) -> bool {
matches!(self, Freshness::Current)
}
/// A one-line operator-facing explanation, always naming the reason.
pub fn explain(&self) -> String {
match self {
Freshness::Current => "running the current on-disk binary".to_string(),
Freshness::DeletedBinary { exe } => format!(
"STALE: the running process holds a DELETED binary ({}) — it was replaced on \
disk and never restarted; `--version` reads the new file and is LYING about \
what is serving",
exe.display()
),
Freshness::OlderThanBinary {
started,
binary_written,
} => format!(
"STALE: the process started {:?} but the binary was written {:?} — rebuilt \
without a restart (restart != rebuild)",
started, binary_written
),
Freshness::ExecMismatch {
running_sha256,
expected_sha256,
binary,
} => format!(
"STALE: the running process is executing sha256 {running_sha256} but {} is \
sha256 {expected_sha256} — a DIFFERENT program is serving; systemd reporting \
`active` is not evidence, and `StartUnit` against an already-active unit is a \
no-op that leaves exactly this state",
binary.display()
),
Freshness::StillHandingOver { pid, helper } => format!(
"UNDETERMINED (not a pass): pid {pid} is still executing the manager's own \
spawn helper ({}) — systemd forks `systemd-executor` and it execs the \
service afterwards, so this pid is not the service's binary YET. A stale \
handle, not a finding: re-read MainPID and ask again",
helper.display()
),
Freshness::PidGone { pid } => format!(
"UNDETERMINED (not a pass): pid {pid} no longer exists — the process was \
already gone when it was read. This is a STALE HANDLE, not a finding about \
any binary: re-read MainPID from the manager and ask again"
),
Freshness::Unknown(why) => {
format!("UNDETERMINED (not a pass): {why}")
}
}
}
}
/// **`sha256` of the bytes `pid` is actually executing**, read back through
/// `/proc/<pid>/exe`.
///
/// That entry is openable, not merely readable as a link, and the kernel keeps the
/// inode alive for as long as the process does — so this returns the true running
/// image **even when the on-disk path has been unlinked**, which is precisely the
/// case where nothing else can.
///
/// Reuses the constellation's one content-hash primitive (`nornir-hash`); it is a
/// streaming read, so a large binary is not held in memory.
pub fn running_exe_sha256(pid: u32) -> Result<String, String> {
nornir_hash::sha256_file(format!("/proc/{pid}/exe"))
.map_err(|e| format!("hash /proc/{pid}/exe: {e}"))
}
/// `sha256` of the binary at `path` — the artifact side of the comparison.
pub fn binary_sha256(path: &Path) -> Result<String, String> {
nornir_hash::sha256_file(path).map_err(|e| format!("hash {}: {e}", path.display()))
}
/// How long to let a just-started process finish `execve` before believing that its
/// `/proc/<pid>/exe` is genuinely unreadable. An exec completes in milliseconds; this
/// is slack, not a wait.
pub const EXEC_SETTLE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(3);
/// **Block until `pid`'s running image can actually be hashed**, or `timeout`.
///
/// A process that systemd has just reported as `MainPID` may still be between `fork`
/// and `execve`, and in that window opening `/proc/<pid>/exe` fails with **EACCES** —
/// not because we lack permission in any lasting sense, but because there is not yet
/// a stable image to read. MEASURED on oden 2026-08-06: without this wait the
/// post-install verification reported
/// `UNDETERMINED … Permission denied` on a perfectly healthy unit, every time.
///
/// That matters more than a flaky test. A verifier that answers "could not tell" on
/// the normal path is a verifier everyone learns to ignore, which is the same
/// outcome as not having one — so the settle wait is part of the check, not part of
/// its test harness.
///
/// Returns the digest once it can be taken, or the last error.
pub fn settle_running_exe_sha256(pid: u32, timeout: std::time::Duration) -> Result<String, String> {
let deadline = std::time::Instant::now() + timeout;
loop {
match running_exe_sha256(pid) {
Ok(sha) => return Ok(sha),
Err(e) if std::time::Instant::now() >= deadline => return Err(e),
Err(_) => std::thread::sleep(std::time::Duration::from_millis(25)),
}
}
}
/// **Does this pid name a process that exists right now?**
///
/// `/proc/<pid>` is created by the kernel for every live task and disappears when the
/// process is reaped, so its presence is the question asked directly — no signal, no
/// permission needed, and it works for a process owned by another account.
///
/// It exists so [`verify_current_binary`] can tell a stale handle from a process that
/// is merely slow to exec WITHOUT paying the settle timeout to find out.
pub fn pid_exists(pid: u32) -> bool {
std::path::Path::new(&format!("/proc/{pid}")).exists()
}
/// The verdict [`verify_current_binary`] reached, and the pid it reached it ABOUT.
///
/// The pid is carried because the caller's message must name the process it actually
/// judged, not the one it first asked for. Reporting `pid 363212` after having
/// re-read and judged `pid 1150227` would be a message an operator cannot check by
/// hand — and checking it by hand is the entire point of printing it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RunningVerdict {
/// The pid the verdict is about; `None` if the manager never reported one.
pub pid: Option<u32>,
/// The verdict itself.
pub freshness: Freshness,
/// How many times `MainPID` had to be re-read before a live process was found.
/// `0` on the ordinary path; non-zero means a restart was still settling, which
/// is worth saying out loud rather than hiding a retry.
pub rereads: u32,
}
/// **Ask "is the running process the binary on disk?" against a pid that may still be
/// MOVING** — re-reading `MainPID` for as long as the answer is a stale handle.
///
/// # The defect this exists for
///
/// `install_systemd` verifies the running binary after `start_or_restart`. On the
/// update-an-already-running-service path — the most common install there is — that
/// restart goes out as a D-Bus `RestartUnit` in `replace` mode, which returns when the
/// job is **enqueued**, not when it is done. The manager then keeps reporting the
/// OUTGOING `MainPID`: non-zero, alive, and doomed.
///
/// MEASURED on t14s 2026-08-25 with a service that drains for 3s, polling every 400ms
/// after the call returned:
///
/// ```text
/// poll 1..7 : MainPID=1176240 <-- PRE-RESTART PID
/// poll 8..14: MainPID=1176658 (new)
/// ```
///
/// A single read lands in the first band every time, and then one of two things
/// happens, both wrong:
///
/// * the old process dies before it is hashed ⇒ `read /proc/<pid>/exe: No such file
/// or directory`, reported as UNDETERMINED. This is what the live install printed.
/// * the old process is still alive when it is hashed ⇒ it is running the OLD bytes,
/// so the hash MISMATCHES and the install is **FAILED** with "the process serving is
/// NOT the binary this install pointed at". A false red, on a correct install.
///
/// The second is the more dangerous and had never been seen, because it needs the
/// outgoing process to outlive the hash — which is exactly what a real server with a
/// graceful shutdown does, and what a `sleep` in a test never does.
///
/// # What it does
///
/// Takes a CLOSURE that re-reads `MainPID` rather than a pid, because a pid is a
/// snapshot of something that is moving. Each round: read the pid, let its image
/// settle (a just-exec'd process is briefly unreadable), judge it. A
/// [`Freshness::PidGone`] means the handle was stale, so it asks the manager again
/// until the deadline. Anything else is a real answer and is returned at once —
/// including a mismatch, which must NOT be retried away.
///
/// The closure is the seam that makes this testable with no systemd: a test feeds it a
/// dead pid twice and a live one third.
pub fn verify_current_binary(
mut main_pid: impl FnMut() -> Option<u32>,
binary: &Path,
timeout: std::time::Duration,
) -> RunningVerdict {
let deadline = std::time::Instant::now() + timeout;
let mut rereads = 0;
let mut last_pid = None;
let mut last_undetermined: Option<Freshness> = None;
loop {
let pid = main_pid();
last_pid = pid.or(last_pid);
if let Some(pid) = pid {
// ORDER MATTERS, and getting it wrong cost a self-inflicted red here:
// settling a pid that is ALREADY GONE blocks for the whole
// `EXEC_SETTLE_TIMEOUT` before failing, so two stale rounds burn 6s of a
// 10s budget and the retry never reaches the live process. The settle wait
// exists for the fork/execve window of a process that EXISTS; a process
// that does not exist is answered immediately and re-read.
if !pid_exists(pid) {
if std::time::Instant::now() >= deadline {
return RunningVerdict {
pid: Some(pid),
freshness: Freshness::PidGone { pid },
rereads,
};
}
rereads += 1;
std::thread::sleep(std::time::Duration::from_millis(100));
continue;
}
// A pid the manager has just published may still be between fork and
// execve, where `/proc/<pid>/exe` is briefly unreadable. Settle first so a
// healthy new process is never mistaken for a stale handle — but never
// past the caller's own deadline.
let settle = EXEC_SETTLE_TIMEOUT
.min(deadline.saturating_duration_since(std::time::Instant::now()));
let _ = settle_running_exe_sha256(pid, settle);
let freshness = running_binary_is_current(pid, binary);
// ── A PROXY VERDICT MUST BE CONFIRMED ON A STABLE PID ────────────────
//
// MEASURED on oden 2026-08-27: `--install` cried STALE while **all three
// sha256 matched** — the source, the installed file, and the running image.
// A false negative, and it races the restart.
//
// How: `running_exe_sha256` is not always available (`/proc/<pid>/exe` of a
// process owned by the service account is EACCES to any other non-root
// user), and `running_binary_is_current` then falls back to TIMESTAMPS. On
// the update-an-already-running-service path the pid the manager reports is
// the OUTGOING one — started long before the file was just rewritten — so
// the fallback answers `OlderThanBinary`, which `is_proven_stale()` counts
// as evidence and the install FAILS. That is exactly the `PidGone` lesson
// this function was written for, arriving through a different door.
//
// The distinction that fixes it without swallowing a real red is
// CONCLUSIVE vs PROXY:
//
// * `ExecMismatch` / `DeletedBinary` are CONTENT evidence about the
// running image. Returned at once, never retried — "retrying a mismatch
// until it goes away is how a verifier becomes a random number
// generator", and that sentence still stands.
// * `OlderThanBinary` is a TIMESTAMP proxy, reached only when the content
// question could not be asked. A proxy about a pid the manager is in the
// middle of replacing is a proxy about the WRONG PROCESS. So it is
// re-read: if `MainPID` moves, the new process is judged; if the SAME pid
// comes back, the finding is real (a rebuild with no restart) and is
// returned.
//
// "rebuilt without a restart" and "restarted 40 ms ago and the manager has
// not caught up" are different facts, and only re-reading the pid can tell
// them apart.
if let Freshness::OlderThanBinary { .. } = freshness {
if std::time::Instant::now() < deadline {
rereads += 1;
std::thread::sleep(std::time::Duration::from_millis(100));
let again = main_pid();
if again != Some(pid) {
// The manager moved on. Whatever that verdict was about, it was
// not about the process that is going to serve.
continue;
}
// Same pid, twice, 100 ms apart: this is a settled process and the
// proxy finding is a real one.
}
}
// Anything that is not a stale handle is a real answer — a match, a
// mismatch, a deleted inode, an unreadable /proc. Returned immediately:
// retrying a MISMATCH until it goes away is how a verifier becomes a
// random number generator.
// A pid mid-handover is the same kind of news as a pid that is gone:
// the question was about the wrong process, so ask the manager again
// rather than returning a verdict systemd is still in the middle of
// making false. Remembered so the out-of-budget report below names the
// reason it actually saw, not a `PidGone` it never observed.
if let Freshness::StillHandingOver { .. } = freshness {
if std::time::Instant::now() < deadline {
last_undetermined = Some(freshness);
rereads += 1;
std::thread::sleep(std::time::Duration::from_millis(100));
continue;
}
return RunningVerdict { pid: Some(pid), freshness, rereads };
}
if !matches!(freshness, Freshness::PidGone { .. }) {
return RunningVerdict {
pid: Some(pid),
freshness,
rereads,
};
}
}
if std::time::Instant::now() >= deadline {
// Out of budget. Report the last thing we knew, honestly — still
// UNDETERMINED, never upgraded to a pass.
return RunningVerdict {
pid: last_pid,
freshness: match (last_undetermined.take(), last_pid) {
(Some(seen), _) => seen,
(None, Some(pid)) => Freshness::PidGone { pid },
(None, None) => Freshness::Unknown(
"the manager never reported a MainPID to verify".to_string(),
),
},
rereads,
};
}
rereads += 1;
std::thread::sleep(std::time::Duration::from_millis(100));
}
}
/// systemd's own process-spawning helpers, by file name.
///
/// Since systemd 254 the manager forks `systemd-executor` and that process execs
/// the service, so for a window `/proc/<MainPID>/exe` is one of these rather than
/// the service's artifact. Older systemd re-execs `systemd` itself in the same
/// role, so both names are here.
pub const MANAGER_SPAWN_HELPERS: &[&str] = &["systemd-executor", "systemd"];
/// **Is `running_exe` systemd's spawn helper rather than a genuinely wrong
/// program?** — the one question that separates
/// [`Freshness::StillHandingOver`] from [`Freshness::ExecMismatch`].
///
/// Narrow on purpose, because the cost of being wide is swallowing the real red
/// this module exists for. All three must hold:
///
/// * the file name is one of [`MANAGER_SPAWN_HELPERS`];
/// * its directory is a systemd library directory (`/usr/lib/systemd`,
/// `/lib/systemd`) — a `/usr/local/bin/systemd-executor` a product installed is
/// not the manager's;
/// * it is not the SAME file name the caller expected. A unit whose `ExecStart`
/// genuinely is one of these gets a real mismatch verdict, unchanged.
pub fn is_manager_spawn_helper(running_exe: &Path, expected: &Path) -> bool {
let name = |p: &Path| p.file_name().map(|n| n.to_string_lossy().to_string());
let Some(running_name) = name(running_exe) else {
return false;
};
if !MANAGER_SPAWN_HELPERS.contains(&running_name.as_str()) {
return false;
}
if name(expected).as_deref() == Some(running_name.as_str()) {
return false;
}
running_exe
.parent()
.map(|dir| dir.to_string_lossy().to_string())
.is_some_and(|dir| dir == "/usr/lib/systemd" || dir == "/lib/systemd" || dir.ends_with("/systemd"))
}
/// Is `pid` running the binary currently at `binary`?
///
/// Linux-only in substance: everywhere else this is [`Freshness::Unknown`] rather
/// than an optimistic pass. See the module docs for the three questions asked —
/// the content hash decides, the timestamp only fills in when it cannot.
pub fn running_binary_is_current(pid: u32, binary: &Path) -> Freshness {
if !cfg!(target_os = "linux") {
return Freshness::Unknown("only /proc (Linux) can answer this".to_string());
}
let proc_dir = PathBuf::from(format!("/proc/{pid}"));
// (1) Is the running inode still linked? Recorded, but NOT yet a verdict: an
// unlinked inode whose bytes still hash equal really is the current program
// (an idempotent reinstall of identical bytes), and calling that stale is
// the false-STALE failure mode this module already paid for once.
let exe_link = proc_dir.join("exe");
let exe_target = match std::fs::read_link(&exe_link) {
Ok(target) => Some(target),
// A process that has EXITED is a stale handle, not an unreadable file. This
// one line is the difference between "re-read MainPID and ask again" and a
// warning nobody can act on — and on the update path it is the difference
// between a verdict and a permanent UNDETERMINED, because the pid a restart
// hands back is dying by construction.
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Freshness::PidGone { pid };
}
Err(e) => {
return Freshness::Unknown(format!("read /proc/{pid}/exe: {e}"));
}
};
// The `(deleted)` marker lives in the PATH STRING, so inspect it lossily rather
// than round-tripping through a Path (which would keep the suffix but hide it
// from a naive comparison). Recorded, but NOT yet a verdict — see (1) above.
let deleted_exe = exe_target
.as_ref()
.filter(|t| t.to_string_lossy().ends_with(DELETED_SUFFIX))
.cloned();
// (2) THE CONCLUSIVE CHECK — do the executing bytes hash to the artifact's
// bytes? This is the only question with no proxy in it, so it is asked
// first and its answer is final in both directions.
//
// EXCEPT for a script: `#!` makes the kernel execute the INTERPRETER, so
// `/proc/<pid>/exe` is `/bin/sh` and can never hash to the script file. The
// hash is not merely unhelpful there, it is actively wrong — it would report
// a mismatch for every healthy script service, and a check that cries wolf
// on healthy services gets muted. Scripts fall through to the timestamps,
// which remain perfectly valid for them.
if !is_script(binary) {
match (running_exe_sha256(pid), binary_sha256(binary)) {
(Ok(running), Ok(expected)) => {
if running == expected {
return Freshness::Current;
}
// A digest that is not the artifact's is normally conclusive — but
// systemd's own spawn helper hashes just as cleanly as a wrong
// program does, and it is not one. Ask which before calling it a
// finding (see `Freshness::StillHandingOver`).
if let Some(helper) = exe_target.as_ref().filter(|t| is_manager_spawn_helper(t, binary)) {
return Freshness::StillHandingOver { pid, helper: helper.clone() };
}
return Freshness::ExecMismatch {
running_sha256: running,
expected_sha256: expected,
binary: binary.to_path_buf(),
};
}
// One side could not be hashed, so the conclusive question is unavailable
// and we fall back to the proxies. This is NOT a corner case:
// `/proc/<pid>/exe` of a process owned by the service account is
// unreadable to any other non-root user (EACCES), while
// `/proc/<pid>/stat` stays world-readable — so an unprivileged status
// check lands here routinely.
//
// A deleted running exe is conclusive on its own and outranks timestamps.
_ => {
if let Some(exe) = deleted_exe {
return Freshness::DeletedBinary { exe };
}
}
}
} else if let Some(exe) = deleted_exe {
return Freshness::DeletedBinary { exe };
}
freshness_from_timestamps(pid, binary)
}
/// Does `path` begin with a `#!` shebang?
///
/// A script is executed by its INTERPRETER, so `/proc/<pid>/exe` names the
/// interpreter and hashing it against the script is guaranteed to mismatch. Reads
/// two bytes; anything unreadable answers `false` and lets the normal error paths
/// report the real problem.
fn is_script(path: &Path) -> bool {
use std::io::Read;
let Ok(mut f) = std::fs::File::open(path) else {
return false;
};
let mut magic = [0u8; 2];
matches!(f.read_exact(&mut magic), Ok(())) && &magic == b"#!"
}
/// The **fallback** proxy check, for when [`running_exe_sha256`] cannot answer:
/// did the process start before the binary was last written?
///
/// Split out from [`running_binary_is_current`] so it can be exercised on its own —
/// a fallback that is only ever reached in configurations the test suite does not
/// create is a fallback nobody has ever seen work.
fn freshness_from_timestamps(pid: u32, binary: &Path) -> Freshness {
let started = match process_start_time(pid) {
Ok(t) => t,
Err(e) => return Freshness::Unknown(format!("start time of pid {pid}: {e}")),
};
let binary_written = match std::fs::metadata(binary).and_then(|m| m.modified()) {
Ok(t) => t,
Err(e) => {
return Freshness::Unknown(format!("stat binary {}: {e}", binary.display()));
}
};
// `started` can only be reconstructed to about a second (see START_TIME_SLACK),
// and its error runs toward "too early" — so require the binary to be newer by
// more than that slack before calling the process stale. Comparing raw, or at
// whole-second resolution, accuses a process whose binary was written moments
// before it started.
if binary_written
.duration_since(started)
.map(|ahead| ahead > START_TIME_SLACK)
.unwrap_or(false)
{
Freshness::OlderThanBinary {
started,
binary_written,
}
} else {
Freshness::Current
}
}
/// Wall-clock start time of `pid`, from `/proc/<pid>/stat` field 22 (`starttime`,
/// clock ticks since boot) plus `btime` from `/proc/stat`.
///
/// Field 2 (`comm`) may contain spaces AND parentheses, so the fixed-position
/// fields are parsed after the LAST `')'` — splitting on whitespace from the start
/// mis-parses any process whose name contains a space.
fn process_start_time(pid: u32) -> Result<SystemTime, String> {
let stat = std::fs::read_to_string(format!("/proc/{pid}/stat"))
.map_err(|e| format!("read /proc/{pid}/stat: {e}"))?;
let after_comm = stat
.rfind(')')
.map(|i| &stat[i + 1..])
.ok_or_else(|| format!("/proc/{pid}/stat has no comm field"))?;
// After the last ')', the first field is `state` (field 3), so `starttime`
// (field 22) sits at index 22 - 3 = 19.
let ticks: u64 = after_comm
.split_whitespace()
.nth(19)
.ok_or_else(|| format!("/proc/{pid}/stat has no field 22 (starttime)"))?
.parse()
.map_err(|e| format!("starttime not a number: {e}"))?;
let sys = std::fs::read_to_string("/proc/stat").map_err(|e| format!("read /proc/stat: {e}"))?;
let btime: u64 = sys
.lines()
.find_map(|l| l.strip_prefix("btime "))
.ok_or("no btime in /proc/stat")?
.trim()
.parse()
.map_err(|e| format!("btime not a number: {e}"))?;
// USER_HZ is fixed at 100 for this procfs interface, independent of CONFIG_HZ.
const USER_HZ: u64 = 100;
// Add the ticks as MILLISECONDS, not as `ticks / USER_HZ` seconds: integer
// division discards up to 0.99 s and always rounds the start time down, i.e.
// always toward falsely accusing the process of being stale.
const MS_PER_TICK: u64 = 1_000 / USER_HZ;
Ok(SystemTime::UNIX_EPOCH
+ std::time::Duration::from_secs(btime)
+ std::time::Duration::from_millis(ticks * MS_PER_TICK))
}
#[cfg(test)]
mod tests {
use super::*;
/// **The systemd-254 handover window, on a real pid and a real `/proc` read.**
///
/// RED before the fix: `running_binary_is_current` answered `ExecMismatch` —
/// `is_proven_stale()`, so `--install` FAILED — for a process whose only sin
/// was being systemd's `systemd-executor` a few milliseconds before it execs
/// the service. Measured on the gunnar.rs front vm 2026-09-04; see
/// `Freshness::StillHandingOver`.
///
/// The helper is IMPERSONATED rather than mocked: a real executable is copied
/// to `<tmp>/systemd/systemd-executor` and run, so the verdict comes from an
/// actual `readlink` + `sha256` of an actual `/proc/<pid>/exe`.
#[cfg(target_os = "linux")]
#[test]
fn a_pid_still_executing_systemds_spawn_helper_is_undetermined_not_a_mismatch() {
let dir = std::env::temp_dir().join(format!("skidbladnir-handover-{}", std::process::id()));
let libdir = dir.join("systemd");
std::fs::create_dir_all(&libdir).unwrap();
// Two distinct real binaries: the "helper" and the "service artifact".
let helper = libdir.join("systemd-executor");
let service = dir.join("the-service");
// The helper is RUN, so it must be a real executable — and one that does
// NOT dispatch on `argv[0]`: on this box `/bin/sleep` and `/bin/cat` are
// symlinks into ONE multicall coreutils binary, which under the name
// `systemd-executor` hashes equal to each other and exits at once with
// "unknown program". `/bin/sh` (dash) is a single-purpose ELF. Both traps
// were caught by this test's own asserts, which is the point of them.
std::fs::copy("/bin/sh", &helper).unwrap();
std::fs::write(&service, b"\x7fELF not really, but distinct bytes and no shebang").unwrap();
assert_ne!(binary_sha256(&helper).unwrap(), binary_sha256(&service).unwrap(), "the two must differ or the test proves nothing");
let mut child = std::process::Command::new(&helper).arg("-c").arg("sleep 30").spawn().unwrap();
let pid = child.id();
wait_until_exec_completed(pid, &helper);
// THE FIX: judged against the service artifact, a pid running the helper is
// undetermined — and it must NOT be able to fail an install.
let verdict = running_binary_is_current(pid, &service);
assert!(
matches!(&verdict, Freshness::StillHandingOver { pid: p, helper: h } if *p == pid && h == &helper),
"{verdict:?}"
);
assert!(!verdict.is_proven_stale(), "an install must not fail on the handover window: {verdict:?}");
assert!(!verdict.is_current(), "nor may it read as a pass: {verdict:?}");
assert!(verdict.explain().contains("systemd-executor") && verdict.explain().contains("UNDETERMINED"), "{}", verdict.explain());
// And a unit whose ExecStart genuinely IS the helper gets `Current`, not an
// excuse: same file, same digest, no handover story needed.
assert_eq!(running_binary_is_current(pid, &helper), Freshness::Current);
// THE REAL RED IS STILL RED, and it has to be proven with a SECOND process:
// the same pid cannot show it, because that pid really is at a helper path.
// A process running from an ordinary path, judged against the service
// artifact, must still be the conclusive `ExecMismatch` that fails installs.
let impostor = dir.join("some-other-program");
std::fs::copy("/bin/sh", &impostor).unwrap();
let mut other_child = std::process::Command::new(&impostor).arg("-c").arg("sleep 30").spawn().unwrap();
let other_pid = other_child.id();
wait_until_exec_completed(other_pid, &impostor);
let real_red = running_binary_is_current(other_pid, &service);
assert!(matches!(real_red, Freshness::ExecMismatch { .. }), "the fix must swallow nothing: {real_red:?}");
assert!(real_red.is_proven_stale(), "{real_red:?}");
let _ = other_child.kill();
let _ = other_child.wait();
let _ = child.kill();
let _ = child.wait();
let _ = std::fs::remove_dir_all(&dir);
}
/// The predicate on its own, narrow in all three directions it must be.
#[test]
fn only_the_managers_own_helper_in_its_own_libdir_counts_as_a_handover() {
let svc = Path::new("/usr/local/bin/monetize-server");
assert!(is_manager_spawn_helper(Path::new("/usr/lib/systemd/systemd-executor"), svc));
assert!(is_manager_spawn_helper(Path::new("/lib/systemd/systemd-executor"), svc));
assert!(is_manager_spawn_helper(Path::new("/usr/lib/systemd/systemd"), svc));
// Not the manager's: a product that installed a binary by that name.
assert!(!is_manager_spawn_helper(Path::new("/usr/local/bin/systemd-executor"), svc));
// Not a spawn helper at all, however systemd-ish the path.
assert!(!is_manager_spawn_helper(Path::new("/usr/lib/systemd/systemd-journald"), svc));
assert!(!is_manager_spawn_helper(Path::new("/usr/bin/python3"), svc));
// A unit whose ExecStart IS the helper keeps its real mismatch verdict.
assert!(!is_manager_spawn_helper(
Path::new("/usr/lib/systemd/systemd-executor"),
Path::new("/opt/thing/systemd-executor")
));
}
/// Block until `pid` has finished `exec`ing `binary`, or panic.
///
/// `Command::spawn` returns after the fork, so a test that immediately acts on
/// the child's executable can race its exec. Polling `/proc/<pid>/exe` is the
/// only honest "it is really running that file now" signal available.
#[cfg(target_os = "linux")]
fn wait_until_exec_completed(pid: u32, binary: &Path) {
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
while std::time::Instant::now() < deadline {
if let Ok(exe) = std::fs::read_link(format!("/proc/{pid}/exe")) {
if exe == binary {
return;
}
}
std::thread::sleep(std::time::Duration::from_millis(10));
}
panic!("pid {pid} never exec'd {} within 10s", binary.display());
}
/// Place a runnable `/bin/sh` at `binary` (optionally stamping its mtime), start
/// it, and return only once it has really exec'd.
///
/// **Serialized across this module's tests on purpose.** `fs::copy` and
/// `set_times` hold a write fd on the destination, and a `fork` on ANOTHER test
/// thread makes the forked child inherit that fd. The fd is `O_CLOEXEC`, but it
/// stays open across the child's fork→exec window, and during it the kernel sees
/// a writer on the file — so an unrelated test's exec fails `ETXTBSY` on a file
/// it does not even own. Cargo runs these tests on parallel threads, so writing
/// and exec'ing must not interleave. The retry covers the same hazard from any
/// writer outside this module.
/// `ETXTBSY`. Matched by raw errno rather than `ErrorKind::ExecutableFileBusy`,
/// which is a much more recent stabilisation than the rest of this crate needs.
#[cfg(target_os = "linux")]
const ETXTBSY: i32 = 26;
/// Serializes every write-then-exec in this module. See [`write_then_exec`].
#[cfg(target_os = "linux")]
static EXEC_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
/// Run `prepare` (which writes an executable) and then spawn it, with the whole
/// pair held under one lock and the spawn retried past `ETXTBSY`.
///
/// **Both halves must be inside the lock, and that is the entire point.**
/// `fs::copy`/`File::options().write(..)` hold a write fd on the destination, and
/// a `fork` on ANOTHER test thread makes the forked child inherit it. The fd is
/// `O_CLOEXEC`, but it stays open across that child's fork→exec window, and
/// during it the kernel sees a writer on the file — so an unrelated test's exec
/// fails `ETXTBSY` on a file it does not even own. Cargo runs these tests on
/// parallel threads, so writing and exec'ing must not interleave.
///
/// Shared rather than copied per test: the script test was written with its own
/// bare `spawn` and flaked on exactly this within one run.
#[cfg(target_os = "linux")]
fn write_then_exec(
binary: &Path,
prepare: impl FnOnce(),
args: &[&str],
) -> std::process::Child {
// A panicking test must not poison the lock and cascade into the others.
let _guard = EXEC_LOCK
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
prepare();
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
loop {
match std::process::Command::new(binary).args(args).spawn() {
Ok(child) => break child,
Err(e)
if e.raw_os_error() == Some(ETXTBSY)
&& std::time::Instant::now() < deadline =>
{
std::thread::sleep(std::time::Duration::from_millis(10));
}
Err(e) => panic!("spawn {}: {e}", binary.display()),
}
}
}
#[cfg(target_os = "linux")]
fn exec_a_sleeper(binary: &Path, written_at: Option<SystemTime>) -> std::process::Child {
let child = write_then_exec(
binary,
|| {
std::fs::copy("/bin/sh", binary).expect("copy /bin/sh");
if let Some(mtime) = written_at {
// Stamped BEFORE exec: once the process is running, this file is
// its text segment and opening it for write is ETXTBSY.
std::fs::File::options()
.write(true)
.open(binary)
.expect("reopen to stamp mtime")
.set_times(std::fs::FileTimes::new().set_modified(mtime))
.expect("set mtime");
}
},
&["-c", "sleep 30"],
);
wait_until_exec_completed(child.id(), binary);
child
}
/// The **positive control**, against a genuinely running process: this very
/// test binary. It is running, its exe is not deleted, and it necessarily
/// started after it was built — so it MUST report `Current`. If this goes red
/// the check has become uselessly pessimistic and would cry stale on healthy
/// services.
#[test]
#[cfg(target_os = "linux")]
fn the_running_test_binary_reports_itself_current() {
let me = std::env::current_exe().expect("current_exe");
let f = running_binary_is_current(std::process::id(), &me);
assert_eq!(
f,
Freshness::Current,
"self-check must be Current: {}",
f.explain()
);
assert!(f.is_current());
}
/// **The oden defect, reproduced.** Start a process from a binary, delete that
/// binary out from under it, and the check must catch it — this is exactly the
/// korp-ui case where `--version` read the replacement file while the old inode
/// kept serving. RED if the `(deleted)` marker stops being detected.
#[test]
#[cfg(target_os = "linux")]
fn a_process_whose_binary_was_deleted_is_caught_not_called_current() {
let dir = tempfile::tempdir().expect("tempdir");
let victim = dir.path().join("victim");
// A real executable to run: copy the system shell rather than build one.
// `exec_a_sleeper` returns only once the exec has actually happened —
// `spawn` alone returns after the FORK, and unlinking in that window makes
// the child's own exec fail with ENOENT, leaving the test to measure a
// corpse. That was a live flake under load.
let mut child = exec_a_sleeper(&victim, None);
// Replace-by-unlink, precisely what a rebuild/redeploy does to a live process.
std::fs::remove_file(&victim).expect("unlink the running binary");
let f = running_binary_is_current(child.id(), &victim);
let _ = child.kill();
let _ = child.wait();
match &f {
Freshness::DeletedBinary { .. } => {}
other => panic!("a deleted running binary must be caught, got {other:?}"),
}
assert!(!f.is_current(), "a deleted binary is never current");
assert!(
f.explain().contains("DELETED"),
"the reason must say so: {}",
f.explain()
);
}
/// A process that started BEFORE the binary it is supposed to be running is
/// stale — the "deployed a new generation, never restarted" half, asserted
/// against the timestamp fallback DIRECTLY.
///
/// Note the scenario this deliberately does NOT use: rewriting a running
/// binary IN PLACE is impossible on Linux (`ETXTBSY` — "Text file busy"), so a
/// real deploy either unlinks and replaces it (caught as
/// [`Freshness::DeletedBinary`]) or writes a NEW generation and swaps the
/// `current` symlink. This models the second: the process runs generation N
/// while the path the layout now points at is generation N+1, written
/// afterwards. Nothing is deleted and the old process looks perfectly healthy to
/// `status()` — only the timestamps give it away.
///
/// **Why the fallback and not the public entry point.** Both generations here
/// are copies of `/bin/sh`, i.e. byte-IDENTICAL, and the content hash now
/// (rightly) calls that `Current`: a generation whose bytes equal the running
/// image is the same program, and a stale verdict on it would be a false
/// accusation. So this fixture can only speak about the timestamp proxy, and it
/// is pointed at the timestamp proxy. `the_timestamp_fallback_is_reached_when_
/// the_hash_cannot_be_taken` covers the wiring.
#[test]
#[cfg(target_os = "linux")]
fn a_process_older_than_the_deployed_binary_is_stale() {
let dir = tempfile::tempdir().expect("tempdir");
let gen_n = dir.path().join("gen-n");
// The service starts from generation N.
let mut child = exec_a_sleeper(&gen_n, None);
// A new generation is deployed; `current` would now resolve here. The oden
// case had an HOUR between start and rebuild, so state that outright by
// setting the mtime rather than sleeping past START_TIME_SLACK — a sleep
// just long enough to clear the slack would make this test a stopwatch race
// against the slack constant, which is exactly the flake being fixed.
let gen_n1 = dir.path().join("gen-n1");
std::fs::copy("/bin/sh", &gen_n1).expect("write generation N+1");
let an_hour_on = SystemTime::now() + std::time::Duration::from_secs(3600);
std::fs::File::options()
.write(true)
.open(&gen_n1)
.expect("open generation N+1")
.set_times(std::fs::FileTimes::new().set_modified(an_hour_on))
.expect("set generation N+1 mtime");
let f = freshness_from_timestamps(child.id(), &gen_n1);
let _ = child.kill();
let _ = child.wait();
match &f {
Freshness::OlderThanBinary { .. } => {}
other => panic!("a process predating the deployed binary must be stale, got {other:?}"),
}
assert!(!f.is_current());
assert!(
f.explain().contains("restart != rebuild"),
"{}",
f.explain()
);
}
/// **The fallback is WIRED, not merely present.** Reproduces the real reason the
/// hash goes unavailable — the expected binary cannot be OPENED (mode `0000`,
/// the unprivileged-caller/EACCES shape) while it can still be STAT'd — and
/// asserts that [`running_binary_is_current`] then still returns a timestamp
/// verdict instead of collapsing to `Unknown`.
///
/// Without this, the timestamp branch would be reachable only through a private
/// function the public entry point might have stopped calling.
#[test]
#[cfg(target_os = "linux")]
fn the_timestamp_fallback_is_reached_when_the_hash_cannot_be_taken() {
use std::os::unix::fs::PermissionsExt;
// Root bypasses the permission bits entirely, so the premise cannot hold.
if crate::systemd::current_euid() == 0 {
eprintln!("skipped: running as root, mode 0000 does not deny us");
return;
}
let dir = tempfile::tempdir().expect("tempdir");
let gen_n = dir.path().join("gen-n");
let mut child = exec_a_sleeper(&gen_n, None);
let gen_n1 = dir.path().join("gen-n1");
std::fs::copy("/bin/sh", &gen_n1).expect("write generation N+1");
let an_hour_on = SystemTime::now() + std::time::Duration::from_secs(3600);
std::fs::File::options()
.write(true)
.open(&gen_n1)
.expect("open generation N+1")
.set_times(std::fs::FileTimes::new().set_modified(an_hour_on))
.expect("set generation N+1 mtime");
// Unreadable, but still stat-able — exactly what an unprivileged caller sees.
std::fs::set_permissions(&gen_n1, std::fs::Permissions::from_mode(0o000))
.expect("chmod 000");
// Prove the premise rather than assuming it: the hash really is unavailable
// and the metadata really is not. A test that silently lost its premise
// would pass for the wrong reason.
assert!(
binary_sha256(&gen_n1).is_err(),
"premise: the artifact must be unhashable"
);
assert!(
std::fs::metadata(&gen_n1).is_ok(),
"premise: it must still be stat-able"
);
let f = running_binary_is_current(child.id(), &gen_n1);
let _ = child.kill();
let _ = child.wait();
match &f {
Freshness::OlderThanBinary { .. } => {}
other => panic!(
"with the hash unavailable the public check must fall back to timestamps, \
got {other:?}"
),
}
assert!(!f.is_current());
}
/// **The false STALE, pinned.** The mirror defect of the false green: a healthy
/// process accused of running old code because its start time could not be
/// reconstructed precisely enough.
///
/// This was live and reproducible — `cargo test` builds the test binary and then
/// immediately runs it, so the binary's mtime lands a fraction of a second before
/// the process start, and the reconstruction (integer `ticks / USER_HZ`, plus a
/// whole-second `btime`) placed the start ~1.5 s BEFORE its own binary was
/// written. The positive control above went red on every fresh build.
///
/// A deploy check that cries stale on healthy services gets muted, and a muted
/// check catches nothing — so this failure mode is as fatal as the false green.
/// Unlike the positive control, this does not depend on having just rebuilt:
/// the sub-slack gap is set explicitly.
#[test]
#[cfg(target_os = "linux")]
fn a_binary_written_moments_before_the_process_started_is_not_called_stale() {
let dir = tempfile::tempdir().expect("tempdir");
let bin = dir.path().join("just-built");
// Exactly the `cargo test` shape: the binary's mtime sits a hair AFTER the
// start time the reconstruction will produce, but well inside the slack.
// Nothing was rebuilt behind this process's back; nothing is stale. The
// stamp goes on before exec — afterwards the file is the running text
// segment and opening it for write is ETXTBSY.
let a_hair_late = SystemTime::now() + std::time::Duration::from_millis(500);
let mut child = exec_a_sleeper(&bin, Some(a_hair_late));
let f = running_binary_is_current(child.id(), &bin);
let _ = child.kill();
let _ = child.wait();
assert_eq!(
f,
Freshness::Current,
"a sub-slack mtime gap must not read as stale: {}",
f.explain()
);
}
/// **THE DEFECT, in the shape the timestamp checks cannot see.** A process runs
/// program A; the path the unit points at now holds program B. Nothing is
/// deleted, and B is NOT newer than the process start — so both pre-existing
/// questions answer "looks fine" and only the content hash tells the truth.
///
/// This is the `StartUnit`-was-a-no-op case: the installer wrote a new
/// `ExecStart=` target, systemd said `active`, and the previous build kept
/// serving. RED if the hash comparison is ever dropped or short-circuited.
#[test]
#[cfg(target_os = "linux")]
fn a_process_executing_different_bytes_is_caught_by_the_hash_alone() {
let dir = tempfile::tempdir().expect("tempdir");
let running = dir.path().join("program-a");
let mut child = exec_a_sleeper(&running, None);
// Program B: genuinely different bytes at a path the unit would now name.
// Stamped OLDER than the process start, so `OlderThanBinary` CANNOT fire and
// the hash is demonstrably the only thing catching this.
let expected = dir.path().join("program-b");
std::fs::copy("/bin/sh", &expected).expect("write program B");
{
use std::io::Write;
let mut f = std::fs::File::options()
.append(true)
.open(&expected)
.expect("open program B");
// A shell script comment: changes the bytes, keeps the file valid.
f.write_all(b"\n# a different program\n")
.expect("append to program B");
}
let an_hour_ago = SystemTime::now() - std::time::Duration::from_secs(3600);
std::fs::File::options()
.write(true)
.open(&expected)
.expect("reopen program B")
.set_times(std::fs::FileTimes::new().set_modified(an_hour_ago))
.expect("stamp program B older than the process");
// Prove the premise, or this test could pass for the wrong reason: the two
// files really do differ, and B really is older than the running process.
let sha_a = binary_sha256(&running).expect("hash program A");
let sha_b = binary_sha256(&expected).expect("hash program B");
assert_ne!(
sha_a, sha_b,
"the fixture must actually use two different programs"
);
let f = running_binary_is_current(child.id(), &expected);
let _ = child.kill();
let _ = child.wait();
match &f {
Freshness::ExecMismatch {
running_sha256,
expected_sha256,
..
} => {
assert_eq!(running_sha256, &sha_a, "must report the EXECUTING digest");
assert_eq!(expected_sha256, &sha_b, "must report the ARTIFACT digest");
}
other => panic!(
"different executing bytes must be caught by the hash, got {other:?} — \
note neither the deleted marker nor the mtime can fire here"
),
}
assert!(!f.is_current());
assert!(f.explain().contains("DIFFERENT program"), "{}", f.explain());
}
/// The mirror: the SAME bytes at a different path are not a mismatch. A digest
/// check that cannot say "yes" is as useless as one that cannot say "no", and
/// this is what stops the new red from being a blanket accusation.
#[test]
#[cfg(target_os = "linux")]
fn the_same_bytes_at_another_path_are_current() {
let dir = tempfile::tempdir().expect("tempdir");
let running = dir.path().join("gen-1");
let mut child = exec_a_sleeper(&running, None);
// A byte-identical copy, stamped an hour in the FUTURE — so the timestamp
// check would cry `OlderThanBinary` and the hash must overrule it.
let identical = dir.path().join("gen-2");
std::fs::copy(&running, &identical).expect("copy identical bytes");
let an_hour_on = SystemTime::now() + std::time::Duration::from_secs(3600);
std::fs::File::options()
.write(true)
.open(&identical)
.expect("reopen identical")
.set_times(std::fs::FileTimes::new().set_modified(an_hour_on))
.expect("stamp identical newer");
let f = running_binary_is_current(child.id(), &identical);
let _ = child.kill();
let _ = child.wait();
assert_eq!(
f,
Freshness::Current,
"identical bytes are the same program however the clock reads: {}",
f.explain()
);
}
/// The hash must be read back from `/proc/<pid>/exe` even once the file is
/// UNLINKED — that is the only route to the running bytes, and the property the
/// whole conclusive check rests on.
#[test]
#[cfg(target_os = "linux")]
fn the_running_image_can_still_be_hashed_after_the_file_is_deleted() {
let dir = tempfile::tempdir().expect("tempdir");
let victim = dir.path().join("victim");
let mut child = exec_a_sleeper(&victim, None);
let before = running_exe_sha256(child.id()).expect("hash while linked");
std::fs::remove_file(&victim).expect("unlink the running binary");
let after = running_exe_sha256(child.id());
let _ = child.kill();
let _ = child.wait();
let after = after.expect("the running image must remain hashable after unlink");
assert_eq!(
before, after,
"unlinking must not change the executing bytes"
);
}
/// **The false RED the hash would otherwise cause on every script service.**
///
/// A `#!` file is executed by its INTERPRETER, so `/proc/<pid>/exe` is
/// `/bin/sh` and can never hash to the script. Comparing them would report
/// `ExecMismatch` for a perfectly healthy service — and a check that accuses
/// healthy services is a check that gets muted, which is the same outcome as
/// having no check at all.
///
/// The premise is asserted, not assumed: the two digests really do differ, so
/// this test would fail if the shebang branch were removed.
#[test]
#[cfg(target_os = "linux")]
fn a_script_service_is_not_accused_by_a_hash_it_can_never_match() {
use std::io::Write;
use std::os::unix::fs::PermissionsExt;
let dir = tempfile::tempdir().expect("tempdir");
let script = dir.path().join("svc.sh");
let mut child = write_then_exec(
&script,
|| {
let mut f = std::fs::File::create(&script).expect("create script");
f.write_all(b"#!/bin/sh\nexec sleep 30\n")
.expect("write script");
drop(f);
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755))
.expect("chmod +x");
},
&[],
);
// The script `exec`s sleep, so wait for /proc/<pid>/exe to settle onto the
// interpreter rather than measuring the shell mid-exec.
std::thread::sleep(std::time::Duration::from_millis(300));
// Premise: the running image is NOT the script, so a naive hash compare
// would mismatch. If this ever stops holding, the test below is vacuous.
if let (Ok(running), Ok(on_disk)) = (running_exe_sha256(child.id()), binary_sha256(&script))
{
assert_ne!(
running, on_disk,
"premise: a script's running image is the interpreter, not the script"
);
}
let f = running_binary_is_current(child.id(), &script);
let _ = child.kill();
let _ = child.wait();
assert!(
!matches!(f, Freshness::ExecMismatch { .. }),
"a healthy script service must not be accused of running different bytes: {}",
f.explain()
);
}
/// **`Unknown` must never read as a pass.** The whole class of defect here is
/// an unanswerable question being scored as healthy.
/// **RED — a pid that is GONE must say so, not "could not read a file".**
///
/// The verifier is handed a `MainPID` and the process behind it can exit before it
/// is read. That is a STALE HANDLE — re-read and retry — and it is a completely
/// different fact from "I read a live process's exe and it does not match", which
/// is news. Today both arrive as `Unknown("read /proc/N/exe: No such file …")`,
/// so the caller cannot retry the first or shout about the second.
#[test]
fn a_pid_that_has_exited_is_reported_as_a_stale_handle_not_an_unreadable_file() {
// A genuinely dead, reaped pid: spawn, wait, then look.
let child = std::process::Command::new("/bin/true")
.spawn()
.expect("spawn /bin/true");
let dead = child.id();
let mut child = child;
child.wait().expect("reap");
let me = std::env::current_exe().expect("current_exe");
let f = running_binary_is_current(dead, &me);
assert!(
matches!(f, Freshness::PidGone { pid } if pid == dead),
"a reaped pid is a stale HANDLE, not a verdict about any binary: {f:?}"
);
assert!(!f.is_current(), "and it is still not a pass");
}
/// **RED — the verifier must re-read `MainPID` when the pid it holds has died.**
///
/// The update-an-already-running-service path hands the checker a pid that is
/// being torn down. One read of it can only ever fail. The seam has to be allowed
/// to ask the manager again.
#[test]
fn a_stale_pid_is_retried_against_a_freshly_read_one() {
let child = std::process::Command::new("/bin/true").spawn().unwrap();
let dead = child.id();
let mut child = child;
child.wait().unwrap();
let me = std::env::current_exe().expect("current_exe");
let mut reads = 0;
// First two reads hand back the doomed pid, as the manager does during a
// restart; the third reports the process that is really serving.
let verdict = verify_current_binary(
|| {
reads += 1;
Some(if reads <= 2 { dead } else { std::process::id() })
},
&me,
std::time::Duration::from_secs(5),
);
assert_eq!(
verdict.freshness,
Freshness::Current,
"a stale handle must be retried, not reported: {verdict:?}"
);
assert_eq!(verdict.pid, Some(std::process::id()));
assert!(reads >= 3, "it has to actually re-read: {reads}");
}
/// Neither UNDETERMINED shape is a pass, and both say "not a pass" out loud.
///
/// `PidGone` was split out of `Unknown` so the caller can RETRY one and report the
/// other; the split must not weaken the property that matters to a reader, which
/// is that neither reads as success.
/// **Exactly three verdicts may fail an install, and UNDETERMINED is not one.**
///
/// The property is pinned as a table rather than trusted to the order of match
/// arms, because the arm-order mistake has already been made here once: the doc
/// comment promised a warning for `Unknown` while a catch-all `stale =>` swallowed
/// it and failed the install. `PidGone` is the same trap one axis over, and it
/// fires on the MOST COMMON install path there is.
#[test]
fn only_a_proven_finding_may_fail_an_install() {
let proven = [
Freshness::ExecMismatch {
running_sha256: "a".into(),
expected_sha256: "b".into(),
binary: PathBuf::from("/bin/x"),
},
Freshness::DeletedBinary {
exe: PathBuf::from("/bin/x (deleted)"),
},
Freshness::OlderThanBinary {
started: SystemTime::UNIX_EPOCH,
binary_written: SystemTime::now(),
},
];
for f in &proven {
assert!(f.is_proven_stale(), "{f:?} is evidence and must fail");
assert!(!f.is_current());
}
let not_proven = [
Freshness::Current,
Freshness::PidGone { pid: 42 },
Freshness::Unknown("permission denied".into()),
];
for f in ¬_proven {
assert!(
!f.is_proven_stale(),
"{f:?} must NOT fail an install — an unanswered question is not a failed one"
);
}
// …and only one of those is a pass.
assert!(Freshness::Current.is_current());
assert!(!Freshness::PidGone { pid: 42 }.is_current());
assert!(!Freshness::Unknown("x".into()).is_current());
}
#[test]
fn an_undetermined_answer_is_not_current() {
// PID 0 is never a real process. It reads as a stale handle now — the same
// "not a pass", with the extra information that re-reading might help.
let f = running_binary_is_current(0, Path::new("/nonexistent/binary"));
assert!(!f.is_current(), "undetermined must not be current: {f:?}");
assert!(matches!(f, Freshness::PidGone { pid: 0 }), "got {f:?}");
assert!(f.explain().contains("not a pass"), "{}", f.explain());
assert!(
f.explain().contains("STALE HANDLE"),
"…and it must say which KIND of undetermined it is: {}",
f.explain()
);
// The other shape — a live pid whose exe cannot be read for some other reason
// — stays `Unknown` and is equally not a pass.
let u = Freshness::Unknown("permission denied".to_string());
assert!(!u.is_current());
assert!(u.explain().contains("not a pass"));
}
}