Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/Skidbladnir

src/service_drive.rs

//! **The `service` kind's drivable seam** — install a unit, start it, wait for a
//! NAMED READINESS MARKER, ask the service a real question, and tear it down.
//!
//! # The wall this exists on the far side of
//!
//! The suite's chain matrix walks every declared build thing BUILT → INSTALLED →
//! RUNS → FUNC. Stage 2 is supposed to go through draupnir's one `BootSpec`, and
//! draupnir grew an [`exe`](https://git.gunnar.rs/mikronova/draupnir) backend on
//! 2026-08-15 that does exactly this shape for a plain child process. The `service`
//! kind — 12 of the 44 rows, MEASURED 2026-08-15 at
//! `edda/crates/nornir-testmatrix/tests/chain_matrix.rs:190` — could not follow it,
//! because a systemd backend inside draupnir is **structurally impossible**:
//!
//! ```text
//! Skidbladnir/Cargo.toml:87   draupnir = { path = "../draupnir" }
//! ```
//!
//! Skidbladnir already depends on draupnir (features `kvm`, `msi-oci`, `pack-oci`),
//! and cargo forbids a package-level cycle even through optional deps. The arrow
//! that DOES exist is Skidbladnir → draupnir, so the systemd shape lives here, on the
//! side that already owns every unit in the constellation, and REUSES draupnir's
//! observation type rather than twinning it.
//!
//! # `draupnir::Seen`, not a second opinion about "seen"
//!
//! [`Seen`] is draupnir's, unchanged: `Marker { line, after }` /
//! `StillRunning { tail, waited }` / `Exited { code, tail, after }`. A `service` row
//! and an `exe` row therefore report readiness in ONE vocabulary, and the chain
//! driver renders both cells the same way. Copying the enum would have been ~60
//! duplicated lines and two definitions of "seen" that drift.
//!
//! It also carries draupnir's rule, which is the whole point: **a wait reports what
//! it observed and never invents a verdict.** A budget elapsing is an observation.
//! Deciding what it means is the caller's job.
//!
//! # ACTIVE IS NOT ANSWERING
//!
//! `ActiveState == "active"` is the answer this crate could already give, and it is
//! not readiness. systemd marks a `Type=simple` unit active the instant it forks the
//! process — before that process has parsed a flag, opened its data root, or bound a
//! port. A driver that grades FUNC on `is_active` grades the fork.
//!
//! So this module waits for a **named string the service itself printed**, read out
//! of the file its `StandardOutput=append:` sends both streams to
//! ([`crate::systemd::SystemdServiceSpec::stdout_file`] — a journal reader would be
//! the other route, and Skidbladnir has none: no `sd-journal`, no `libsystemd`).
//!
//! And a marker is still not proof that anything answered — see
//! [`ServiceDrive::await_marker`] for exactly what it does and does not prove, and
//! [`ServiceDrive::answered`] for the half that closes it.
//!
//! # Zero-shell
//!
//! Every systemd verb goes over pure-Rust D-Bus ([`crate::systemd::Systemd`], zbus).
//! No `systemctl`, no `journalctl`, no shell. The only filesystem work is writing the
//! unit (which this crate has always done) and reading a log file with `std::fs`.

use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};

use draupnir::Seen;

use crate::service::{HealthProbe, Lifecycle, ServiceSpec};
use crate::systemd::{RestartPolicy, Scope, Systemd};
use crate::uninstall::UninstallReport;

/// Default readiness budget. Long enough for a real server to open its store and
/// bind (gunnar-server: 18 ms on this box, MEASURED 2026-08-15), short enough that a
/// row that will never come up does not hold the matrix hostage.
pub const DEFAULT_MARKER_BUDGET: Duration = Duration::from_secs(30);

/// Default poll interval while waiting. The marker file is a local read; 100 ms costs
/// nothing and bounds the reported `after` to a tenth of a second.
pub const DEFAULT_MARKER_POLL: Duration = Duration::from_millis(100);

/// **Drive one `service`-kind build thing.** Wraps the ship's own
/// [`Lifecycle`] — this adds no second way to write a unit, it adds the WAIT and the
/// QUESTION that `Lifecycle` had no shape for.
#[derive(Debug, Clone)]
pub struct ServiceDrive {
    life: Lifecycle,
    log: PathBuf,
}

/// What a real request to the running service got back.
///
/// Deliberately three states, not a boolean: "nobody declared a probe" and "the probe
/// ran and got nothing" are different facts, and collapsing them is how a row with no
/// health check reads as healthy.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Answered {
    /// The declared probe got a correct answer. `said` carries the server's own words
    /// when the probe is one that hears words ([`HealthProbe::Banner`]); the TCP and
    /// HTTP arms return a verdict without a transcript, so it is `None` there.
    Yes {
        /// What the server said, when the probe heard anything.
        said: Option<String>,
        /// Which probe asked, named.
        probe: String,
    },
    /// The probe ran and the service did not answer correctly.
    No {
        /// Which probe asked, named.
        probe: String,
    },
    /// No probe is declared on the spec. **Not a pass.** A row landing here has no
    /// FUNC evidence at all and must render as absent, never green.
    NotDeclared,
}

impl Answered {
    /// Whether a real request got a correct answer. [`Answered::NotDeclared`] is
    /// `false` — an unasked question is not a passed one.
    pub fn is_yes(&self) -> bool {
        matches!(self, Answered::Yes { .. })
    }

    /// A one-line, quotable summary carrying the applied output — the same contract
    /// [`Seen::detail`] has, so a matrix cell reads the same for both halves.
    pub fn detail(&self) -> String {
        match self {
            Answered::Yes {
                said: Some(s),
                probe,
            } => format!("{probe} answered: {s}"),
            Answered::Yes { said: None, probe } => format!("{probe} answered"),
            Answered::No { probe } => format!("{probe} got NO correct answer"),
            Answered::NotDeclared => {
                "no health probe declared on the spec — NOTHING was asked".to_string()
            }
        }
    }
}

/// What [`ServiceDrive::drive`] observed, end to end. Every field is an observation;
/// none is a verdict.
#[derive(Debug, Clone)]
pub struct DriveReport {
    /// The unit name that was driven.
    pub unit: String,
    /// The marker file the wait read.
    pub log: PathBuf,
    /// Why the install failed, when it did. `Some` here means every field below is
    /// the un-run default.
    pub install_error: Option<String>,
    /// The readiness observation — draupnir's vocabulary.
    pub seen: Option<Seen>,
    /// What a real request got back.
    pub answered: Answered,
    /// The `MainPID` systemd reported while the service was up — the handle
    /// [`crate::running`] needs to ask whether that process is the binary on disk.
    pub main_pid: Option<u32>,
    /// How many times systemd restarted the unit during the wait. Non-zero means the
    /// process DIED, whatever `ActiveState` said at any sample.
    pub restarts: Option<u32>,
    /// Teardown outcome — the error text if stop/uninstall failed, so a leaked unit
    /// is never silent.
    pub teardown_error: Option<String>,
    /// Facts the caller must see: the restart policy hiding a death, a data root that
    /// was erased, and so on. NO SILENT CAPS.
    pub notes: Vec<String>,
}

impl DriveReport {
    /// The one line a matrix cell's `detail` should carry: what was seen, and what
    /// answered.
    pub fn detail(&self) -> String {
        if let Some(e) = &self.install_error {
            return format!("`{}` was never installed: {e}", self.unit);
        }
        let seen = self
            .seen
            .as_ref()
            .map(Seen::detail)
            .unwrap_or_else(|| "no readiness wait ran".to_string());
        let mut d = format!("`{}`: {seen}; {}", self.unit, self.answered.detail());
        if let Some(n) = self.restarts.filter(|n| *n > 0) {
            d.push_str(&format!(" [systemd restarted it {n}×]"));
        }
        if let Some(e) = &self.teardown_error {
            d.push_str(&format!(" [TEARDOWN FAILED: {e}]"));
        }
        d
    }

    /// **Did the service come up AND answer?** Both halves, because either alone is a
    /// known false green: a marker without an answer is a process that printed and
    /// hung, and an answer without a marker means the driver never watched the thing
    /// it was told to watch.
    pub fn is_func(&self) -> bool {
        self.install_error.is_none()
            && self.seen.as_ref().is_some_and(Seen::saw_marker)
            && self.answered.is_yes()
    }
}

/// The one call's inputs.
#[derive(Debug, Clone)]
pub struct DriveRequest<'a> {
    /// The NAMED string the service prints when it is up. Never empty — an empty
    /// marker matches every line, which is a wait that always passes.
    pub marker: &'a str,
    /// How long to wait for it.
    pub budget: Duration,
    /// How often to look.
    pub poll: Duration,
    /// Leave the service RUNNING instead of tearing it down. What the chain driver
    /// wants when later stages still have questions for it; the teardown is then the
    /// caller's debt, and [`DriveReport::teardown_error`] stays `None`.
    pub keep_running: bool,
}

impl<'a> DriveRequest<'a> {
    /// A request with the default budget and poll, tearing the service down after.
    pub fn new(marker: &'a str) -> Self {
        DriveRequest {
            marker,
            budget: DEFAULT_MARKER_BUDGET,
            poll: DEFAULT_MARKER_POLL,
            keep_running: false,
        }
    }
    /// Set the readiness budget.
    pub fn budget(mut self, b: Duration) -> Self {
        self.budget = b;
        self
    }
    /// Set the poll interval.
    pub fn poll(mut self, p: Duration) -> Self {
        self.poll = p;
        self
    }
    /// Leave the service running for later stages.
    pub fn keep_running(mut self, k: bool) -> Self {
        self.keep_running = k;
        self
    }
}

impl ServiceDrive {
    /// Wrap a [`ServiceSpec`] that declares its marker channel.
    ///
    /// **Refuses a spec with no [`ServiceSpec::stdout_file`], by name.** Without it a
    /// service's words go to the journal, which this crate cannot read, and the only
    /// question left is `ActiveState` — the false green this module exists to kill.
    /// Refusing at construction beats a driver that quietly downgrades to "active".
    pub fn new(spec: ServiceSpec) -> anyhow::Result<Self> {
        let log = spec.stdout_file.clone().ok_or_else(|| {
            anyhow::anyhow!(
                "service `{}` declares no stdout_file, so it has no readiness marker \
                 channel: its output would go to the journal, Skidbladnir has no journal \
                 reader, and the only remaining question would be `ActiveState == active` \
                 — which systemd answers the instant the process FORKS. Call \
                 ServiceSpec::stdout_file(<absolute path>).",
                spec.name
            )
        })?;
        Ok(ServiceDrive {
            life: Lifecycle::new(spec),
            log: PathBuf::from(log),
        })
    }

    /// The spec being driven.
    pub fn spec(&self) -> &ServiceSpec {
        self.life.spec()
    }
    /// The ship's lifecycle for this spec — install/start/stop/status/health/uninstall.
    pub fn lifecycle(&self) -> &Lifecycle {
        &self.life
    }
    /// The systemd unit name (`<name>.service`).
    pub fn unit(&self) -> String {
        self.life.unit()
    }
    /// The unit text that will be written — by the ONE renderer, so a caller can
    /// assert the applied `StandardOutput=` line without installing anything.
    pub fn unit_text(&self) -> String {
        self.life.unit_text()
    }
    /// The marker file both streams are appended to.
    pub fn log_path(&self) -> &Path {
        &self.log
    }
    /// Which systemd manager this drives — system bus, or the caller's session bus.
    pub fn scope(&self) -> Scope {
        self.spec().scope
    }

    /// **Everything the service has said so far** — the file its stdout+stderr are
    /// appended to, read lossily (a service is free to emit non-UTF-8 and that must
    /// not be an error). A file that does not exist yet reads as empty: the unit is
    /// installed before the process has written a byte.
    pub fn captured(&self) -> String {
        match std::fs::read(&self.log) {
            Ok(b) => String::from_utf8_lossy(&b).into_owned(),
            Err(_) => String::new(),
        }
    }

    /// **Can this spec's death be observed at all?**
    ///
    /// `Restart=no` ⇒ a dead service rests in `failed` and the wait reports
    /// [`Seen::Exited`] directly. `Restart=on-failure`/`always` ⇒ it never rests
    /// there; the wait catches the death in the `SubState == "auto-restart"` window
    /// (with `NRestarts` as a backstop for a window shorter than one poll). Either way
    /// the death is caught — this reports WHICH mechanism is doing it, because the
    /// second one is sampled and the backstop leg cannot attribute an exit code.
    pub fn death_rests_in_failed(&self) -> bool {
        self.spec().restart == RestartPolicy::No
    }

    /// **INSTALL** — truncate the marker file, then hand the spec to the ONE unit
    /// writer ([`Lifecycle::install`] → `install_systemd_scoped`): write the unit,
    /// `daemon-reload`, enable, start. All over D-Bus; no `systemctl`.
    ///
    /// Truncating first is not tidiness. A marker left in the file by the PREVIOUS
    /// run is read by the next wait in microseconds, and the row goes green without
    /// the service ever having started — the exact stale-evidence false green this
    /// whole module is built against.
    #[cfg(unix)]
    pub fn install(&self) -> anyhow::Result<()> {
        if let Some(dir) = self.log.parent() {
            std::fs::create_dir_all(dir).map_err(|e| {
                anyhow::anyhow!("create marker-file dir {}: {e}", dir.display())
            })?;
        }
        // Truncate-or-create. `write(&self.log, "")` is the whole operation.
        std::fs::write(&self.log, b"").map_err(|e| {
            anyhow::anyhow!(
                "truncate marker file {}: {e} — refusing to wait against output a \
                 previous run left behind",
                self.log.display()
            )
        })?;
        let out = self.life.install();
        crate::surface_status!(
            "skidbladnir/service_drive",
            "install_unit",
            out.is_ok(),
            "`{}`: {}",
            self.unit(),
            match &out {
                Ok(()) => "installed + started".to_string(),
                Err(e) => format!("install failed: {e}"),
            }
        );
        out
    }

    /// **WAIT FOR A NAMED MARKER — never for the unit to stop.**
    ///
    /// Polls the marker file for `marker` and reports what it [`Seen`]:
    ///
    /// * [`Seen::Marker`] — the line the marker appeared on, and how long it took.
    /// * [`Seen::Exited`] — the service DIED before saying it. Reached three ways: the
    ///   unit came to rest in `failed`/`inactive` (`Restart=no`); its `SubState` is
    ///   `auto-restart` (`Restart=on-failure`/`always`, where `ActiveState` says
    ///   `activating` and would otherwise hide the death forever); or, as a backstop,
    ///   systemd's `NRestarts` went up during the wait. `code` is `ExecMainStatus` in
    ///   the first two cases and **`None` in the third** — by then the property
    ///   describes the REPLACEMENT process, and a wrong number is worse than no number.
    /// * [`Seen::StillRunning`] — the budget elapsed with the unit up and the marker
    ///   absent. **Not an error and not a failure**: for a server, still-running is
    ///   the healthy state. It carries the tail so the caller can say why.
    ///
    /// `Err` is reserved for "the bus could not be reached" — a broken observer, not
    /// an observation.
    ///
    /// # What a marker PROVES
    ///
    /// That the service's own code reached the point where it prints that string.
    /// For `gunnar-server` the chosen marker is emitted after the listener is bound,
    /// so it proves the bind happened.
    ///
    /// # What a marker does NOT prove
    ///
    /// **That anything answers.** A process can print `listening` and then wedge, or
    /// bind and then fail every request. It also does not prove the marker came from
    /// THIS run if the file was not truncated (which is why
    /// [`install`](ServiceDrive::install) truncates), nor that the string means what
    /// the driver thinks — the marker is a contract with the product's source, and a
    /// product that changes its startup line silently turns this red.
    ///
    /// The half that closes it is [`answered`](ServiceDrive::answered): a real
    /// request over the wire. Grade FUNC on BOTH.
    #[cfg(unix)]
    pub fn await_marker(
        &self,
        marker: &str,
        budget: Duration,
        poll: Duration,
    ) -> anyhow::Result<Seen> {
        anyhow::ensure!(
            !marker.trim().is_empty(),
            "an empty readiness marker matches every line — that is a wait that always \
             passes, not a wait"
        );
        let started = Instant::now();
        let deadline = started + budget;
        let unit = self.unit();
        let bus = Systemd::connect_scoped(self.scope())?;
        // The restart count BEFORE the wait: a unit that was already restarting when
        // we arrived must not be reported as having died during our watch.
        let restarts_at_start = bus.n_restarts(&unit)?.unwrap_or(0);

        let hit = |captured: &str| -> Option<String> {
            captured
                .lines()
                .find(|l| l.contains(marker))
                .map(str::to_string)
        };

        let seen = loop {
            let captured = self.captured();
            if let Some(line) = hit(&captured) {
                break Seen::Marker {
                    line,
                    after: started.elapsed(),
                };
            }
            // "Is it gone?" has THREE shapes, and asking only the first is how a
            // crash-looping service reads as "still coming up" forever.
            //
            //  1. it came to REST — `failed`/`inactive`/not-loaded. `Restart=no` only.
            //  2. it is in the AUTO-RESTART window — `SubState == "auto-restart"`: the
            //     main process has already died and systemd is waiting out `RestartSec`.
            //     `ActiveState` says `activating` here, indistinguishable from a first
            //     start. This is also the ONLY window where `ExecMainStatus` still holds
            //     the DEAD process's code (see `Systemd::sub_state`).
            //  3. the backstop — `NRestarts` went up. Catches a death whose auto-restart
            //     window was shorter than one poll. It fires AFTER the replacement has
            //     started, so the status it reads may belong to the successor; the code
            //     is reported as `None` there rather than as a number that is a lie.
            let state = bus.active_state(&unit)?.unwrap_or_else(|| "not-loaded".into());
            let sub = bus.sub_state(&unit)?.unwrap_or_default();
            let restarted = bus.n_restarts(&unit)?.unwrap_or(0) > restarts_at_start;
            let at_rest = matches!(state.as_str(), "failed" | "inactive" | "not-loaded");
            let restarting = sub == "auto-restart";
            let dead = at_rest || restarting || restarted;
            if dead {
                // One poll's grace, then re-read: a service that prints its marker and
                // immediately exits must not be reported as having exited WITHOUT it.
                // `append:` writes go through the kernel, not through us, so this races
                // with nothing we control — the grace is the whole mitigation.
                std::thread::sleep(poll);
                let captured = self.captured();
                if let Some(line) = hit(&captured) {
                    break Seen::Marker {
                        line,
                        after: started.elapsed(),
                    };
                }
                break Seen::Exited {
                    // Only claim a code when it can still be the DEAD process's. Once
                    // the replacement has started, `ExecMainStatus` describes the
                    // replacement — MEASURED 2026-08-15: it read `Some(0)` for a service
                    // that had just exited 7. `None` is the honest answer there.
                    code: if at_rest || restarting {
                        bus.exec_main_status(&unit)?
                    } else {
                        None
                    },
                    tail: captured,
                    after: started.elapsed(),
                };
            }
            let now = Instant::now();
            if now >= deadline {
                break Seen::StillRunning {
                    tail: captured,
                    waited: started.elapsed(),
                };
            }
            std::thread::sleep(poll.min(deadline.saturating_duration_since(now)));
        };

        crate::surface_status!(
            "skidbladnir/service_drive",
            "await_marker",
            seen.saw_marker(),
            "`{}`: {}",
            unit,
            seen.detail()
        );
        Ok(seen)
    }

    /// **ASK THE SERVICE A REAL QUESTION** — the FUNC half.
    ///
    /// Runs the spec's declared [`HealthProbe`] over the wire and reports what came
    /// back. A [`HealthProbe::Banner`] service hands over its own greeting (gunnar's
    /// `SSH-2.0-gunnar_1.1.3 …`, which names the product AND the build), so the report
    /// quotes what answered rather than asserting that something did.
    ///
    /// # What each probe does NOT prove
    ///
    /// * `Tcp`/`TcpAt` — a socket is bound. Nothing about WHO bound it or whether it
    ///   can serve. This is the weakest arm and a row graded on it alone should say so.
    /// * `Banner` — the process on that port speaks the right protocol and identifies
    ///   itself. Not that it can authenticate, read its store, or serve a repository.
    /// * `Http`/`HttpAt` — one `GET` returned a 2xx status line. One route, no body
    ///   check.
    /// * `None` — nothing was asked. Reported as [`Answered::NotDeclared`], which is
    ///   never a pass.
    ///
    /// A `HealthProbe::None` spec deliberately does NOT fall through to
    /// [`Lifecycle::health`]'s `is_active` fallback here: that fallback is right for a
    /// product asking "am I up?" and wrong for a matrix asking "did a client get a
    /// correct answer?".
    pub fn answered(&self) -> Answered {
        let probe = match &self.spec().health {
            HealthProbe::None => return Answered::NotDeclared,
            HealthProbe::Tcp(p) => format!("TCP 127.0.0.1:{p}"),
            HealthProbe::TcpAt { host, port } => format!("TCP {host}:{port}"),
            HealthProbe::Http(path) => format!("HTTP GET {path}"),
            HealthProbe::HttpAt { host, path } => format!("HTTP GET {host}{path}"),
            HealthProbe::Banner { host, port, expect } => {
                format!("banner {host}:{port} expecting `{expect}`")
            }
        };
        let ok = self.life.health().unwrap_or(false);
        let out = if ok {
            Answered::Yes {
                said: self.life.banner(),
                probe,
            }
        } else {
            Answered::No { probe }
        };
        crate::surface_status!(
            "skidbladnir/service_drive",
            "answered",
            out.is_yes(),
            "`{}`: {}",
            self.unit(),
            out.detail()
        );
        out
    }

    /// **TEAR DOWN** — stop, disable, remove the unit, reload.
    ///
    /// ⚠ This is [`Lifecycle::uninstall`], and it obeys THE UNINSTALL LAW: it also
    /// **erases the declared [`ServiceSpec::data_root`]** (sparing ssh keys and
    /// tokens). That is correct for a matrix row, which wants a clean slate — and
    /// catastrophic if a driver points `data_root` at a real deployment's state. A
    /// driven spec should declare a scratch root or none at all.
    #[cfg(unix)]
    pub fn teardown(&self) -> anyhow::Result<UninstallReport> {
        let out = self.life.uninstall();
        crate::surface_status!(
            "skidbladnir/service_drive",
            "teardown",
            out.is_ok(),
            "`{}`: {}",
            self.unit(),
            match &out {
                Ok(r) => format!("unit_removed={} erased={}", r.unit_removed, r.erased.len()),
                Err(e) => format!("teardown FAILED: {e}"),
            }
        );
        out
    }

    /// **THE ONE CALL a chain driver makes for a `service` row.**
    ///
    /// install → start → await the named marker → ask a real question → (optionally)
    /// tear down. Returns a [`DriveReport`] and never an `Err`: a stage driver wants a
    /// cell, and a `?` that discards the transcript is how a red becomes "something
    /// went wrong".
    ///
    /// Teardown runs even when the wait failed — a leaked unit poisons every later
    /// row for the same product — and its own failure is reported, not swallowed.
    #[cfg(unix)]
    pub fn drive(&self, req: &DriveRequest<'_>) -> DriveReport {
        let unit = self.unit();
        let mut rep = DriveReport {
            unit: unit.clone(),
            log: self.log.clone(),
            install_error: None,
            seen: None,
            answered: Answered::NotDeclared,
            main_pid: None,
            restarts: None,
            teardown_error: None,
            notes: Vec::new(),
        };
        if !self.death_rests_in_failed() {
            rep.notes.push(format!(
                "Restart={} — this unit never rests in `failed`, so a death is caught in \
                 the SubState=auto-restart window (NRestarts as the backstop, and that leg \
                 cannot attribute an exit code)",
                self.spec().restart.as_systemd()
            ));
        }
        if let Some(root) = &self.spec().data_root {
            rep.notes.push(format!(
                "teardown ERASES the declared data_root `{root}` (THE UNINSTALL LAW)"
            ));
        }

        if let Err(e) = self.install() {
            rep.install_error = Some(format!("{e:#}"));
            return rep;
        }

        match self.await_marker(req.marker, req.budget, req.poll) {
            Ok(seen) => rep.seen = Some(seen),
            Err(e) => rep.notes.push(format!("readiness wait could not RUN: {e:#}")),
        }
        // Only ask a question of a service that is up. Probing a dead unit produces a
        // connection refusal that reads like a wrong answer.
        if rep.seen.as_ref().is_some_and(|s| !matches!(s, Seen::Exited { .. })) {
            rep.answered = self.answered();
        } else {
            rep.notes
                .push("no request was sent: the service was not up".to_string());
        }
        if let Ok(bus) = Systemd::connect_scoped(self.scope()) {
            rep.main_pid = bus.main_pid(&unit).ok().flatten();
            rep.restarts = bus.n_restarts(&unit).ok().flatten();
        }

        if req.keep_running {
            rep.notes.push(format!(
                "left RUNNING at the caller's request — `{unit}` is the caller's to stop"
            ));
        } else if let Err(e) = self.teardown() {
            rep.teardown_error = Some(format!("{e:#}"));
        }
        rep
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    fn spec() -> ServiceSpec {
        ServiceSpec::new("drive-probe", "a driven service", "/bin/true")
            .scope(Scope::User)
            .stdout_file("/tmp/never-written-by-this-test.log")
    }

    #[test]
    fn a_spec_with_no_stdout_file_is_refused_by_name() {
        // RED-when-broken: default the log path instead of refusing, and a driver
        // silently waits on a file nothing writes.
        let err = ServiceDrive::new(ServiceSpec::new("x", "x", "/bin/true")).unwrap_err();
        let msg = format!("{err}");
        assert!(msg.contains("stdout_file"), "names the field: {msg}");
        assert!(msg.contains("journal"), "names WHY: {msg}");
        assert!(msg.contains("FORKS"), "names the false green: {msg}");
    }

    #[test]
    fn the_rendered_unit_carries_the_marker_channel_on_both_streams() {
        // APPLIED OUTPUT, not a round-trip: assert the unit TEXT the one renderer
        // produced, because that text is what systemd reads.
        let d = ServiceDrive::new(spec()).unwrap();
        let text = d.unit_text();
        assert!(
            text.contains("StandardOutput=append:/tmp/never-written-by-this-test.log"),
            "stdout redirected: {text}"
        );
        assert!(
            text.contains("StandardError=append:/tmp/never-written-by-this-test.log"),
            "stderr redirected too — a service that dies says why on stderr: {text}"
        );
    }

    #[test]
    fn a_spec_without_the_field_renders_no_redirection_at_all() {
        // Additive: every unit written before this field existed must be unchanged.
        let plain = ServiceSpec::new("plain", "p", "/bin/true").scope(Scope::User);
        let text = Lifecycle::new(plain).unit_text();
        assert!(!text.contains("StandardOutput="), "no line: {text}");
        assert!(!text.contains("StandardError="), "no line: {text}");
    }

    #[test]
    fn an_empty_marker_is_refused_rather_than_matching_every_line() {
        // RED-when-broken: drop the ensure! and `""` matches the first line of any
        // output, so every row passes readiness instantly.
        let d = ServiceDrive::new(spec()).unwrap();
        let err = d
            .await_marker("   ", Duration::from_millis(1), Duration::from_millis(1))
            .unwrap_err();
        assert!(format!("{err}").contains("always passes"), "{err}");
    }

    #[test]
    fn a_spec_with_no_health_probe_reports_not_declared_never_a_pass() {
        let d = ServiceDrive::new(spec()).unwrap();
        let a = d.answered();
        assert_eq!(a, Answered::NotDeclared);
        assert!(!a.is_yes(), "an unasked question is not a passed one");
        assert!(a.detail().contains("NOTHING was asked"), "{}", a.detail());
    }

    #[test]
    fn func_needs_the_marker_AND_the_answer() {
        // Either half alone is a known false green, so the predicate must refuse both
        // one-sided shapes.
        let base = DriveReport {
            unit: "u.service".into(),
            log: PathBuf::from("/x"),
            install_error: None,
            seen: Some(Seen::Marker {
                line: "listening".into(),
                after: Duration::from_millis(5),
            }),
            answered: Answered::Yes {
                said: Some("SSH-2.0-gunnar".into()),
                probe: "banner".into(),
            },
            main_pid: Some(1),
            restarts: Some(0),
            teardown_error: None,
            notes: Vec::new(),
        };
        assert!(base.is_func(), "both halves ⇒ FUNC");

        let mut marker_only = base.clone();
        marker_only.answered = Answered::No {
            probe: "banner".into(),
        };
        assert!(!marker_only.is_func(), "printed but nothing answered");

        let mut answer_only = base.clone();
        answer_only.seen = Some(Seen::StillRunning {
            tail: "…".into(),
            waited: Duration::from_secs(1),
        });
        assert!(!answer_only.is_func(), "answered but never said it was ready");

        let mut never_installed = base.clone();
        never_installed.install_error = Some("no such binary".into());
        assert!(!never_installed.is_func());
        assert!(
            never_installed.detail().contains("never installed"),
            "{}",
            never_installed.detail()
        );
    }

    #[test]
    fn the_report_names_a_restart_policy_that_hides_a_death() {
        // NO SILENT CAPS: the latency cost of the NRestarts route is stated, by name.
        let d = ServiceDrive::new(spec().restart(RestartPolicy::Always)).unwrap();
        assert!(!d.death_rests_in_failed());
        let d2 = ServiceDrive::new(spec().restart(RestartPolicy::No)).unwrap();
        assert!(d2.death_rests_in_failed());
    }
}