deploy/upcloud/src/front.rs
//! **The front VM's services, restarted after the appliance was re-imaged** —
//! over russh, in Rust, with no `ssh` process on the path.
//!
//! # The defect this step exists to close
//!
//! MEASURED on 2026-09-08, minutes after a successful re-image: `/login/repos`
//! on `gunnar.rs` returned `[]` for a repository that is public and present.
//! Nothing was wrong with the appliance — a direct read of the same repository
//! worked, because it opened a fresh channel. What was wrong was the FRONT:
//! `gunnar-ui-server` was still holding a gRPC channel to the box that no
//! longer exists. A restart of the unit fixed it instantly.
//!
//! Three services on the front dial the appliance:
//!
//! ```text
//! 203.0.113.203 (the front, 1xCPU-1GB, serving the public site)
//! monetize-server.service 127.0.0.1:50061 -> the appliance as `monetize`
//! monetize-ui-server.service 127.0.0.1:8091 -> monetize-server
//! gunnar-ui.service 127.0.0.1:3210 -> the appliance directly
//! ```
//!
//! A re-image replaces the box all three of them are pointed at, and nothing
//! restarted them. That is not a race that closes itself: a gRPC channel to a
//! dead peer is a channel, and the front will keep answering `[]` until somebody
//! notices. So the re-image is not finished when the banner is read — it is
//! finished when the front is serving the NEW appliance, and this module is that
//! last step.
//!
//! # ⚠ Why an `ssh` subprocess is not an option and this is
//!
//! THE RULE: pure Rust, no shelling. `deploy/upcloud`'s whole existence is that
//! argument already — `xtask` spawns `cargo`, which is a BUILD TOOL opening a
//! door onto a Rust `main`. `ssh` is not a build tool. It IS the work: the
//! transport, the authentication and the remote invocation are the whole of what
//! this step does, and doing that in another program is exactly what the rule
//! refuses. `ssh2`/libssh2 would be FFI to C while a Rust library exists, which
//! the rule calls a last resort and this is not last.
//!
//! What is taken instead is [`nornir_git::ssh::connect_exec_with`] — the russh
//! dialler `gunnar-client`'s send-pack uses every day, with the connect /
//! authenticate / `exec` / blocking-I/O-bridge / in-runtime-teardown sequence
//! already written and already tested against a real russh server on loopback.
//! That teardown is not incidental: getting its `Drop` wrong panics with "there
//! is no reactor running", a bug that repo found and fixed once already.
//!
//! **What is NOT refused is running a command on the far end.** A step that
//! could not say `systemctl restart` would not be this step. systemd's own
//! remote API is D-Bus over a transport that would have to be opened anyway.
//! The line the rule draws is about which language this procedure is WRITTEN in.
//!
//! # The shape is monetize-impl's, lifted deliberately
//!
//! The private monetize-impl repository's `deploy/front/src/remote.rs` built
//! this the same night, for the same box: the [`Remote`] trait, the [`guarded`] wrapper,
//! the [`SENTINEL`], `active_gate`/`http_gate`. What is REUSED across the fence
//! is the library — `nornir_git::ssh` — and what is restated is sixty lines of
//! shape, on purpose:
//!
//! * gunnar is the OPEN product and monetize-impl is private. A path dependency
//! from here to there would make every gunnar build require a checkout of a
//! private repository, which is not a cost the open build may carry.
//! * The two verbs also do different work. That one BUILDS and SWAPS binaries
//! and needs `put`, digests and a release bay; this one restarts three units
//! that are already installed and needs `exec` and nothing else.
//!
//! The twin is named here so that a third caller is the moment this moves into
//! edda as a leaf, next to `nornir-hash` — the same note `banner.rs` carries
//! about `installtest`'s `ssh_banner`.
//!
//! # Verify by OUTPUT, never by exit code
//!
//! [`nornir_git::ssh::SshChannel`] hands over the channel's bytes and not the
//! remote command's exit status — it was built to carry a git pack. That
//! constraint is taken as the design, because the exit of `systemctl restart` is
//! not evidence of anything anyway: it reports that systemd accepted the job.
//!
//! So every command is wrapped by [`guarded`]:
//!
//! ```text
//! timeout 60 sh -c 'systemctl restart gunnar-ui' 2>&1 && echo __FRONT_RESTART_OK__
//! ```
//!
//! `&&` makes the sentinel evidence of success rather than of running;
//! `timeout` is the bounded wait enforced by the far end, because the channel
//! has no read timeout and a `systemctl restart` on a stuck unit would otherwise
//! block this verb forever; `2>&1` is not cosmetic, because SSH carries stderr
//! as EXTENDED data on a stream the blocking bridge does not read — without it
//! the one thing an operator needs when a step fails is the one thing that never
//! arrives.
//!
//! And a restart that *ran* still proves nothing about the service. Two more
//! readings, per unit, both from the box's own mouth:
//!
//! * `systemctl is-active` says `active` — and equality, never `contains`:
//! `inactive` and `activating` both contain `active`.
//! * the port answers `/auth/status`. **A unit that is `active` is not a unit
//! that works**: systemd reports a process that has not exited yet, and a
//! server that binds nothing and logs a config error is `active` for as long
//! as it takes somebody to notice. This is the exact failure that made the
//! step necessary, one layer up.
use std::io::Read as _;
use std::path::{Path, PathBuf};
use crate::wait::{wait_for, Budget, Clock, Progress};
/// Printed by a command that actually succeeded, and by nothing else.
pub const SENTINEL: &str = "__FRONT_RESTART_OK__";
// The front's ADDRESS is a site fact and lives nowhere in this source tree:
// see [`crate::site`]. It arrives as `--front [user@]host[:port]` or from the
// site file, and `USER`/`PORT` below are only the ordinary ssh defaults for the
// halves of that spec an operator left out.
pub use crate::site::{DEFAULT_FRONT_PORT as PORT, DEFAULT_FRONT_USER as USER};
/// One unit on the front, and how to tell whether it came back.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Unit {
pub name: &'static str,
/// The loopback port whose `/auth/status` must answer afterwards. `None` for
/// `monetize-server`, which speaks gRPC on 50061 and has no such door —
/// `is-active` plus the fact that `monetize-ui-server` (which DIALS it)
/// answers is what covers it, and inventing an HTTP probe for a gRPC port
/// would be a check that fails for the wrong reason.
pub probe: Option<u16>,
}
/// **The three services that dial the appliance, in RESTART ORDER.**
///
/// Backends first: `monetize-ui-server` dials `monetize-server`, so a console
/// restarted before its backend is a console whose first request fails. `caddy`
/// is deliberately absent — it terminates TLS and proxies, holds no channel to
/// the appliance, and restarting it would take the site down for the one thing
/// that cannot be stale.
pub const UNITS: &[Unit] = &[
Unit {
name: "monetize-server",
probe: None,
},
Unit {
name: "monetize-ui-server",
probe: Some(8091),
},
Unit {
name: "gunnar-ui",
probe: Some(3210),
},
];
/// Where to reach the front, and how long its units may take.
#[derive(Clone, Debug)]
pub struct Front {
pub host: String,
pub port: u16,
pub user: String,
pub key: PathBuf,
/// How long one `systemctl` call may take, at the far end.
pub command_s: u64,
/// How long to keep asking a restarted unit whether it is up. A unit
/// restarted a second ago has not bound its port yet, so the first `000`
/// from curl is expected and is not a verdict.
pub settle: Budget,
}
impl Front {
/// Build the front from the operator's `[user@]host[:port]` and a key path.
/// There is no `Default`: a default front is somebody's live box, and this
/// crate does not know whose. See [`crate::site`].
pub fn from_spec(spec: &crate::site::FrontSpec, key: PathBuf) -> Front {
Front {
host: spec.host.clone(),
port: spec.port,
user: spec.user.clone(),
key,
command_s: 60,
settle: Budget {
timeout_ms: 90_000,
poll_ms: 3_000,
report_every_ms: 30_000,
},
}
}
}
/// `~/.ssh/id_ed25519`, the ordinary key path when `--front-key` says nothing.
/// A key path is not a site fact — it names a file on the operator's own box.
pub fn default_key() -> PathBuf {
let home = std::env::var("HOME").unwrap_or_else(|_| "/root".to_string());
PathBuf::from(home).join(".ssh/id_ed25519")
}
/// What happened to the front, for the summary and the receipt.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum FrontRefresh {
/// `--no-front-restart`. A skip that is PRINTED and RECORDED, never a
/// silence — the front is now pointed at a box that no longer exists, and
/// the operator has to be told which command they still owe.
Skipped { manual: String },
/// Every unit restarted, reported `active`, and answered on its port.
Done {
/// unit → what `systemctl is-active` said.
restarted: Vec<(String, String)>,
/// url → the HTTP status curl read off it.
answered: Vec<(String, String)>,
},
/// The appliance is serving and the front is NOT refreshed. Carried in the
/// [`crate::plan::Outcome`] rather than returned as an error so the receipt
/// still records the re-image that DID work; the verb then exits non-zero
/// on it, so nothing reads as green.
Failed { why: String, manual: String },
}
/// The command a human runs when this step was skipped or failed.
///
/// Spelled out in full, with the host, because "restart the front" at 03:00 is
/// not an instruction.
pub fn manual_command(front: &Front) -> String {
format!(
"ssh {}@{} 'systemctl restart {}' && ssh {}@{} 'systemctl is-active {}'",
front.user,
front.host,
UNITS.iter().map(|u| u.name).collect::<Vec<_>>().join(" "),
front.user,
front.host,
UNITS.iter().map(|u| u.name).collect::<Vec<_>>().join(" "),
)
}
// ── the transport ───────────────────────────────────────────────────────────
/// The far end, injectable — so this whole step runs in a test with no box, the
/// way `upcloud_api::UpCloudApi` does for the UpCloud API.
pub trait Remote {
/// Run `command` and return everything it wrote (stdout and, thanks to the
/// `2>&1` that [`guarded`] adds, stderr).
fn exec(&mut self, command: &str) -> Result<String, String>;
}
/// Wrap a command so its success is visible in the OUTPUT and its duration is
/// bounded by the far end. See the module header for all three pieces.
pub fn guarded(command: &str, budget_s: u64) -> String {
format!(
"timeout {budget_s} sh -c {} 2>&1 && echo {SENTINEL}",
shell_quote(command)
)
}
/// Single-quote a command for `sh -c`, the only quoting this module does.
///
/// Every command string here is built from [`UNITS`], which is a constant, so
/// there is no untrusted input to escape. A quoting function that exists is one
/// that cannot be forgotten the day something else is interpolated.
pub fn shell_quote(s: &str) -> String {
format!("'{}'", s.replace('\'', "'\\''"))
}
/// Run one command and insist on the evidence.
pub fn run(r: &mut dyn Remote, what: &str, command: &str, budget_s: u64) -> Result<String, String> {
let out = r
.exec(&guarded(command, budget_s))
.map_err(|e| format!("{what}: {e}"))?;
if !out.contains(SENTINEL) {
return Err(format!(
"{what} DID NOT SUCCEED — `{command}` never printed the success marker, so either \
it failed or it ran past its {budget_s} s budget. It said:\n{}",
indent(out.trim())
));
}
Ok(out.replace(SENTINEL, "").trim().to_string())
}
fn indent(s: &str) -> String {
if s.is_empty() {
return " (nothing at all, which usually means the timeout fired)".to_string();
}
s.lines()
.map(|l| format!(" {l}"))
.collect::<Vec<_>>()
.join("\n")
}
// ── the two readings ────────────────────────────────────────────────────────
/// `systemctl is-active <unit>` said `active`, and nothing else will do.
///
/// Equality, because `inactive` CONTAINS `active` and so does `activating`.
pub fn active_gate(unit: &str, is_active_output: &str) -> Result<(), String> {
let seen = is_active_output.trim();
if seen == "active" {
return Ok(());
}
Err(format!(
"{unit}.service is {seen:?}, not \"active\" — `journalctl -u {unit} -n 50 --no-pager` on \
the front says why"
))
}
/// The loopback port answered.
///
/// `code` is what `curl -o /dev/null -w '%{http_code}'` printed. `000` is
/// curl's "no reply at all", which is exactly what a unit that is `active` but
/// listening on nothing produces — the case [`active_gate`] structurally cannot
/// see.
pub fn http_gate(what: &str, url: &str, code: &str) -> Result<(), String> {
let code = code.trim();
match code.parse::<u16>() {
Ok(0) | Err(_) => Err(format!(
"{what}: nothing answered {url} (curl said {code:?}). The unit can be `active` and \
still be listening on nothing — a bind failure is logged, not exited"
)),
// 401/403 are ANSWERS: `/auth/status` on an unauthenticated request is
// allowed to refuse, and a refusal proves the server is serving. 5xx is
// not — it is the process admitting it cannot do its job.
Ok(n) if (200..500).contains(&n) => Ok(()),
Ok(n) => Err(format!(
"{what}: {url} answered {n}, which is the server saying it is broken rather than the \
port saying it is closed"
)),
}
}
// ── the step ────────────────────────────────────────────────────────────────
/// **Restart the front's three services and prove each one came back.**
///
/// Called AFTER the appliance's banner has been verified, and only then: a
/// front pointed at a box that has not proved it is serving would be restarted
/// onto the same dead channel it started with.
pub fn refresh<C: Clock>(
r: &mut dyn Remote,
clock: &C,
front: &Front,
log: &mut dyn FnMut(&str),
) -> Result<FrontRefresh, String> {
let mut restarted = Vec::new();
let mut answered = Vec::new();
for unit in UNITS {
log(&format!(" restarting {} on the front", unit.name));
run(
r,
&format!("restarting {}", unit.name),
&format!("systemctl restart {}", unit.name),
front.command_s,
)?;
}
// Both readings AFTER all three restarts, not interleaved: `monetize-server`
// going down takes `monetize-ui-server`'s backend with it for a moment, and
// asking the second one whether it is healthy while the third is still being
// restarted measures the restart order rather than the result.
for unit in UNITS {
let mut seen = String::new();
wait_for(
clock,
&format!("{} to report active", unit.name),
front.settle,
log,
|| {
let out = run(
r,
&format!("asking whether {} is active", unit.name),
&format!("systemctl is-active {} 2>/dev/null || true", unit.name),
15,
)?;
match active_gate(unit.name, &out) {
Ok(()) => {
seen = out.trim().to_string();
Ok(Progress::Done)
}
Err(e) => Ok(Progress::Waiting(e)),
}
},
)?;
log(&format!(" {} is {seen}", unit.name));
restarted.push((unit.name.to_string(), seen));
let Some(port) = unit.probe else { continue };
let url = format!("http://127.0.0.1:{port}/auth/status");
let mut code = String::new();
wait_for(
clock,
&format!("{} to answer {url}", unit.name),
front.settle,
log,
|| {
let out = run(
r,
&format!("probing {url}"),
&format!("curl -s -o /dev/null -w '%{{http_code}}' -m 5 {url} || echo 000"),
15,
)?;
match http_gate(unit.name, &url, &out) {
Ok(()) => {
code = out.trim().to_string();
Ok(Progress::Done)
}
Err(e) => Ok(Progress::Waiting(e)),
}
},
)?;
log(&format!(" {url} answers {code}"));
answered.push((url, code));
}
Ok(FrontRefresh::Done {
restarted,
answered,
})
}
// ── the real one ────────────────────────────────────────────────────────────
/// The front VM, over russh. See the module header for why this is not `ssh`.
pub struct Ssh {
pub loc: nornir_git::ssh::SshLocation,
pub auth: nornir_git::ssh::SshAuth,
}
impl Ssh {
/// `root@<the front>` with the operator's key — both from [`crate::site`].
///
/// [`nornir_git::ssh::HostKeyPolicy::TofuKnownHosts`] rather than the
/// `AcceptAny` that `SshAuth::key` selects for git deploy keys: this is a
/// named public address whose key `known_hosts` already carries, and a
/// CHANGED host key on the box serving `gunnar.rs` is the one event that
/// must stop this step dead rather than type a command into whatever
/// answered.
pub fn front(host: &str, port: u16, user: &str, key: &Path) -> Ssh {
use nornir_git::ssh::{HostKeyPolicy, SshAuth, SshLocation};
Ssh {
loc: SshLocation {
user: user.to_string(),
host: host.to_string(),
port,
path: String::new(),
},
auth: SshAuth::key(key).with_host_key_policy(HostKeyPolicy::TofuKnownHosts),
}
}
}
impl Remote for Ssh {
fn exec(&mut self, command: &str) -> Result<String, String> {
// `&[]` for the environment: `connect_exec_auth` sends
// `GIT_PROTOCOL=version=2` before the `exec`, which is right for
// `git-upload-pack` and meaningless noise on `systemctl`.
let mut ch = nornir_git::ssh::connect_exec_with(&self.loc, &self.auth, command, &[])
.map_err(|e| format!("ssh {}@{}: {e:#}", self.loc.user, self.loc.host))?;
let (reader, _writer) = ch.io();
let mut buf = Vec::new();
reader
.read_to_end(&mut buf)
.map_err(|e| format!("reading the reply to `{command}`: {e}"))?;
Ok(String::from_utf8_lossy(&buf).to_string())
}
}
// ── the fake ────────────────────────────────────────────────────────────────
/// A scripted front: canned replies, and a recording of every command.
///
/// The recording IS the assertion about the order — the same technique
/// the scripted UpCloud world in `tests/` uses for the API, and for the same
/// reason: the ORDER is the part that cannot be checked by reading, and it is
/// the part a refactor breaks.
#[derive(Default)]
pub struct ScriptedRemote {
/// Replies, matched by substring against the command. First match wins;
/// anything unmatched gets the sentinel and nothing else.
pub replies: Vec<(String, String)>,
/// Commands that must FAIL — matched by substring, answered with no
/// sentinel, which is what a real failure looks like on this channel.
pub failures: Vec<(String, String)>,
pub commands: Vec<String>,
}
impl ScriptedRemote {
/// A healthy front: every unit `active`, every port answering 200.
pub fn healthy() -> ScriptedRemote {
ScriptedRemote::default()
.reply("is-active", "active")
.reply("http_code", "200")
}
/// A canned reply. **Later calls win**: the rules go to the FRONT, so a test
/// can start from [`Self::healthy`] and override one answer.
pub fn reply(mut self, matching: &str, with: &str) -> Self {
self.replies
.insert(0, (matching.to_string(), with.to_string()));
self
}
/// A command that fails: answered without a sentinel. Also front-inserted.
pub fn fails(mut self, matching: &str, saying: &str) -> Self {
self.failures
.insert(0, (matching.to_string(), saying.to_string()));
self
}
/// Did a command containing `needle` run?
pub fn ran(&self, needle: &str) -> bool {
self.commands.iter().any(|c| c.contains(needle))
}
/// The index of the first command containing `needle`, for ordering
/// assertions.
pub fn at(&self, needle: &str) -> Option<usize> {
self.commands.iter().position(|c| c.contains(needle))
}
}
impl Remote for ScriptedRemote {
fn exec(&mut self, command: &str) -> Result<String, String> {
self.commands.push(command.to_string());
for (needle, saying) in &self.failures {
if command.contains(needle.as_str()) {
return Ok(saying.clone());
}
}
for (needle, with) in &self.replies {
if command.contains(needle.as_str()) {
return Ok(format!("{with}\n{SENTINEL}\n"));
}
}
Ok(format!("{SENTINEL}\n"))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::wait::FakeClock;
fn silent() -> impl FnMut(&str) {
|_: &str| {}
}
/// A front that is nobody's: RFC 5737 documentation space, and a key path
/// that cannot be the operator's own.
fn front() -> Front {
Front::from_spec(
&crate::site::FrontSpec::parse("root@203.0.113.203").unwrap(),
PathBuf::from("/dev/null"),
)
}
/// The three units, in the order that does not break `monetize-ui-server`,
/// and both readings for each one that has a port.
#[test]
fn all_three_are_restarted_backend_first_and_each_one_is_read_back() {
let mut r = ScriptedRemote::healthy();
let out = refresh(&mut r, &FakeClock::new(), &front(), &mut silent()).unwrap();
let money = r.at("restart monetize-server").unwrap();
let money_ui = r.at("restart monetize-ui-server").unwrap();
let console = r.at("restart gunnar-ui").unwrap();
assert!(
money < money_ui && money_ui < console,
"a console restarted before its backend logs a connection failure on its first \
request: {:?}",
r.commands
);
match out {
FrontRefresh::Done {
restarted,
answered,
} => {
assert_eq!(restarted.len(), 3);
assert!(restarted.iter().all(|(_, seen)| seen == "active"));
assert_eq!(
answered,
vec![
(
"http://127.0.0.1:8091/auth/status".to_string(),
"200".into()
),
(
"http://127.0.0.1:3210/auth/status".to_string(),
"200".into()
),
],
"the two doors the front actually serves"
);
}
other => panic!("{other:?}"),
}
}
/// RED FIRST, MEASURED with the sentinel check taken out of [`run`] so it
/// returns whatever came back:
///
/// ```text
/// called `Result::unwrap_err()` on an `Ok` value: Done { restarted:
/// [("monetize-server", "active"), ("monetize-ui-server", "active"),
/// ("gunnar-ui", "active")], answered:
/// [("http://127.0.0.1:8091/auth/status", "200"),
/// ("http://127.0.0.1:3210/auth/status", "200")] }
/// ```
///
/// A fully green front refresh, out of a run in which `systemctl restart
/// gunnar-ui` printed `Job for gunnar-ui.service failed`. The SSH channel
/// gives bytes and not an exit status, so without the sentinel a failed
/// restart is indistinguishable from a successful one.
#[test]
fn a_restart_whose_sentinel_never_arrived_is_a_failure_that_quotes_the_output() {
let mut r = ScriptedRemote::healthy().fails(
"restart gunnar-ui",
"Job for gunnar-ui.service failed. See 'systemctl status gunnar-ui'.",
);
let err = refresh(&mut r, &FakeClock::new(), &front(), &mut silent()).unwrap_err();
assert!(err.contains("restarting gunnar-ui"), "{err}");
assert!(err.contains("DID NOT SUCCEED"), "{err}");
assert!(err.contains("Job for gunnar-ui.service failed"), "{err}");
}
/// **The failure the whole step exists for**: the unit is up and the port is
/// dead. `is-active` cannot see it, so the HTTP reading is not a belt on a
/// pair of braces — it is the only one of the two that can fail here.
#[test]
fn a_unit_that_is_active_but_answers_nothing_is_a_failure() {
let mut r = ScriptedRemote::healthy().reply("http_code", "000");
let err = refresh(&mut r, &FakeClock::new(), &front(), &mut silent()).unwrap_err();
assert!(
err.contains("3210/auth/status") || err.contains("8091/auth/status"),
"{err}"
);
assert!(err.contains("listening on nothing"), "{err}");
}
/// RED FIRST, MEASURED with `active_gate` written as
/// `seen.contains("active")`:
///
/// ```text
/// called `Result::unwrap_err()` on an `Ok` value: ()
/// ```
///
/// on the first bad word in the list, which is `inactive`. `activating`
/// passes the same way. This is the substring bug in its natural habitat:
/// a dead unit reported as a live one, by a gate that reads correctly.
#[test]
fn inactive_contains_active_and_that_is_why_the_gate_is_equality() {
assert!(active_gate("gunnar-ui", "active\n").is_ok());
for bad in ["inactive", "activating", "failed", ""] {
let err = active_gate("gunnar-ui", bad).unwrap_err();
assert!(err.contains("journalctl"), "{err}");
}
}
/// A refusal is an answer: `/auth/status` unauthenticated is allowed to say
/// 401, and a 401 proves the server is serving. A 500 does not.
#[test]
fn a_refusal_proves_the_port_is_serving_and_a_five_hundred_does_not() {
let url = "http://127.0.0.1:3210/auth/status";
assert!(http_gate("gunnar-ui", url, "401").is_ok());
assert!(http_gate("gunnar-ui", url, "200").is_ok());
assert!(http_gate("gunnar-ui", url, "000").is_err());
assert!(http_gate("gunnar-ui", url, "").is_err());
let err = http_gate("gunnar-ui", url, "502").unwrap_err();
assert!(err.contains("saying it is broken"), "{err}");
}
/// A port that is not up on the first ask is not a verdict: a unit restarted
/// three seconds ago has not bound yet. The wait is bounded, and its end IS
/// the verdict.
#[test]
fn a_port_that_comes_up_on_the_third_ask_is_a_pass_not_a_failure() {
struct Settling {
asks: u32,
}
impl Remote for Settling {
fn exec(&mut self, command: &str) -> Result<String, String> {
if command.contains("http_code") {
self.asks += 1;
let code = if self.asks >= 3 { "200" } else { "000" };
return Ok(format!("{code}\n{SENTINEL}\n"));
}
if command.contains("is-active") {
return Ok(format!("active\n{SENTINEL}\n"));
}
Ok(format!("{SENTINEL}\n"))
}
}
let mut r = Settling { asks: 0 };
refresh(&mut r, &FakeClock::new(), &front(), &mut silent()).unwrap();
assert!(r.asks >= 3);
}
#[test]
fn a_bounded_command_says_how_long_it_may_take_and_brings_stderr_home() {
let g = guarded("systemctl restart gunnar-ui", 60);
assert!(g.contains("timeout 60"), "{g}");
assert!(g.contains("&& echo __FRONT_RESTART_OK__"), "{g}");
assert!(
g.contains("2>&1"),
"stderr rides a different SSH stream than the bridge reads: {g}"
);
}
/// A command that produced nothing at all timed out, and the message has to
/// say so instead of printing an empty block.
///
/// RED FIRST, same edit as above — with no sentinel check, silence is a
/// PASS: `called `Result::unwrap_err()` on an `Ok` value: ""`. That is a
/// front whose `systemctl restart` hung for its whole budget, reported as
/// refreshed.
#[test]
fn a_command_that_timed_out_silently_still_names_its_budget() {
struct Silent;
impl Remote for Silent {
fn exec(&mut self, _c: &str) -> Result<String, String> {
Ok(String::new())
}
}
let err = run(
&mut Silent,
"restarting gunnar-ui",
"systemctl restart gunnar-ui",
30,
)
.unwrap_err();
assert!(err.contains("30 s budget"), "{err}");
assert!(err.contains("the timeout fired"), "{err}");
}
/// The manual command names the host and every unit, because "restart the
/// front" is not an instruction at 03:00.
#[test]
fn the_manual_command_is_one_a_human_can_paste() {
let m = manual_command(&front());
assert!(m.contains("root@203.0.113.203"), "{m}");
for u in UNITS {
assert!(m.contains(u.name), "{} missing from {m}", u.name);
}
}
}