crates/gunnar-server/tests/appliance_powercut_recovery.rs
//! **A power cut after repo writes must not wedge the appliance.**
//!
//! MEASURED 2026-08-31 on the appliance-shaped KVM (4 vCPU / 16 GiB / 4×75 GiB
//! raid0-xfs /data): a power cut after a create/push/delete/purge sequence
//! wedged gunnar — PID 1 — on the next boot with
//! `Kernel panic - not syncing: Attempted to kill init! exitcode=0x00006500`.
//! The Rust panic captured off ttyS0 was NOT in the store-open path the entry
//! first suspected; it was
//!
//! ```text
//! panicked at tokio/src/runtime/blocking/shutdown.rs:51:21:
//! Cannot drop a runtime in a context where blocking is not allowed.
//! ```
//!
//! The chain, every link confirmed on the crashed disk image:
//!
//! 1. `ReplicationConfig::save` wrote `replication.json` and renamed it with no
//! fsync, so the power cut left it **0 bytes** on xfs (rename journaled, data
//! blocks not). The image showed exactly a 0-byte `replication.json` beside
//! an intact `control.redb` and the pushed repo.
//! 2. On the next boot `ReplicationConfig::load` did `serde_json::from_str("")`,
//! which fails, and the `?` in `Server::start` unwound the async body.
//! 3. `start` had already built the control plane's own tokio `Runtime` as a
//! bare local; unwinding dropped it on the async stack, which blocks, which
//! tokio turns into the panic above — and as PID 1 that is the kernel panic.
//!
//! Both halves are guarded here, each seen RED against the pre-fix tree:
//!
//! * the 0-byte artifact must let the box **come up serving** (so it can rejoin
//! the pair as a standby), not refuse to boot on it — RED before
//! `ReplicationConfig::load` tolerated an empty file (the process aborted);
//! * a genuinely malformed config is still a hard error, but a **returned** one,
//! never a `Runtime`-drop abort — RED before the `ControlRuntime` guard (the
//! process died with the tokio panic instead of returning `Err`).
//!
//! Only built with `replication`, which is the appliance's own configuration
//! (gunnar-cli's `server` feature pulls `gunnar-server/replication`): the
//! vulnerable `?` and the serve-anyway recovery both live behind it.
#![cfg(feature = "replication")]
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::Arc;
use gunnar_server::mirror::REPLICATION_CONFIG;
use gunnar_server::{ServeConfig, Server, ServerStores};
/// A fresh, unique data root under the OS temp dir.
fn data_root(tag: &str) -> std::path::PathBuf {
static N: AtomicU32 = AtomicU32::new(0);
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos();
let dir = std::env::temp_dir().join(format!(
"gunnar-s083-{tag}-{}-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed),
nanos
));
std::fs::create_dir_all(&dir).unwrap();
dir
}
/// **The reframe's GREEN: a 0-byte `replication.json` boots and serves.**
///
/// The appliance is deployed as a pair; a power-cut box loses no data (the
/// standby holds it), so what the fix must give is that the box can REBOOT and
/// rejoin — cheaply, on its own — instead of dying and needing a reinstall. So
/// the assertion is applied output: `Server::start` returns a server that has
/// actually bound its SSH and gRPC listeners on the exact artifact the crash
/// left behind.
///
/// # 🔴 Seen red
///
/// Against the pre-fix tree this did not fail — it **aborted the test process**
/// with tokio's "Cannot drop a runtime in a context where blocking is not
/// allowed", the same panic ttyS0 showed. With `ReplicationConfig::load`
/// tolerating the empty file it comes back `Ok` and both listeners are up.
#[test]
fn a_powercut_zero_byte_replication_json_boots_and_serves() {
let root = data_root("empty-repl");
// The artifact itself, byte for byte: 0 length.
std::fs::write(root.join(REPLICATION_CONFIG), b"").unwrap();
let stores = Arc::new(ServerStores::open(&root).expect("open stores"));
let config = ServeConfig::ephemeral();
// The appliance runs `start` under exactly this multi-thread runtime via
// `block_on`; that async context is what turns a stray `Runtime` drop into
// the PID-1 abort, so the recovery has to be proven HERE and not on a
// current-thread runtime that would never reproduce it.
let rt = gunnar_server::serve::runtime().expect("runtime");
let server = rt
.block_on(async move { Server::start(stores, config).await })
.expect(
"a 0-byte replication.json (the power-cut artifact) must not wedge the boot: \
the box has to come up serving so it can rejoin the pair as a standby",
);
// Serving, not merely constructed.
let ssh = server.ssh_addr();
assert_ne!(ssh.port(), 0, "the SSH listener did not bind a real port");
let grpc = server
.grpc_endpoint()
.expect("the gRPC control plane did not come up on the recovered box");
assert_ne!(
grpc.0.port(),
0,
"the gRPC listener did not bind a real port"
);
rt.block_on(server.shutdown()).expect("clean shutdown");
let _ = std::fs::remove_dir_all(&root);
}
/// **The `ControlRuntime` guard: a malformed config is a returned `Err`, never a
/// PID-1 abort.**
///
/// A non-empty malformed `replication.json` is an operator's typo and stays a
/// hard startup error (an appliance won't come up on it — but that is a wrong
/// human input, not a power cut). What must never happen is that the hard error
/// aborts the *process* by dropping the control runtime on the async stack.
///
/// # 🔴 Seen red
///
/// Before the guard, reaching this test's assertions was impossible: the `?` in
/// `Server::start` dropped the bare `Runtime` and the process died with the
/// tokio blocking-drop panic. The guard routes that drop through
/// `shutdown_background`, so `start` returns `Err` and control comes back here.
#[test]
fn a_malformed_replication_json_is_a_clean_error_not_a_pid1_abort() {
let root = data_root("bad-repl");
std::fs::write(root.join(REPLICATION_CONFIG), b"{ this is not json").unwrap();
let stores = Arc::new(ServerStores::open(&root).expect("open stores"));
let config = ServeConfig::ephemeral();
let rt = gunnar_server::serve::runtime().expect("runtime");
let outcome = rt.block_on(async move { Server::start(stores, config).await });
// Reaching this line at all is the guard: the process did not abort dropping
// its control runtime inside `block_on`.
assert!(
outcome.is_err(),
"a non-empty malformed replication.json must be a clean, returned startup error"
);
let _ = std::fs::remove_dir_all(&root);
}