crates/gunnar-server/src/clock.rs
//! **The appliance's wall clock**: glue between PID 1 and edda `estate-clock`.
//!
//! Every rule — the sources, the quorum, the sanity window, the signed-peer
//! protocol, step-at-boot and slew-while-running, the first fed syncs and the
//! running triggers — lives in edda's `estate-clock` crate (where Gunnar's own
//! `gunnar-clock` moved, lane L10 2026-10-21), which the front's gunnar-front
//! links too. This file decides only *whether* this process owns the clock,
//! *where* its feed is, and *what it says* about the answer.
//!
//! **Both wires during the move**: a box asks in `ESTATE-TIME/1` and asks again
//! in `GUNNAR-TIME/1` when an old front answers in Gunnar's dialect
//! (`estate_clock::peer::query`); `GUNNAR_CLOCK*` (the kit's
//! `korp.env.GUNNAR_CLOCK=`) is read as an alias of `ESTATE_CLOCK*`
//! (`estate_clock::config::var`).
//!
//! # What went wrong with the first cut (gunnar `5792dda`), measured
//!
//! The appliance booted two hours fast; `5792dda` asked three internet HTTPS
//! hosts for a `Date` at boot and was re-imaged on 2026-09-14 — and the box
//! still refused credentials `7199529 ms away`. Through the control plane, with
//! a credential deliberately signed 7 199 500 ms fast so the gate let it
//! through, `gunnar control mirror add … https://github.com/…` answered
//! `github.com could not be resolved: failed to lookup address information: Try
//! again`. UpCloud's firewall catch-all `drop` drops the replies to outbound
//! UDP (from the front: `dig +notcp` times out, `dig +tcp` answers), musl's
//! resolver asks over UDP only, so every source was unreachable, the verdict was
//! "no quorum", and the clock was left alone. The only record was a log line on a
//! VGA console. See the `estate-clock` crate note for the full measurement.
//!
//! # What it does now
//!
//! * **The source is configuration** (`korp.env.GUNNAR_CLOCK=` on the kernel
//! command line). On gunnar.rs that is `peer://10.13.8.99:50123` — the front,
//! by IP literal over the utility network, answering a nonce with a signed
//! timestamp (`GUNNAR_CLOCK_KEY` names its public key). No internet, no DNS.
//! Unset means **not configured**, and nothing is asked: an appliance in an
//! airgap has no business dialling Stripe for the time.
//! * **At boot**, after the network and before any listener: one sync, steps.
//! * **While running**, only when a credential is refused for skew (rate-limited
//! to one sync a minute) and on a 6-hour safety timer — `estate_clock::trigger`.
//! Small offsets slew, large ones step — `estate_clock::adjust`.
//! * **Every attempt is recorded** (`estate_clock::report`) and served on the
//! control plane: `Admin.Health` carries the boot and the last report, and
//! `Bootstrap.Status` — the one call that needs no credential — carries the
//! last one and this server's own clock. When the clock is wrong a credential
//! is exactly what gets refused, so the credential-free call is the one that
//! makes a wrong clock diagnosable from the front.
//!
//! # Why it can never stop the boot or the server
//!
//! Every function here returns `()` and swallows everything. A forge that
//! refuses to come up because a clock fetch failed is strictly worse than a
//! forge with a wrong clock: the wrong clock refuses credentials, the dead
//! forge refuses everything — including the call an operator uses to find out
//! why. The periodic sync runs on the blocking pool, never on a request path.
use estate_clock::config::{Config, CLOCK_ENV};
use estate_clock::offer::FedRun;
use estate_clock::report::{Outcome, Report, Trigger};
pub use estate_clock::offer::{last_outcome, outcome_is_right};
/// **One sync** (AIRGAP, owner rule 2026-10-25). With `clock-fetch` it is estate-clock's own,
/// HTTPS Date sources included. Without it (the default build) the same rule over peer and
/// SNTP sources only: an `https://` source is refused by [`NO_HTTPS`] and never dialled.
#[cfg(feature = "clock-fetch")]
fn sync(config: &Result<Config, String>, phase: Trigger) -> Report {
estate_clock::fetch::sync(config, phase)
}
/// The refusal an `https://` clock source gets in a build without `clock-fetch`.
pub const NO_HTTPS: &str =
"an https:// clock source needs gunnar-server's `clock-fetch` feature (off by default: this build asks peer and SNTP sources only)";
#[cfg(not(feature = "clock-fetch"))]
fn sync(config: &Result<Config, String>, phase: Trigger) -> Report {
let report = match config {
Ok(Config::Sources { sources, peer_key }) => {
let polled = estate_clock::poll::poll_with(sources, peer_key.as_deref(), &|_| {
Err(NO_HTTPS.to_owned())
});
let action = estate_clock::adjust::choose(phase, &polled.verdict.decision);
let applied = estate_clock::adjust::apply(action).map_err(|err| err.to_string());
Report::from_poll(phase, &polled, action, applied)
}
Err(why) => Report::without_sources(phase, Outcome::Misconfigured, why.clone()),
Ok(Config::NotConfigured) => Report::without_sources(
phase,
Outcome::NotConfigured,
format!("no clock source is configured ({CLOCK_ENV})"),
),
Ok(Config::Off) => Report::without_sources(
phase,
Outcome::Off,
format!("{CLOCK_ENV}=off: this machine was told not to set its own clock"),
),
};
estate_clock::report::record(report.clone());
report
}
/// Whether this process may set the machine's clock at all.
///
/// Gated exactly as `crate::disktest` and `crate::dashboard` are — PID 1 **and**
/// `korp.entry=` on the kernel command line. A container entrypoint is PID 1 of
/// its namespace too, and a privileged container shares the host's clock:
/// stepping it there would move somebody's real machine. Naming sources in
/// [`CLOCK_ENV`] (or its Gunnar alias `GUNNAR_CLOCK`) explicitly turns it on
/// regardless, which is what the installtest rig and a developer reproducing a
/// refusal need; the syscall still needs `CAP_SYS_TIME` and a refusal is still
/// only a report.
fn owns_the_clock() -> bool {
std::env::var_os(CLOCK_ENV).is_some()
|| estate_clock::config::gunnar_alias(CLOCK_ENV)
.is_some_and(|alias| std::env::var_os(alias).is_some())
|| (appliance_pid1::net::is_init() && appliance_pid1::net::booted_by_korp_start())
}
/// Say what an attempt concluded, at the level it deserves.
fn log(report: &Report) {
let summary = report.summary();
match report.outcome {
Outcome::Stepped | Outcome::Slewed | Outcome::AlreadyRight => {
tracing::info!(
outcome = report.outcome.as_str(),
offset_ms = report.offset_ms,
"clock: {summary}"
)
}
Outcome::NotConfigured | Outcome::Off => {
tracing::info!(outcome = report.outcome.as_str(), "clock: {summary}")
}
Outcome::Misconfigured | Outcome::NoQuorum | Outcome::Disagree | Outcome::ApplyFailed => {
tracing::warn!(
outcome = report.outcome.as_str(),
"clock: {summary}. If this box's clock is wrong, every credential it is handed \
will be refused."
)
}
}
for row in &report.sources {
match (&row.offset_ms, &row.error) {
(_, Some(why)) => tracing::warn!(source = %row.source, %why, "clock source failed"),
(Some(offset_ms), None) => {
tracing::info!(source = %row.source, offset_ms, "clock source answered")
}
(None, None) => {}
}
}
}
/// One attempt on the blocking pool; the report is recorded by `sync` itself.
async fn attempt(phase: Trigger) -> Report {
let config = crate::feed::clock_config();
match tokio::task::spawn_blocking(move || sync(&config, phase)).await {
Ok(report) => report,
Err(err) => {
let report = Report::without_sources(
phase,
Outcome::ApplyFailed,
format!("the clock sync task did not complete: {err}"),
);
estate_clock::report::record(report.clone());
report
}
}
}
/// **The boot call.** Sync once, stepping if the source says so.
pub async fn synchronise_at_boot() {
if !owns_the_clock() {
let report = Report::without_sources(
Trigger::Boot,
Outcome::Off,
"not the appliance's PID 1, so the system clock is somebody else's to set",
);
tracing::debug!("clock: {}", report.summary());
estate_clock::report::record(report);
return;
}
log(&attempt(Trigger::Boot).await);
}
/// Whether a clock is configured at all (the medium's or the feed's).
fn configured() -> bool {
!matches!(
crate::feed::clock_config(),
Ok(Config::NotConfigured) | Ok(Config::Off)
)
}
/// **The running triggers**: a credential refused for skew, and the 6-hour
/// safety timer (`estate_clock::trigger::arm` has the rules and why there are
/// only these, and arms them once per process). Called after the boot sync, and
/// by a fed run before its first pause. Does nothing on a box that does not
/// own its clock, or whose clock is not configured — a clock the installer's
/// door feeds later arms them then ([`fed_arrived`]).
///
/// Both run on plain threads, never on the runtime: the skew handler is started
/// by the credential gate and must not borrow a worker for a TCP exchange, and a
/// sync is blocking by nature.
pub fn arm_triggers() {
if !owns_the_clock() || !configured() {
return;
}
estate_clock::trigger::arm(|trigger| {
let config = crate::feed::clock_config();
log(&sync(&config, trigger));
});
}
/// One sync of the clock in force (the feed read now), logged.
fn fed_sync() -> Outcome {
let config = crate::feed::clock_config();
let report = sync(&config, Trigger::Timer);
log(&report);
report.outcome
}
/// This process's fed clock runs (`estate_clock::offer::FedRun`): the
/// triggers armed first, then syncs backing off while the front does not
/// answer; one run at a time, off the caller's thread; the tunnel coming up
/// cuts a pause short.
static FED: FedRun = FedRun::new(fed_sync, arm_triggers);
/// **The installer's door fed a clock source or key while this box runs**
/// (the pairing loop's first answer, `crate::feed`). Every sync already reads
/// the feed ([`crate::feed::clock_config`]), so this only makes the first one
/// happen NOW — a first boot whose medium carries no clock would otherwise
/// wait for the 6-hour timer, which was never armed — and arms the triggers.
///
/// ★ **Returns at once: the syncs run on their own thread** (cov-E r3
/// 2026-10-06, runs 2 and 4): called from the installer's FEED, a twin fed
/// before its tunnel was up once ran the whole backoff on the feed's thread
/// and for 219 s answered nobody. A clock re-fed meanwhile gets one more run
/// of its own when the one under way ends.
pub fn fed_arrived() {
if !owns_the_clock() || !configured() {
return;
}
if !FED.start() {
tracing::warn!(
"clock: the fed sync could not get a thread of its own; the triggers take over"
);
}
}
/// **This box's tunnel just came up** (the pairing loop's `TunnelUp`).
///
/// The fed clock source is the front's utility address, reached through
/// this tunnel on a box whose fed syncs ran before it was up (i54a run23:
/// the twin's three syncs timed out, then it paused 20 s while its tunnel
/// came up and the front's `BecomeTwin` was refused for skew seven times).
/// So: a clock not yet set right is asked NOW — the pause under way is cut
/// short, and a run is started if none is. Once per process; nothing when
/// the clock is already right.
pub fn tunnel_up() {
if !owns_the_clock() || !configured() {
return;
}
FED.tunnel_up(last_outcome());
}
#[cfg(test)]
mod airgap {
use super::*;
use estate_clock::config::Source;
/// A loopback listener that would see any dial, and an https source aimed at it.
fn trap() -> (std::net::TcpListener, Result<Config, String>) {
let trap = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
trap.set_nonblocking(true).unwrap();
let url = format!("https://{}/", trap.local_addr().unwrap());
(
trap,
Ok(Config::Sources {
sources: vec![Source::Https(url)],
peer_key: None,
}),
)
}
fn dialled(trap: &std::net::TcpListener) -> bool {
std::thread::sleep(std::time::Duration::from_millis(100));
trap.accept().is_ok()
}
/// ★ **AIRGAP: a default build asks no https:// clock source.** The source is refused by
/// the feature's name, and the trap on loopback saw no connection.
#[cfg(not(feature = "clock-fetch"))]
#[test]
fn a_default_build_refuses_an_https_clock_source_by_name_and_dials_nothing() {
let (trap, config) = trap();
let report = sync(&config, Trigger::Timer);
let said = format!("{report:?}");
assert!(
said.contains("clock-fetch"),
"the refusal names the feature: {said}"
);
assert!(
!dialled(&trap),
"a default build dialled the https clock source"
);
}
/// With `clock-fetch` on, today's behaviour: the https source is asked.
#[cfg(feature = "clock-fetch")]
#[test]
fn a_clock_fetch_build_asks_its_https_clock_source_as_before() {
let (trap, config) = trap();
let report = sync(&config, Trigger::Timer);
assert!(
dialled(&trap),
"a clock-fetch build did not ask its https source: {report:?}"
);
}
}