mikronova/draupnir

draupnir boot-status readback — await_power_state / boot_and_await

Added draupnir 0.1.5 (additive). The cross-backend boot-status readback seam: a Boot::boot fires an instance up but returns before it has actually powered on, so a consumer that needs to know the instance is up polls its Lifecycle::status until it reports the wanted state. This is the power-lifecycle analogue of the container run_to_completion drive loop, but written against the plain Lifecycle trait so it drives any backend — KVM, container, Redfish — and a mock alike.

Why it exists

The chain is app -> jera (job) / Skidbladnir (service) -> draupnir (provision). Boot::boot returns a Machine the instant the backend accepts the launch:

• a KVM guest returns as soon as tunnr has the QEMU process up — the kernel is still booting;
• a Redfish node returns after the power-on REST call is accepted — the metal is still POSTing / PXE-ing, minutes from ready;
• a container returns once the runtime schedules it — it may still be starting.

So boot() succeeding does not mean the instance is up. Before this seam a consumer had only Lifecycle::status — a single instantaneous read — and had to hand- roll its own poll-until-up loop (and its own timeout / never-comes-up handling) per call site. That is exactly the duplicated orchestration draupnir exists to own once. await_power_state provides the loop once; boot_and_await is the one-call "provision and confirm up" seam jera / Skidbladnir actually want.

The API (the only design — no alternatives)

pub struct WaitOptions {
    pub timeout: Option<Duration>,  // None = wait forever; Some(d) = timeout after d
    pub poll_interval: Duration,    // default 200ms
}

// Poll a Lifecycle until it reports `want`; Ok(()) the moment it does.
// A live status() error propagates; awaiting PowerState::Unknown is an Err(Spec);
// a bounded budget makes a never-reached state an Err(Backend) timeout, not a hang.
pub fn await_power_state<L: Lifecycle>(
    lifecycle: &L, machine: &Machine, want: PowerState, opts: &WaitOptions,
) -> Result<()>;

// The one-call provision seam: validate + boot, then await PowerState::On,
// returning the live Machine only once it is actually up.
pub fn boot_and_await<B: Boot + Lifecycle>(
    backend: &B, spec: &BootSpec, opts: &WaitOptions,
) -> Result<Machine>;

Design decisions, and why:

• Built on the always-compiled Lifecycle trait, generic over the backend. No reach into any backend internal, no new dependency — the default build stays pure std. That is what makes it mock-testable with no live instance: a scripted Lifecycle transitions Unknown -> Unknown -> On and the driver proves poll -> compare -> Ok end to end, and a stuck-Off node proves the timeout.
• boot_and_await reuses boot() (validate + Boot), not a bespoke path. An invalid spec is rejected at validate() before the backend — or any wait — is ever touched (asserted: boot_and_await_rejects_an_invalid_spec_before_booting), so the readback never waits on an unbooted instance. One boot seam, no duplication.
• Awaiting PowerState::Unknown is a Spec error, up front. Unknown means "not observed"; a target of Unknown is nonsensical (it would spuriously "succeed" the first time a backend could not read the state). Rejected before the first poll.
• A status() error propagates as Err. If the backend can't read the state (BMC unreachable, socket gone), the readback itself failed — that is not a state to keep polling through, it is surfaced immediately.
• Timeout is opt-in, mirroring RunOptions. Default / poll_every wait indefinitely (parity with run_to_completion); bounded(timeout, poll) returns a clear Error::Backend naming the instance and the last-observed state — a booted-but-dead node is a diagnosable timeout, never a silent hang. WaitOptions deliberately mirrors container::RunOptions field-for-field so the two readback loops read the same.

Tests (all no-instance, RED-when-broken)

src/lib.rs:

• await_power_state_returns_ok_once_the_state_is_reached — polls past two Unknowns to On and returns Ok (asserts it actually polled ≥3 times).
• await_power_state_times_out_when_the_state_is_never_reached — a stuck-Off node is a bounded Error::Backend timeout. RED-when-broken: drop the observed == want gate (return Ok on the first poll) and this + the reached-Ok test fail.
• await_power_state_rejects_awaiting_unknown — want = Unknown is Err(Spec) before any poll (status_calls == 0).
• await_power_state_propagates_a_backend_status_error — a status() Err propagates.
• boot_and_await_boots_then_confirms_power_on — boot then confirm On, returns the live Machine (power == On).
• boot_and_await_times_out_when_the_instance_never_comes_up — a never-up node makes the one-call seam a bounded timeout, not a hang.
• boot_and_await_rejects_an_invalid_spec_before_booting — an ISO-on-container spec is Err(Spec) before the backend / any wait is touched.

tests/testmatrix_rows.rs:

• await_power_state_confirms_a_boot_readback_across_backends — records the draupnir/lifecycle matrix row (reached-Ok · never-up-timeout · reject-Unknown · boot_and_await confirms On).

Fleet-level readback — boot_fleet_and_await (draupnir 0.1.5, additive)

The single-node boot_and_await confirms one instance. A multi-machine dispatcher (jera) boots a fleet from one image and needs to know who came up — not to stop at the first dead node. boot_fleet_and_await is the fleet analogue: it plan_fleets the spec into n members and drives each through boot_and_await (no per-backend twin — L5), rolling up a per-member verdict plus aggregate tallies.

pub enum MemberOutcome {
    Up(Machine),      // booted AND confirmed On within the budget — ready
    Timeout(String),  // Err(Backend): readback timed out / backend couldn't bring it up in budget
    Error(String),    // Err(Spec|Unsupported): hard misconfig — waiting can't help
}
pub struct MemberReadback { pub name: String, pub outcome: MemberOutcome }
pub struct FleetReadback { pub members: Vec<MemberReadback> }  // in fleet order
impl FleetReadback { fn ready(&self)->usize; fn failed(&self)->usize; fn all_ready(&self)->bool; }

// Boots n members from one spec, awaits each to On under `opts`, rolls up who is ready.
pub fn boot_fleet_and_await<B: Boot + Lifecycle>(
    spec: &BootSpec, n: usize, backend: &B, opts: &WaitOptions,
) -> FleetReadback;   // infallible by construction

Design decisions, and why:

• Reuses boot_and_await per member, not a bespoke fleet loop. Each member gets the exact same validate → boot → confirm-On path; the fleet call only fans it and classifies. One readback seam, no twin (L5).
• Infallible — returns a rollup, never an early Err. A dead or misconfigured member is a per-member Timeout/Error line, so a partial fleet is always observable (parity with boot_fleet). The aggregate is ready() = the Up count, failed() = the rest, all_ready() = non-empty and none failed.
• Bounded per-member timeout is what makes it safe. Each member is awaited independently within opts, so one node that never powers on rolls up as a Timeout while the healthy members are Up — it cannot hang the whole rollup. Pass a WaitOptions::bounded(..); the default unbounded options would block forever on the first dead node (documented, same contract as await_power_state).
• Three honest buckets from boot_and_await's result. Ok → Up; Err(Backend) → Timeout (the backend did not get the node to On in budget — a readback timeout or a transient backend failure while booting/polling); Err(Spec|Unsupported) → Error (a hard misconfiguration or an unavailable backend — retrying cannot help). The split lets a dispatcher retry Timeouts and fail fast on Errors.

How jera calls it: build the shared BootSpec + fleet size n from the Roster/WorkPayload, pass a bounded WaitOptions, then read FleetReadback: all_ready() to gate the whole fleet, ready()/failed() for the tally, and the members[] lines (name → Up/Timeout/Error) to decide which members to retry (the Timeouts) versus reject (the Errors).

Fleet tests (all no-instance, RED-when-broken) — src/lib.rs:

• boot_fleet_and_await_rolls_up_one_dead_member_as_timeout_others_up — a fleet of 3 with a dead middle member rolls up node-2 = Timeout, node-1/node-3 = Up, ready 2 / failed 1. RED-when-broken: neuter the observed == want gate (dead node returns Ok → ready 3) or flip the Err(Backend) -> Timeout arm to Error (the matches!(.., Timeout) verdict fails).
• boot_fleet_and_await_reports_a_fully_ready_fleet — every member Up, all_ready.
• boot_fleet_and_await_rolls_up_an_invalid_spec_as_error_not_timeout — an invalid spec is Error on every member (rejected at validate() before the backend), never Timeout — proves the Spec/Backend bucket split.
• boot_fleet_and_await_of_zero_is_an_empty_not_ready_rollup — n=0 is an empty rollup, all_ready() false.

tests/testmatrix_rows.rs:

• boot_fleet_and_await_rolls_up_who_is_ready — records the draupnir/lifecycle matrix row (partial fleet node-2 timeout / others up, ready 2 / failed 1, plus a full-fleet all-ready arm).

Parallel fleet boot — boot_fleet_and_await_parallel (draupnir 0.1.5, additive)

boot_fleet_and_await boots+awaits its members serially, so a fleet's wall-clock is the sum of the per-member waits: 8 nodes on a 5-minute budget is up to 40 minutes, one node at a time. But the per-member wait is almost entirely the backend's own boot latency (a metal node POSTing, a guest booting) — idle time draupnir spends sleeping in await_power_state. boot_fleet_and_await_parallel overlaps that idle time: it fans each member onto the gatling fork-join pool (one worker per member) so all n are booted+awaited at once and the wall-clock collapses to roughly the slowest member's wait.

// Same arguments + same FleetReadback as the serial call — just concurrent.
pub fn boot_fleet_and_await_parallel<B: Boot + Lifecycle + Sync>(
    spec: &BootSpec, n: usize, backend: &B, opts: &WaitOptions,
) -> FleetReadback;

Design decisions, and why:

• gatling::gatling_forkjoin::gatling_for_each — the ONE sanctioned threading home. ROOT-LAW #0 (rayon-free / core-saturation) forbids a bare std::thread::spawn in any non-engine crate; all concurrency originates in gatling. The fork-join pool is scoped, so each member still borrows the shared &backend / &opts and the fleet plan off the stack with no Arc/clone, and it returns results in index order (deterministic rollup). Worker count is pinned to plan.len() — one worker per member, NOT the default core count — because classify_member is blocking-IO (park on a boot), not CPU: every member needs its own parked thread so the wall-clock collapses to the slowest, a deliberate exemption from the core-saturation law (which governs CPU hot paths).
• Identical rollup to the serial path, by construction. Both paths classify every member through the one shared classify_member step (validate → boot → confirm-On → bucket), and the parallel path joins the member threads back in fleet order (node-1…node-n), not completion order. So the FleetReadback — per-member MemberOutcome, member order, and the ready()/failed()/all_ready() tallies — is byte-for-byte the same whichever path booted it. The parallel path changes when members are awaited, never what the rollup says. This is asserted directly (parallel_fleet_boot_is_identical_to_serial_for_a_mixed_fleet).
• Still infallible, still one-dead-node-safe. Each member is awaited on its own thread within the bounded opts, so a dead node times out on its thread and is joined as a Timeout line while the healthy threads finish independently — one dead node cannot hang the others (same guarantee as the serial call, now truly parallel). Pass a bounded WaitOptions for the same reason.
• B: Sync bound — the serial path stays the general fallback. Sharing &backend across threads needs B: Sync. All three real backends satisfy it (KvmBoot/ContainerBoot are Arc<Mutex<..>>, RedfishBoot is plain owned fields). A backend that is not Sync simply calls the serial boot_fleet_and_await, which is kept unchanged as the fully-general path — the working path is never replaced (L2). Two functions, one shared classification, no twin (L5).

Parallel tests (all no-instance, RED-when-broken, mutation-verified) — src/lib.rs:

• parallel_fleet_boot_is_identical_to_serial_for_a_mixed_fleet — the equivalence guard: a mixed fleet of 5 (node-3 dead → Timeout, node-5 unbootable → Error, node-1/2/4 Up) rolls up to a FleetReadback equal to the serial one. RED-when-broken: reversing the join order (or any bucket/member divergence) breaks the assert_eq!.
• parallel_fleet_boot_actually_overlaps_the_members — the concurrency guard: a barrier backend that records peak in-flight status calls proves peak == n (all members awaited at the same instant). RED-when-broken: pointing it at the serial boot_fleet_and_await collapses the peak to 1.
• parallel_fleet_boot_of_zero_is_an_empty_rollup — parity with the serial n=0.

tests/testmatrix_rows.rs:

• boot_fleet_and_await_parallel_matches_the_serial_rollup — records the draupnir/lifecycle matrix row (parallel == serial over the mixed fleet).

Perf / bench

await_power_state and the serial boot_fleet_and_await are poll loops gated on the backend's own status latency (a QEMU handle read, a container-state query, a BMC REST round-trip) — no compute hot path to guard, and a real measurement would time the backend / the network, not draupnir. The parallel fleet boot IS a clean measure, though: it overlaps exactly that per-member latency, so benches/fleet_boot.rs (criterion, dev-dependency only — the lib's runtime deps are untouched) benches serial vs _parallel over a fake backend whose status sleeps a small fixed latency modelling the readback round-trip. The serial arm pays it n times in series, the parallel arm once:

(Loki 12c/64GiB, 2 ms per-member latency, light --sample-size 10 run; serial ≈ n × latency, parallel ≈ one latency.) These are indicative — the authoritative HEAVY numbers are the Loki bencher's; noted in draupnir_agents_problems.md.

Related — teardown + functional-status emit

await_power_state / boot_fleet_and_await[_parallel] each record their verdict as a draupnir/lifecycle functional_status row, and the undo seam rollback() tears a booted instance back down (power-off through the same Lifecycle trait), likewise a GREEN/RED row. The full table of emitting lifecycle surfaces — boot · rollback · await_power_state · wait_ready · the seed builders — and the RED-when-broken failed_boot_and_rollback_emit_red_rows test are documented in functional-status-emit.