dwarves-gui/src/populate_steps.rs
//! **WHICH PHASE a populate is in, published by dwarves at the transition.**
//!
//! A populate is not one thing. The members are cloned, then a warehouse is built from
//! them, then the lenses are scanned over it — three phases, minutes each, and until now
//! the pane drew one flat `repos_done / repos_total` bar over all of them. That bar
//! reaches 100 % when the *clone* finishes and then sits there, full, for the entire
//! warehouse build and code-graph consume. A full bar over work that has not started is the
//! hollow green the step bar exists to replace.
//!
//! The producer is [`observe`], and the registry is the shared
//! [`step_board::Board`] — the same one korp's region cuts publish into, so a bar drawn
//! from a dwarves populate and a bar drawn from a korp convert are the same widget
//! reading the same vocabulary.
//!
//! # What each phase is observed FROM, and what it is not
//!
//! | phase | live signal | measured? |
//! |---|---|---|
//! | `clone` | `WorkspaceMember.sync_state` per member — `fetching` / `ok` / `error…` | **yes**: `(n_ok, n_members)` |
//! | `warehouse` | `dep_graph_edges` gaining rows (the build's product) | no — nothing counts it |
//! | `code-graph` | the workspace's `current_snapshot` — the stamp nornir's consumer sets on a release | no — nothing counts it |
//!
//! The clone step is the constellation's **first genuinely `Determinate` step**. korp's
//! four are all `count: None` by design because osm-katana exposes no counter; here the
//! server reports one row per member every poll, so `(n_ok, 19)` is measured rather than
//! estimated and the segment fills to a real fraction.
//!
//! The other two are honestly unmeasured and say so: an active step with no count renders
//! [`ProgressMeter::Indeterminate`](facett_jobview::ProgressMeter::Indeterminate) — a
//! sweep, a positive statement that work is in flight and nobody is counting it — and its
//! `state_json` carries **no** `fraction` key. A zero there would read as a measurement,
//! and one invented percentage beside a measured one makes the measured one worthless.
//!
//! # Why `clone_events` is corroboration and not the live counter
//!
//! `clone_events` is the durable ledger: one row per member attempt with `op`, `status`
//! and the error chain in `detail`. It is the right place to explain *why* a member
//! failed, and [`explain_failure`] uses it for exactly that.
//!
//! It is **not** where the moving fraction comes from, because of when it is written.
//! `nornir::warehouse::clone_events::record_fetch_report` is called once, after the whole
//! `Workspaces.Fetch` returns (`nornir/src/server_grpc.rs:4391`), so all 19 rows land
//! together at the end. A bar fed from it would sit at 0/19 for the entire clone and jump
//! to 19/19 — which is the same lie the flat bar told, in the other direction. Measured
//! on the live `nordisk` warehouse at 2026-08-04 00:47 UTC: 40 rows, 19 distinct members,
//! two full sweeps plus a partial — and note that a raw count over that table gives 40 of
//! 19, so even as corroboration it must be read as the latest row per member.
//!
//! The per-member `sync_state` on `Workspaces.Get`, which the pane already polls every
//! 1.5 s, moves *during* the clone. That is the live counter, and it is measured, not
//! inferred.
//!
//! # Two owners, one run
//!
//! [`step_board::Board::enter`] returns [`Result<(), OrderFault>`] and refuses a step that
//! overtakes an unfinished predecessor. For korp that is theoretical — one producer, one
//! thread. Here it is load-bearing: the clone phase is driven by the poll worker while a
//! `Workspaces.Fetch` started by the Re-sync button, by nornir's own poll loop, or by the
//! CLI drives the build phase, and nothing else stops the second announcing itself while
//! the first is still cloning. [`observe`] **returns** the fault rather than swallowing
//! it, and the pane shows it.
use facett_jobview::{JobEntry, JobStatus};
#[cfg(any(
test,
feature = "fixtures",
feature = "robot-remote",
feature = "watch"
))]
use nornir_warehouse_embed::Counter;
use nornir_warehouse_embed::PopulateProgress;
use step_board::{board_state_json, shared, Board, OrderFault, StepDef, StepRun};
/// The board every dwarves populate publishes into.
pub fn board() -> Board {
shared("dwarves/populate")
}
/// The job-instance id for one warehouse's populate — one bar per warehouse, namespaced
/// so a second kind of dwarves job can never collide with it.
pub fn job_id(workspace: &str) -> String {
format!("populate/{workspace}")
}
/// **The three phases of a populate**, in dependency order.
pub fn populate_steps() -> Vec<StepDef> {
vec![
StepDef::new("clone", "clone the members"),
StepDef::new("warehouse", "build the warehouse"),
StepDef::new("code-graph", "consume the code graph"),
]
}
/// Index of each phase — named, so a caller never writes a bare `1`.
pub const CLONE: usize = 0;
/// Index of the warehouse-build phase.
pub const WAREHOUSE: usize = 1;
/// Index of the code-graph phase (nornir's consumer feeding the code room).
pub const CODE_GRAPH: usize = 2;
/// **Which phase a poll says we are in** — the pure decision, split out from the board
/// write so it is assertable without a registry, a server or a frame.
///
/// `None` means *no run is visibly under way*: nothing is fetching and no member has
/// landed, which is the resting state of a warehouse nobody is touching. It is
/// deliberately not "phase 1 at 0 %", because a bar drawn for work that is not happening
/// is the thing this module removes.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Phase {
/// Members are being cloned or fetched.
Clone,
/// Every member landed; the warehouse has no dependency graph yet.
Warehouse,
/// The warehouse is built; nornir's consumer has not stamped the workspace yet.
CodeGraph,
/// All three finished.
Complete,
/// Nothing is happening.
Idle,
}
impl Phase {
/// The step index this phase publishes into, if it is a step at all.
pub fn step(self) -> Option<usize> {
match self {
Phase::Clone => Some(CLONE),
Phase::Warehouse => Some(WAREHOUSE),
Phase::CodeGraph => Some(CODE_GRAPH),
Phase::Complete | Phase::Idle => None,
}
}
}
/// Read a poll as a phase.
///
/// The order of the tests is the order of the chain, and each one is a *positive*
/// observation rather than the absence of a later one:
///
/// - anything still fetching ⇒ **clone**;
/// - members landed but `dep_graph_edges` empty ⇒ **warehouse build** (its product has
/// not appeared);
/// - the dep graph exists but the workspace carries no consumer stamp ⇒ **code graph**;
/// - both products present ⇒ **complete**.
///
/// **This function is stateless and therefore backwards-blind.** A
/// [`Counter::Pending`](nornir_warehouse_embed::Counter::Pending)
/// means *not measured yet*, and on the fast ticks the pane skips both table scans — so
/// for a run that has reached `CodeGraph` this will honestly answer `Warehouse`, because it
/// can see no evidence of anything later. Resolving that absence needs the run's own
/// position, which is a fact this function deliberately does not have; [`observe`] holds
/// the phase instead. Do not "fix" it here by threading state in — a stateful phase
/// reader cannot be asserted without a board.
pub fn phase_of(p: &PopulateProgress) -> Phase {
if p.repos_total == 0 {
return Phase::Idle;
}
if p.repos_fetching > 0 {
return Phase::Clone;
}
if p.repos_done == 0 {
return Phase::Idle;
}
if p.repos_done < p.repos_total {
// Not fetching, not all landed: the sweep stopped short. Still the clone phase —
// the failure is reported through `fail`, not by pretending we moved on.
return Phase::Clone;
}
if p.crates.value().is_none() {
return Phase::Warehouse;
}
if p.code_graph.value().is_none() {
return Phase::CodeGraph;
}
Phase::Complete
}
/// **Publish one poll onto the board** — the producer.
///
/// Idempotent: called every poll, it writes only on a real transition, so a bar does not
/// flicker and `updated_unix` means what it says. It starts a run when one becomes
/// visible (whoever launched it — the Re-sync button here, nornir's poll loop, or the
/// CLI), advances it as the phases land, and fails it loudly when a member does not.
///
/// # Errors
///
/// The [`OrderFault`] from [`Board::enter`], unswallowed. A caller that ignores it still
/// gets a red bar — the board marks the offending step [`Failed`](facett_jobview::StepPhase::Failed)
/// itself — but the pane shows the sentence, which is how a human finds out that two
/// producers are racing on one run.
pub fn observe(workspace: &str, p: &PopulateProgress, reason: &str) -> Result<(), OrderFault> {
let b = board();
let id = job_id(workspace);
let phase = phase_of(p);
if phase == Phase::Idle {
return Ok(());
}
// A run this process has not seen before — or one that restarted, which is what a
// fresh clone phase after a finished run means. `begin` bumps the run number, so the
// bar going from "3 of 3" back to "step 1 of 3" reads as run #2, not as lost ground.
let existing = b.get(&id);
let restarted = match &existing {
None => true,
Some(run) => {
run.generation == 0
|| (phase == Phase::Clone && (run.is_complete() || run.failed().is_some()))
}
};
if restarted {
b.begin(&id, workspace, populate_steps(), reason);
}
// ── THE MEASURED STEP ────────────────────────────────────────────────────────
// One row per member on every poll, so this moves while the clone is happening.
// It is the only count in this chain that exists, and it is why the segment can
// fill to a real fraction instead of sweeping.
b.set_count(&id, CLONE, p.repos_done as u64, p.repos_total as u64);
// ── A MEMBER THAT DID NOT LAND ───────────────────────────────────────────────
// Reported before any advance: a clone that lost members has not finished, and a
// chain must not walk past a phase that failed.
if p.repos_failed > 0 {
b.fail(&id, Some(CLONE), explain_failure(p));
return Ok(());
}
let Some(want) = phase.step() else {
// Phase::Complete — every product is on disk. Say so once.
if !b.get(&id).is_some_and(|r| r.is_complete()) {
b.finish(&id);
}
return Ok(());
};
let run = b
.get(&id)
.unwrap_or_else(|| StepRun::new(&id, workspace, populate_steps()));
if run.active() == Some(want) {
return Ok(()); // already there — no write, no flicker
}
// **A run never walks backwards within its generation.** Restarting is what `begin`
// is for, and it bumps the run number so the reset is legible.
//
// This is load-bearing rather than defensive, because of how the poll is sampled:
// the probes behind `crates` and `code_graph` only run on every CRATE_REFINE_EVERY-th
// tick, so on the three fast ticks in between BOTH counters read `Pending` and
// `phase_of` — which is pure, and honestly reports "no evidence of the build's
// product" — says `Warehouse` even for a run that has reached `CodeGraph`. Publishing
// that would enter an earlier step behind a later active one, which is exactly the
// `OrderFault::Backwards` the board refuses: the bar would go RED three ticks out of
// four, on a populate that is perfectly healthy.
//
// The alternative — teaching `phase_of` to remember where the run got to — would make
// the phase decision stateful and untestable without a board. Absence of evidence is
// resolved here, where the run's own position is known, instead of being smuggled
// into a pure function.
// ⚠ The clamp is against the furthest step the run has REACHED, not against the one
// that happens to be ACTIVE right now. `run.active()` matches `StepPhase::Active`
// only, so the moment `code-graph` finished there was no active step at all, the guard
// below stopped firing, and the very next Pending-counter tick re-entered
// `warehouse` — behind a `code-graph` that had already begun. That is exactly the
// `OrderFault::Backwards` this block exists to prevent, and it fired on a healthy
// 18-member populate on 2026-08-08:
// "OUT OF ORDER: step warehouse started after step code-graph had already begun"
//
// The reasoning above is right and unchanged — a Pending counter means NOT MEASURED,
// never "empty", so `phase_of` honestly under-reports on the fast ticks. It was the
// NOTION OF POSITION that was too narrow: a run's position is the furthest step it
// has touched, and a step that is `Done` is every bit as much "already begun" as one
// that is `Active`. Asking only "what is active?" makes the guard blind for the whole
// tail of a run, which is precisely when the completed code-graph step is sitting there to be
// walked backwards over.
let reached = run
.steps
.iter()
.rposition(|s| s.phase != facett_jobview::StepPhase::Pending);
if let Some(reached) = reached {
// `<=`, not `<`: re-entering the furthest step is also backwards when that step
// is already `Done`. The still-`Active` case returned above, so reaching here
// with `want == reached` means re-opening something finished.
if want <= reached {
return Ok(());
}
}
// Everything before the phase we have observed genuinely finished: we are looking at
// ITS PRODUCT on disk, which is the evidence, not an inference. Claim it explicitly —
// `enter` refuses to infer it, which is the whole point of the order check.
if want > 0 {
b.complete_through(&id, want - 1);
}
b.enter(&id, want)
}
/// **Why the clone lost members, in the words the server used.**
///
/// `PopulateStatus` rolls up `clone_events` — the durable per-member ledger — into a
/// first failing member and its error chain. Naming both beats "3 member(s) failed",
/// which tells a person at 03:00 nothing they can act on.
pub fn explain_failure(p: &PopulateProgress) -> String {
let mut s = format!(
"{} of {} member(s) failed to clone",
p.repos_failed, p.repos_total
);
if !p.failing_member.is_empty() {
s.push_str(&format!(" — first: {}", p.failing_member));
}
if !p.last_error.is_empty() {
s.push_str(&format!(": {}", p.last_error));
}
s
}
/// **Say what is happening before phase 1 has anything to show.** A populate can be
/// underway before a single member reports `fetching` (the server is still opening the
/// workspace), and a bar reading "0 of 3" tells a watcher nothing about whether it is
/// advancing or wedged.
pub fn note(workspace: &str, what: impl Into<String>) {
board().note(&job_id(workspace), what);
}
/// **Start a run because a human pressed Re-sync** — so the bar is up on the frame after
/// the click, rather than on whichever poll first sees a member go `fetching`. Returns
/// the new run number.
pub fn begin_for(workspace: &str, reason: &str) -> u64 {
board().begin(&job_id(workspace), workspace, populate_steps(), reason)
}
/// The populate rows for the 🧰 Jobs pane — **one bar per warehouse**.
///
/// `kind = "populate"` so they sort and filter apart from the durable ledger rows, which
/// record what *happened*; these show what *is happening*.
pub fn to_job_entry(run: &StepRun) -> JobEntry {
let status = if run.failed().is_some() {
JobStatus::Failed
} else if run.in_progress() {
JobStatus::Running
} else if run.is_complete() {
JobStatus::Done
} else {
JobStatus::Queued
};
let mut e = JobEntry::new(run.id.clone(), "populate", status);
e.target = run.title.clone();
e.detail = run.error.clone().unwrap_or_default();
// The producer's own sentences travel as data — the widget paints them and
// republishes them, never composing a second copy.
e.with_steps(&run.to_meter())
}
/// Every populate's row for the Jobs pane, in id order.
pub fn job_entries() -> Vec<JobEntry> {
board().snapshot().iter().map(to_job_entry).collect()
}
/// The whole board as `state_json` — the list a robot walks to assert that a bar shows
/// what it shows.
pub fn state_json() -> serde_json::Value {
board_state_json(&board(), "populates")
}
/// A [`PopulateProgress`] a test can build without a server. Not `Default`: every field
/// is named at each call site on purpose, because the phase decision reads most of them.
#[cfg(any(
test,
feature = "fixtures",
feature = "robot-remote",
feature = "watch"
))]
pub fn progress_fixture(
workspace: &str,
total: usize,
done: usize,
fetching: usize,
failed: usize,
crates: Counter,
code_graph: Counter,
) -> PopulateProgress {
PopulateProgress {
workspace: workspace.to_string(),
running: fetching > 0,
repos_total: total,
repos_done: done,
repos_fetching: fetching,
repos_failed: failed,
crates,
code_graph,
bytes: Counter::Unavailable {
why: nornir_warehouse_embed::BYTES_UNAVAILABLE_WHY,
},
verdict: if failed > 0 {
"red".into()
} else {
"green".into()
},
last_synced: String::new(),
failing_member: if failed > 0 {
"znippy".into()
} else {
String::new()
},
last_error: if failed > 0 {
"ssh: connect to host codeberg.org port 22: Connection timed out".into()
} else {
String::new()
},
}
}
/// **`$DWARVES_ROBOT_POPULATE_STEPS` — drive the board to a named posture.**
///
/// A UI opened against a warehouse nobody is populating has an empty board, and a
/// screenshot of it would prove only that no bar crashes. This seeds one — **through the
/// real producer** ([`observe`]), from real [`PopulateProgress`] values, so what the pane
/// draws went down the same path a live poll does. That is the difference between a
/// guard that DRIVES the operation and one that stages its side effect.
///
/// | value | what the board ends up saying |
/// |---|---|
/// | `midrun` | nordisk cloning, 12 of 19 — a genuine `Determinate` — plus a second warehouse mid-build |
/// | `building` | every member landed, warehouse build ACTIVE and `Indeterminate` |
/// | `failed` | the clone lost a member, step 1 RED with the ssh error |
/// | `complete` | `run #1 · 3 of 3 steps complete` |
/// | `fault` | nordisk on the clone step; a second producer announces warehouse-build over it and the step goes RED |
///
/// Anything else (including unset) leaves the board alone.
#[cfg(any(
test,
feature = "fixtures",
feature = "robot-remote",
feature = "watch"
))]
pub fn apply_robot_posture() -> Option<&'static str> {
let want = std::env::var("DWARVES_ROBOT_POPULATE_STEPS").ok()?;
let b = board();
// The posture seeds the workspace the shell will actually SELECT, so the bar it
// draws is the bar this seeded. Point `$NORNIR_WORKSPACE` at a name the server does
// not serve and the poll returns nothing, which leaves the posture standing — that
// is how a mid-clone bar gets photographed without a real 19-member clone to catch
// in the act.
let ws0 = std::env::var("NORNIR_WORKSPACE").unwrap_or_else(|_| "nordisk".to_string());
let ws0 = ws0.as_str();
let known = |v: u64| Counter::Known { value: v };
match want.trim() {
// 12 of 19 members in — the first genuinely measured fraction in the constellation.
"midrun" => {
let p = progress_fixture(ws0, 19, 12, 3, 0, Counter::Pending, Counter::Pending);
observe(ws0, &p, "Re-sync clicked").ok()?;
let q = progress_fixture("holger", 4, 4, 0, 0, Counter::Pending, Counter::Pending);
observe("holger", &q, "nornir's poll loop found new commits").ok()?;
Some("midrun")
}
"building" => {
let p = progress_fixture(ws0, 19, 19, 0, 0, Counter::Pending, Counter::Pending);
observe(ws0, &p, "Re-sync clicked").ok()?;
Some("building")
}
"failed" => {
let p = progress_fixture(ws0, 19, 16, 0, 3, Counter::Pending, Counter::Pending);
observe(ws0, &p, "Re-sync clicked").ok()?;
Some("failed")
}
"complete" => {
let p = progress_fixture(ws0, 19, 19, 0, 0, known(1_284), known(1));
observe(ws0, &p, "Re-sync clicked").ok()?;
Some("complete")
}
// **The two-owner race, staged through the real API.** The poll worker has the
// clone running; a second producer announces the warehouse build over it. Nothing
// here fakes the red — `enter` refuses the publish and marks the step itself.
"fault" => {
let p = progress_fixture(ws0, 19, 12, 3, 0, Counter::Pending, Counter::Pending);
observe(ws0, &p, "Re-sync clicked").ok()?;
let violation = b.enter(&job_id(ws0), WAREHOUSE);
assert!(
violation.is_err(),
"the order check must refuse this publish"
);
Some("fault")
}
_ => None,
}
}