nordisk/dwarves

Release doctor — design

The release doctor is the forge's (dwarves') answer to the one question you actually have across a whole multi-repo Rust constellation: can I release, what's off, and in what order? — and, increasingly, it fixes what it can and explains what it can't. It replaces the Maven-style per-project dependency:tree grind with a single constellation-wide command that assesses, self-heals, proves, and finally ships the entire dependency cascade to crates.io.

This is the design / model chapter. The practical command sequence is in release-doctor-guide.md; the one-screen recipe is in release-doctor-for-idiots.md. The engine now lives in dwarves: crates/dwarves/src/release_doctor/doctor.rs; the cascade in release_doctor/promote.rs + release_doctor/publish.rs; the CLI handlers are the standalone crate-release / workspace-release subcommands (Cmd::CrateRelease / Cmd::WorkspaceRelease) in crates/dwarves/src/main.rs — there is no longer a ReleaseOp enum or a nornir release … verb group.

The four-stage escalation

Four escalating tiers — each a strict superset of the trust the last one earned. The DRY-RUN default (no --execute) is read-only; the real publish is opt-in behind --execute. The fix/prove tiers are ENGINE capabilities the cascade composes (--rehearse is compiled in under the rehearse cargo feature); they are not separate CLI verbs — the current surface is the two subcommands plus their flags:

The escalation is the point: --execute runs only after the assess + fix + (ideally) rehearse tiers have cleared the tree. --execute itself refuses to run while any hard blocker remains (see the gate, below).

Stage 1 — assess (the DRY-RUN default of dwarves crate-release / dwarves workspace-release)

The read-only tier is what a plain dwarves crate-release / dwarves workspace-release (no --execute) prints: it plans and reports, mutating nothing. Read-only, no mutation, no server clone: every signal is computed from the LOCAL checkouts of the configured constellation (nornir.toml's [repo] set). The entry point is doctor::run(repos, policy) -> DoctorReport; format_report renders the advisory table. The composed signals:

• Working trees — real per-repo git status via gitio::worktree_freshness (check_dirty). A dirty repo must be re-validated before shipping.

• Publish pre-flight — path deps need a version (scan_path_dep_version_gaps). Every publishable crate's Normal/Build path deps — including [target.'cfg(…)'] variants — must carry a version =. This is the exact cargo publish manifest-verify blocker ("all dependencies must have a version requirement specified when publishing"), caught statically, up front, before any registry work. Dev-deps are exempt (cargo drops a path dev-dep's requirement). Each gap is classified:

  • auto-fixable — dep_owner.is_some(): the dep is a constellation member, so the fix and a suggested version (read from the dep's own manifest, resolving version.workspace = true) are known.
  • needs-decision — dep_owner.is_none(): the dep is a NON-member (e.g. modgunn, korp-collectors). It must join the release cascade or be published to crates.io first. The engine will not guess.

• External dependency skew (analyze_skew + enrich_transitive_pins). Per shared external crate the target = the highest declared version — the semver semilattice join (§graph-theory framing). Each repo is Ok / Behind / Forbidden, with bump hints. A repo that is Behind but whose Cargo.lock already resolves the target major is flagged ⛔ held_by_transitive_pin: a transitive dep pins the old major, so a plain manifest bump won't take (the hint would be misleading). Policy is inferred (target = highest anyone already uses) plus a tiny forbidden override, e.g. --forbid arrow=56.

• Promote gate — fork riders (compute_promote_block). Scans every [patch.crates-io] redirect and classifies FOREIGN forks (a path/git patch of a crate produced nowhere in the constellation — a hard blocker, because cargo publish strips [patch] and the published crate would then resolve the incompatible stock dep) vs safe own-crate path overrides (publish-order covers those; the strip is safe). The transitive per-crate closure marks every crate that rides a foreign fork as held.

• Topo publish order + cycles (crate_publish_order, detect_cycles). Computed at crate granularity — cargo publishes crates, not repos — so a repo-level mutual dep that is a clean crate-level DAG (facett ⇄ knut) is not a false cycle. Kahn's algorithm over SCC-condensed super-nodes.

• Blast radius (blast_radius) — for each dirty repo, its transitive dependents to re-validate (reverse-reachability).

Stage 2 — --fix (self-heal the mechanical blocker)

--fix applies the publish pre-flight automatically (fix_path_dep_versions): for every path dep missing a version, it writes version = "…" sourced from the dep crate's own manifest (following the path, resolving version.workspace), falling back to the report's suggested_version. It is formatting-preserving (toml_edit), idempotent (a dep that already grew a version is simply not re-found), and covers [target.'cfg(…)'] tables. On the nordisk constellation it fixed 41 gaps in one shot where a --rehearse had surfaced only 5 — the rehearse dies at the first error per crate; the static scan finds them all. After --fix, the doctor re-runs run() so the report (and any following --rehearse / --publish) sees the fixed tree.

Stage 3 — --rehearse (prove it against /sparring)

--rehearse escalates from advice to a real dry-publish against the embedded holger /sparring registry (release::stage): plan the cascade, cut an ephemeral release/staging branch per repo, cargo publish in topo order, then build each dependent FROM /sparring. It mutates (hence opt-in) and self-restores every repo to its main branch afterward via BranchGuard. This is where a residual, non-static blocker shows up as a real cargo error — which, in invertive mode, feeds back into the solve loop as a new symptom. The rehearsal refuses to execute if the plan carries a dependency cycle.

Stage 4 — --execute (the one-command crates.io cascade)

--execute is the whole-constellation release, folded into the tool (it used to be a hand-driven shell loop — never again). The handler:

1. Refuses on any blocker. It bails if path_dep_version_gaps, promote_blocked, or cycles is non-empty — "run --fix first."
2. Uses report.topo.order verbatim — the exact deps-first order --rehearse trusts. Private repos (private = true: holger, korp, ordning) are skipped at publish time (staged/verified but never pushed to crates.io).
3. Publishes each repo via promote_one_repo — the same per-repo unit shared with the per-repo dwarves crate-release path. Per repo the cascade (promote::run) is: docs-regen (README from .nornir/ → docs/book.pdf, best-effort, skipped on dry-run) → gate (compute_promote_block holds fork riders) → bump (patch/minor/major, applied + committed on the working branch; unified [workspace.package].version bumped once for facett-style workspaces, per-crate otherwise) → strip [patch.crates-io] on an ephemeral branch (deleted on return, so the dev tree keeps its fork) → publish deps-first with wait_for_index between a crate and its dependents (avoids resolve failures on propagation lag) → record every mutating step to the release_changes ledger the undo engine reverses.
4. FAIL-STOP. promote::run records per-crate publish failures in errors and still returns Ok, so the cascade must inspect them — the handler bails the instant a repo reports any error, rather than sailing on to a dependent whose dependency never shipped. crates.io's "already uploaded" is treated as a skip, not an error (a partially-shipped tree is a routine re-run, not a failure).

--bump keep — the idempotent finish / resume path

--bump takes patch|minor|major or keep/none. keep/none maps to a promote bump of None: publish the CURRENT versions unchanged. Because publish treats "already uploaded" as a skip, a keep cascade is idempotent — it ships exactly the not-yet-published crates and leaves the already-published ones untouched, with no gratuitous re-bump churn on the irreversible registry. It is the finish/resume mode: after a fail-stop, fix the cause and re-run with --bump keep (optionally --members <subset> to scope the run to what still needs shipping in topo order). The DRY-RUN default (no --execute) plans the whole cascade and runs cargo publish --dry-run — the real crates.io rehearsal, mutating nothing.

Undo

Every real publish run has a release_id and a release_changes ledger, and the reverse-run engine lives beside the cascade in release_doctor/undo.rs + release_doctor/release_changes.rs. It reverses a run newest-first: un-bump (restore the exact pre-bump manifests), restore a stripped [patch.crates-io] block, delete/reset the release branch; a crates.io upload is immutable and cannot be un-published, so it is yanked and reported. NOTE: this is engine capability — the current dwarves CLI surface exposes no standalone undo subcommand (there is no nornir release undo / ReleaseOp verb any longer).

The publish cascade

Order is a crate-level topological sort — each crate publishes only once everything it depends on is on the registry. Private members are staged and verified but never pushed.

Invertive mode — self-heal to a fixpoint

The doctor's north star (PLAN EPIC pt 9, "INVERTIVE MODE") is to run the diagnosis→remedy map backwards: read its own diagnosis, derive the action, apply the safe ones, re-assess, and repeat until a fixpoint (green) or a minimal set of human decisions. The four stages above are exactly the operators this loop composes; --auto (planned) closes the loop automatically.

Every finding lands in exactly one of three lanes — the engine solves what it can and explains what it can't (it never guesses):

Not everything can be solved automatically — when it can't, a proper explanation is given. The gate-held fork riders and the "needs-decision" pre-flight gaps are exactly that: lane ② surfaces one crisp ask (e.g. "modgunn is a non-member hard dep — add it to the cascade, feature-gate it, or hold?"), lane ③ explains a blocker outside the engine's power (e.g. "no crates.io token — create one at crates.io → Account → API Tokens, then cargo login"). The engine never fabricates a fix it can't justify.

The graph-theory framing

The remedy engine is deliberately not a tree of if/else. Per the PLAN EPIC "Mathematical Foundations for release doctor" (top of workspace_nordisk/PLAN.md, points 1–10), it is modelled as search/planning over an outcome space (utfallsrum Ω): a state s ∈ Ω is the vector of findings (dirty set, path-dep-version gaps, skew vector, promote-block set, cycles/SCCs); each remedy is a declarative STRIPS-style operator ⟨precondition, effect⟩ matched generically against the state; adding a remedy = adding an operator to a table, never a new conditional.

Graph methods give Ω its structure — and each maps to a concrete function in doctor.rs:

--rehearse is the empirical transition oracle for effects that can't be predicted statically (does it actually publish?): apply → observe the resulting state → continue. The planner searches the structured space, so the engine generalizes to new symptoms by adding operators/data, with zero new if arms.

The Gate-A analysis legs (code/policy)

The catalogue is not written here. It is EMITTED — see release-strategy.md, generated from release_doctor::strategy::REGISTRY.

That is a deliberate change, and the reason is measurable. The list that used to sit in this section was written by hand and had already drifted: it named a P4 edition leg that does not exist in GateALegs, and omitted min_req and dirty_sweep, which do. Five of the eight real legs described correctly, plus one fiction — while reading as an authoritative catalogue.

That is the same defect the feature gate had, one level up. The feature gate stated the right condition (Φ(e) ⊆ F(v)) and was fed the wrong side of it (F_loc instead of F_pub), so it was green by construction for eight releases. A hand-written spec states the right invariants and is fed nothing at all — nothing compares it to the code, so it drifts silently and is believed anyway. A document that cannot go red is worth exactly what a test that cannot go red is worth.

So the run is now driven FROM the registry:

• strategy::active_legs(&legs) computes the set of legs a run executes by filtering REGISTRY. The merge iterates that set through an exhaustive match on LegId, so a leg cannot run without a registry entry (compile-time), and cannot sit in the registry unexecuted (active_legs_is_derived_from_the_registry_and_matches_the_toggles).
• .nornir/release-strategy.md is regenerated and byte-compared by checked_in_document_matches_the_registry. Editing it by hand fails the test.

Each entry declares the two fields the eight-release blindness actually lived in:

Every leg still has a PURE binary-free core (unit-tested, red-when-broken) + a best-effort runner that degrades to a NOTE when its tool/DB/toolchain is absent — never a false block. Two auto-reconcile legs (auto_reconcile.rs, #47) additionally bump a ship version PAST the crates.io max and rewrite API-stale consumer reqs before the shoot. Per-leg reuse notes remain in release-doctor-rejection-coverage.md.

The feature strategy

Feature-Availability is split across two modules, one per side of Φ(e) ⊆ F(v), so neither side has two writers (LAW #5):

The rule. Φ is the union of every syntax a consumer can use to ask. F is what the registry will serve at the version that ships — local source is authoritative only for a version that does not exist yet, because crates.io bytes are immutable.

Why Φ needed rebuilding. The old builder read only a dependency entry's features = [...] list. Measured over 337 working-tree manifests, the [features]-table form (pkg/feat) appears in 178 of 337 (53%) — the dominant syntax, and the one korp uses exclusively. Its dependency entry on korp-collectors carries no features list at all, so if d.variants.is_empty() { continue } dropped the edge before any check could run. The graph held zero korp → korp-collectors edges: not a wrong comparison, no comparison at all.

Two things deliberately NOT asserted, because a red that is invented is worse than a red that is missed:

• default-features = true does not synthesize a default requirement. Verified against cargo 1.97.1: depending on a crate with no default feature is a no-op, not an error. Synthesizing it would have invented a blocker on korp itself, which declares default-features = false. The field is still carried, because dropping a default feature is a breaking change (Cargo book, cargo-feature-remove-another) — a provider drift question, not an edge assertion.
• dev-dependency requests are dropped. cargo publish strips them, so they cannot break a third party. Verified: a path dev-dep without a version is silently removed from the published manifest (cargo 1.40+), and since cargo 1.78 its [features] references are stripped too. Measured 81 such entries across 48 of 337 manifests — asserting on them would be 81 false blockers.

Scope, with the measurements that decided it

The last two rows are a claim with a date on it, not a permanent truth. feature_facts::unmodelled_constructs re-measures them on every run, so a construct the model does not handle is reported rather than silently ignored.

Cargo semantics this rests on, verified rather than remembered

Several widely-held beliefs are wrong, and encoding one would have made the graph wrong everywhere:

• Resolver 3 does not change feature unification. Its only documented change is MSRV-aware version selection; it inherits resolver 2's rules.
• The explicit root resolver wins over member editions. A workspace root saying resolver = "2" with edition = "2024" members builds on resolver 2, silently. A virtual workspace with no resolver key falls back to resolver 1 with only a warning — measured: 3 of 18 real workspace roots in the constellation declare none.
• features2 does not exist in the crates.io JSON API — it is an index-only split for pre-1.19 cargo. The API returns one pre-merged features map. On the sparse index the truth is the union of features and features2; reading only features would report a crate as missing features it has, i.e. a false red.
• "crates.io 403s without a User-Agent" is true only of the JSON API. index.crates.io served 200 anonymously (measured 2026-08-05).
• required-features are skipped silently. cargo build --bins builds only the targets whose features are met and exits 0, with no warning; naming a target explicitly errors 101. This is documented behaviour, and it means a CI step using a wildcard selector can never fail for an unbuilt binary — assert on artifacts, not the exit code.
• An unactivated cross-crate feature reference is still a hard error. Measured: a [features] row referencing a feature the provider lacks fails the build (exit 101) even when that row is never enabled — while cargo metadata --no-deps exits 0 on the same tree. The discovery path is therefore structurally unable to see it, which is why the check is static rather than delegated to cargo.

Also available over MCP

release_doctor is exposed as an embedded-only MCP tool (src/bin/nornir-mcp.rs): read-only advisory, refused when a NORNIR_SERVER target is set (it reads the local git trees, so it must run on the host). An agent calls it to see skew / dirty-trees / promotability before cutting a release, alongside the release GATE tools (no-path-patches, nexus_floor, no_regression, docs_fresh, …).