nordisk/nornir

The nornir build/release/introspection "thing" — technical manual

Class: guide · audience: an operator driving a real crates.io release, or a maintainer reading the release subsystem. Deadpan and factual: what each command does, in what order, what gates it, and how it fails.

This is the operations manual for the whole nornir release machine. Adjacent docs: release-design.md (the why), the per-topic files release-doctor-design.md, release-graph-math.md, release-publish-traps.md, release-rehearsal.md, and the running log of imperfections found in the field: nornir-release-improvements.md (referenced throughout by item number, e.g. item #7).

⚠ MOVED — read this first (doctrine L5a, 2026-07-14: release is a build op). The release ENGINE — the whole publish machine described below (cargo/depdag/ graph_math/preflight/registry/root_cause/source_drift/edition/promote/publish/undo, and the doctor --publish cascade orchestration) — moved to dwarves, where it is the single implementation the dwarves tool, the nornir CLI and nornir-server all share. It now lives in dwarves::release_doctor (dwarves/crates/dwarves/src/release_doctor/) and is driven by dwarves release <workspace-path> [--execute]. The nornir release <op> ACT verbs described here are retained only as still-working shims that print a deprecation banner (release::release_moved_banner); prefer dwarves release. What STAYS in nornir is the KNOWLEDGE / read-only lineage side, reading from the Mimir warehouse: the release HISTORY/lineage verbs (trace, events, changelog, impacted-crates, undo-ledger view) plus nornir's own quality gates — src/release/{doctor,gate,pipeline,progress, regression,stage}.rs. Every src/release/<engine-module>.rs path cited below is now dwarves::release_doctor::<module> unless it names one of those six retained files.

0. The xtask model — pure Rust, no shell

There is no release.sh, no Makefile, no publish.sh. The entire build/release machine is pure Rust. The engine now lives in dwarves::release_doctor (dwarves/crates/dwarves/src/release_doctor/) and is driven by dwarves release; the nornir binary keeps a still-working release shim tree (wired from src/bin/nornir.rs) plus the read-only lineage verbs in src/release/. This is the constellation's zero-shell / pure-Rust charter:

• Every step that another project would write in bash — bump a version, strip a [patch.crates-io] block, poll crates.io until a version indexes, walk crates in dependency order, cargo publish each — is a Rust function with unit tests.
• Network I/O uses the vendored pure-Rust ureq client (no reqwest / openssl / native-tls), so the release path has no C toolchain or system-TLS dependency.
• cargo itself is the only external process spawned, and it is resolved robustly (item #1): prefer $CARGO, then rustup which cargo, then ~/.cargo/bin/cargo, then PATH, with one clear pre-flight error if none is found instead of a cryptic per-crate spawn failure.

Module map. The engine modules moved to dwarves::release_doctor (dwarves/crates/dwarves/src/release_doctor/); the retained nornir-side modules stay under src/release/ (the KNOWLEDGE / lineage / nornir-quality half):

1. The command surface

All commands are nornir release <op>. Config is discovered by walking up for workspace_*/release/nornir.toml (see config, and src/config.rs::discover), or pinned with --config / $NORNIR_CONFIG.

doctor — assess, self-heal, and (opt-in) publish the whole constellation

nornir release doctor is read-only by default: it prints per-repo dirty trees plus cross-workspace external-dependency version skew with bump hints. It becomes a mutating command only with explicit flags:

• --fix — write version = "…" for every path dep of a publishable crate that lacks one (sourced from the dep crate's own manifest). Idempotent, formatting-preserving.
• --auto [--dry] — invertive self-heal: read its own diagnosis and close every AUTO symptom class to a fixpoint (fix path-dep versions, join semver skew, cut cheap cycles), re-assessing after each. --dry prints the ordered remedy PLAN without applying. Terminates green or at a crisp DECISION ask.
• --explain — surface every crate the cascade withholds (gate-held fork riders) with its per-crate reason (fork_kind, the patched_dep it rides, direct-vs-transitive, unblock advice).
• --publish — the ONE-COMMAND RELEASE: after assessing with zero blockers, publish the whole constellation to crates.io, every public repo in crate-topo order, deps-first, FAIL-STOP, each mutation recorded for release undo. Private repos are skipped. This is the path the rest of this manual is about.

--publish modifiers:

Real lines:

nornir release doctor                                   # read-only advisory
nornir release doctor --auto --dry                      # what would self-heal do
nornir release doctor --publish --bump patch --publish-dry-run   # rehearse the cascade
nornir release doctor --publish --bump patch            # the real fat release
nornir release doctor --publish --bump keep --publish-from facett   # resume a torn run

publish <repo> — one repo's cascade

nornir release publish <repo> [--bump patch] [--dry-run] [--no-wait-index] [--wait-secs 300] walks a single repo's publish order from nornir.toml. It drives the same per-repo promote unit the constellation cascade uses, so the two paths are byte-for-byte identical.

stage — the /sparring rehearsal

nornir release stage plans the cascade against holger's embedded /sparring sparse registry (prints the topo publish order + per-repo verify targets). stage --execute actually runs steps 1–3: spawn holger → cargo publish each crate into /sparring in topo order → build each dependent from /sparring. This rehearses the cascade end-to-end against a real registry that is NOT crates.io, so nothing becomes immutable.

nornir release stage                                    # print the plan
nornir release stage --execute                          # run the rehearsal into /sparring

Known rehearsal caveats (open items): the sparring store is PERSISTENT and lives under holger's own --data dir, not git/.nornir/stage/registry/ (item #2 — a --fresh reset flag is the planned fix); private repos' publish attempts print alarming-looking failures the real publish correctly skips (item #3).

promote — the per-repo publish unit (internal)

There is no standalone nornir release promote command; promote is the shared unit (dwarves::release_doctor::promote, driven by promote_one_repo in src/bin/nornir.rs) that both publish <repo> and the doctor --publish cascade call per repo. One promote does, in order: regenerate docs → run the repo's enabled gates → bump versions (reversibly recorded) → strip [patch.crates-io] blocks → cargo publish each crate → wait-for-index → (on a real run) commit + push the bumped manifests. Understanding "promote" as the unit is what makes the cascade legible: the cascade is promote looped over repos in topo order.

Supporting ops

• wait-for-index <crate> <version> [--timeout-secs 300] — poll crates.io until the version is max_version (item #3's fix; replaces a fixed sleep).
• undo [release-id] [--dry-run] [--yank] — reverse a run's mutations from the release_changes ledger newest-first: un-bump (restore the exact pre-bump manifest), restore a stripped [patch], delete/reset a release branch. A crates.io publish is immutable, so it is yanked (with --yank) rather than removed.
• run [--repo …] [--skip-tests|--skip-bench|--skip-gates|…] — the test → bench (persist) → gate-all orchestration in topo order (no publish).
• trace, events, changelog, impacted-crates, tarball-stats, strip-patch-blocks, yank-cascade, mirror-to-holger, bump-version — the introspection + surgical-edit verbs.

2. The deps-first topological cascade

The release order is not hand-maintained. graph_math.rs builds the crate dependency graph, runs Tarjan SCC to find cycles, condenses each SCC to a node, and topologically sorts the condensation into publish waves: a crate is only published after every crate it depends on is already live and indexed. This is why wait-for-index exists — a dependent's cargo publish verify build resolves its dependency FROM crates.io, so the dependency must be indexed first or the dependent fails to build.

The cascade is FAIL-STOP: promote::run records per-crate publish failures in its report and returns Ok, so the cascade explicitly inspects them and halts rather than sailing on to a dependent whose dependency never shipped. A halted cascade leaves the already-shipped crates live-and-immutable and the rest unshipped — a torn release, which is exactly what preflight (§3) and the registry-state check (§5) exist to prevent, and what --publish-from + --bump keep exist to recover from.

Cycles that block the topo sort are reported; graph_math's MFAS ranking proposes the cheapest set of edges to cut, and doctor --auto can apply the cheap cuts.

3. Preflight — refuse to start a red cascade

cargo publish is irreversible. dwarves::release_doctor::preflight moves every catchable failure BEFORE the first real upload: it validates the WHOLE plan and refuses to begin unless clean (or --force). One report — preflight: N blocking issue(s) — lists every blocker with its exact fix. The checks:

1. membership — every planned -p target is a member of its repo-root workspace (catches the detached-[workspace] bug).
2. metadata — required crates.io fields present per crate (description, license/license-file, repository), resolving field.workspace = true inheritance against [workspace.package].
3. dep-API skew (verify) — cargo publish -p <crate> --dry-run per crate; a compile error citing an already-published sibling is caught up front. See §3.1 — this check is now bump-aware.
4. detached workspaces — listed as notes, so intentional exclusions are visible.
5. version/registry clash — for a real bump, the target version must not already be live on crates.io (an immutable clash); for --bump keep, an already-live version is a benign idempotent skip.
6. registry-state desync — the direct question, see §5.

3.1 Bump-aware preflight (item #7)

The trap it fixes. Under --bump patch, the cascade publishes crates in dep order at their about-to-be-bumped versions. But the naive verify (check 3) builds each crate against the CURRENTLY-published versions of its siblings. So a crate B that needs sibling A's new API fails cargo publish --dry-run — A's new API is only in the local, not-yet-published A vN+1. The real cascade publishes A vN+1 FIRST and waits for it to index, at which point B resolves the new API. The naive preflight had no way to know that, and reported ~44 false blockers. The real fat release only got through with --force.

How it now works. The --publish site in src/bin/nornir.rs builds a planned publish map — the union of every repo plan's publish_set, each crate mapped to the version the cascade WILL publish it at (current local version + bump):

• src/bin/nornir.rs — let cascade_publishes: BTreeMap<String,String> = … built from plans.iter().flat_map(|p| p.publish_set.iter().cloned()) (empty for --bump keep).
• dwarves::release_doctor::preflight::run takes this map as its final cascade_publishes param and threads it into the verify loop.
• dwarves::release_doctor::preflight::skew_is_cascade_resolvable(dep, cascade_publishes) is the pure decision: it returns Some(planned_version) when the crate the skew BLAMES is in the planned map (⇒ the cascade republishes it forward ⇒ NOT a blocker), and None otherwise (⇒ a real blocker). An unnamed dep is conservatively a blocker.
• dwarves::release_doctor::preflight::verify_one returns a VerifyOutcome: Clean, Blocker, or CascadeResolvable { dep, planned }. A CascadeResolvable skew is recorded as a NOTE (… verify skew against \dep` is cascade-resolvable (dep will be published at vN+1 earlier in the wave) — not a blocker`), never pushed as a blocking issue.

What still blocks (real blockers preserved):

• a verify skew blaming a crate NOT in the release set (external, or one the cascade won't republish) — a genuine unpublished-API failure;
• a missing-metadata gap (check 2), e.g. a crate without repository;
• a version already live on crates.io for a real bump (check 5);
• a local < published DESYNC from the registry-state check (check 6, §5);
• --bump keep|none passes an EMPTY planned map, so the strict behavior is preserved: an edited-but-unbumped / AlreadyPublished sibling still blocks (this keeps the item #9 / #10 resume semantics intact).

Net effect: nornir release doctor --publish --bump patch passes preflight WITHOUT --force on a coherent tree, while a genuinely missing external dependency still stops the cascade before any irreversible upload.

Tests (dwarves::release_doctor::preflight, #[cfg(test)] mod tests): skew_is_cascade_resolvable_only_for_in_release_deps (a skew blaming an in-release sibling like facett_core clears with its planned version, while a skew blaming an external crate like serde still blocks, and an empty planned map suppresses nothing) and cascade_resolvable_skew_becomes_a_note_not_an_issue.

4. The config: release/nornir.toml

Loaded by src/config.rs. The workspace root is the config's grandparent-of-grandparent (…/workspace_<name>/release/nornir.toml → …/), and every relative path inside is resolved against it.

# One block per repo the cascade knows about. A repo present on disk but WITHOUT a
# [repo.<name>] block is NOT in repo_order → it is silently excluded from the cascade.
[repo.facett]
path = "facett"                 # on-disk dir relative to workspace root (default: <name>)
remote = "…"                    # doc-link metadata only (README/CHANGELOG "repository")
private = false                 # true ⇒ NEVER published to crates.io; promote hard-refuses
test_features = "gui,testmatrix"# extra cargo features for `nornir test run` (NORNIR_TEST_FEATURES)
publish_order = [["a","b"], ["c"]]  # optional explicit wave override

[repo.facett.gates]             # per-repo release gates (all default false / off)
no_path_patches = true
nexus_floor = true
no_regression = true
max_regression_pct = 5.0
core_saturation = true          # CORE-SATURATION LAW: fail if a parallel bench under-used cores
min_core_busy_frac = 0.9
docs_fresh = true               # README sections in sync
guard_intact = true             # [guard].forbidden paths unchanged
tests = true                    # run the repo test MATRIX at release (heavy)
coverage = true                 # AUT5 completeness: every surface covered or allowlisted (light)
integration_roundtrip = ["kind"]

[repo.facett.bench]
required_results = ["text_500mb", "rust_crate_mt32"]  # the bench corpus schema
network_required = false

# The private nordisk repos — SSH-cloned, never public.
[repo.nornir]
private = true
[repo.korp]
private = true

# The registry-STATE-check backend (item #10). Default = crates.io; no config needed.
[registry]
backend = "crates-io"           # or "custom"
url = ""                        # sparse-index base when backend = "custom" (private/holger mirror)

[guard]
forbidden = ["path/to/protected"]   # guard_intact gate watches these

[storage]
kind = "local"                  # or "remote" (Flight endpoint via remote_url)
local_path = ""                 # empty ⇒ <home>/.nornir/workspaces/<name>/builds

Gate names (for nornir release gate <name> <repo>): path_patches, path_dep_versions, crate_metadata, links_conflicts, edition, nexus_floor, no_regression, docs_fresh, roundtrip, guard_intact, coverage, all.

5. The registry-state check (item #10)

Verify (check 3) builds against the currently-published siblings, but never asks the registry the ONE direct question: what version is actually published? So a local-vs-published DESYNC slips through — the local tree carries gatling 0.1.4 while crates.io already serves 0.1.6 (a teammate's or a prior partial cascade's upload). A dependent then verify-builds against the newer published API and breaks mid-cascade.

dwarves::release_doctor::registry adds a pluggable RegistryBackend trait (published_max_version(krate) -> Option<String>) with two impls:

• CratesIoBackend (default) — the crates.io sparse index (https://index.crates.io/<ab>/<cd>/<crate>, cargo's index-path convention), parsing newline-delimited JSON for the semver-max NON-yanked vers; falls back to the JSON API (https://crates.io/api/v1/crates/<crate>). A descriptive User-Agent is baked in — crates.io returns HTTP 403 to a UA-less request, which is why a bare curl fails.
• CustomBackend { base_url } — a configurable sparse-index base, to point the check at a private registry / holger mirror.

Both use pure-Rust ureq. The pure classify(local, published) -> RegistryState maps: local > published → WillPublish (good); local == published → AlreadyPublished (the cascade skips it); local < published → Desync (the bug — a blocker); published = None → Unpublished; unparsable → Unknown (never a false block).

Wiring: preflight check 6 turns a Desync into a blocking [desync] issue. Because it's a first-class issue, it flows into the topological root-cause report (§6) — a desync crate is a node, so dependents that blame it attach as its downstream. Config: the [registry] section; CLI override --registry-url <base> (the flag wins).

6. The topological root-cause report (item #9)

A flat preflight list is often ONE problem seen from N angles: a single dependency edited-but-not-published makes every dependent fail its verify ("uses API from gatling that isn't in the PUBLISHED gatling"). dwarves::release_doctor::root_cause partitions the failure set against the publish dependency graph: a failure is a ROOT iff none of its transitive publish-deps also failed; everything else is DOWNSTREAM, attributed to the root(s) it depends on. Multiple independent roots are supported.

Two edge sources combine — the doctor's already-computed repo_edges (reused as a WorkspaceGraph) AND each failure's blames (the dep a verify error names) — so the true root is recovered even when it was SKIPPED and never became its own failure (the gatling/AlreadyPublished case). The report leads with:

ROOT CAUSE(S) — fix these, in order:
  ⛔ gatling — <reason> → unblocks: lbzip2, lgz, ljar, lzip-parallel, osm-katana, znippy-zoomies
(6 downstream failures explained by 1 root)

The flat per-issue list (with fixes) stays below it.

7. The end-to-end flow

The safe path from "nothing" to a published constellation:

1. doctor — read-only assessment. Fix any genuine metadata/version-skew blockers (--fix, --auto, or by hand). This is where you add a missing repository field or a version = on a path dep.
2. stage then stage --execute — rehearse the whole cascade against holger's /sparring registry. Nothing becomes immutable; this is the last dress rehearsal that exercises real cargo publish + real dependent builds.
3. doctor --publish --bump patch --publish-dry-run — the plan + cargo publish --dry-run end-to-end against the real preflight (now bump-aware), no upload.
4. doctor --publish --bump patch — the real cascade. Preflight runs first (and, with the bump-aware fix, should be clean without --force on a coherent tree); then the deps-first cascade publishes wave by wave, waiting for each crate to index before its dependents, fail-stopping on any real failure.

If step 4 tears: recover with doctor --publish --bump keep --publish-from <first-unshipped-repo> — idempotent, it ships only the not-yet-published crates without re-bumping the rest. release undo [--yank] reverses the mutations of a run.

8. Failure modes (what the errors mean)

• [desync] local gatling 0.1.4 is BEHIND the registry (published 0.1.6) — the registry has a newer version than your tree (§5). Bump past the published version so the local tree leads the registry before starting the cascade. This is a hard blocker because a dependent would otherwise verify against the newer published API and tear the cascade.
• [version] gatling 0.1.2 is already live on crates.io (immutable) — the target (post-bump) version already exists. crates.io versions can't be overwritten; bump past it, or use --bump keep to skip already-published crates. (Note: under a bump this targets the POST-bump version, so a mere "current version already live" is not flagged.)
• **[verify] uses API from \facett_core` not in the published `facett_core`** — dep-API skew. If facett_core` is IN the release set, the bump-aware preflight now records this as a NOTE (the cascade publishes it first) — NOT a blocker. If it is NOT in the release set, it is a real blocker: bump and publish that dependency first.
• [metadata] missing crates.io metadata: repository — a genuine gate failure. Add the field (e.g. repository.workspace = true) to the crate's [package]. The 6 knut crates in the real fat release (graphar, graphar-falkordb, graphar-flight, knut-bifrost, knut-pipelines, knut-popviz) hit exactly this.
• [non-member] not a member of the workspace at … — a detached [workspace]. Mark it publish = false or publish it from its own workspace directory.
• Missing catalog entry — a publishable crate/repo that has no [repo.<name>] block in nornir.toml is never scheduled into repo_order, so it silently never publishes (a member added after the first stage, e.g. facett-graphnav, is the same class — item #2). Add the block, or verify the crate appears in stage's printed plan. The AUT5 coverage gate is the analogous "surface with no catalog/allowlist entry" check for test coverage (e.g. an autonom surface uncovered and un-allowlisted fails the gate).
• Unpushed bumps — after a green publish, the per-repo bumped Cargo.toml/Cargo.lock edits must be committed + pushed so origin (and the dwarf server's pull) reflect the published versions (item #5). The real cascade commits + pushes on a non-dry run; if you interrupt it, push the bumps by hand.
• cargo publish spawn failed: No such file or directory — cargo not on PATH (item #1). source ~/.cargo/env; the resolver now also tries $CARGO / rustup which cargo / ~/.cargo/bin/cargo and pre-flights one clear error.

8.5 nornir build — the declarative build-thing (variants + the suite driver)

Distinct from the release machine above, nornir build drives the nornir-build-thing cells (.nornir/build-variants.toml) — the data that says what a repo can build.

• nornir build --variant <name> — build one declared variant (its exact cargo build -p … --bin … --features … --profile … argv). --list-variants lists them; --dry-run prints the argv without running; --json emits the build-thing surface. --image/--vm/--installer <name> drive the engine targets (container / VM / bootable ISO).

• nornir build --all — the ONE suite build driver. Walks every [repo.<name>] in the loaded roster (workspace_nordisk/release/nornir.toml), discovers each repo's .nornir/build-variants.toml, VALIDATES each cell, and builds every CARGO variant (Exe / Lib / Wasm / Service) with its declared feature set. Container/VM/ISO targets are skipped (drive those per-repo with --image/--vm/--installer).

  • --dry-run — enumerate the whole-suite cargo plan without compiling.
  • --role <server|client|library|tool> — build just one role across the suite.
  • --no-dist — skip the canonical-name landing (below).
  • A repo with no cell (a build-tooling gap) or no checkout is LOGGED and skipped, never a hard error — --all reports coverage. Exits non-zero only if a declared variant actually fails.

• Canonical named landing (dist/). After a successful Exe/Service build the binary is hardlinked (else copied) into <repo>/dist/<variant.name> — the human-readable canonical artifact name. The target dir is resolved honoring CARGO_TARGET_DIR and a repo-local .cargo/config.toml [build].target-dir (so a repo like znippy that redirects cargo to ~/.cache/znippy-target still lands its dist/ copy). Pure-fs; the only subprocess is cargo itself.

Every repo's cell + this driver are the subject of ../../workspace_nordisk/.nornir/suite-build-inventory-and-scheme-2026-07-30.md.

8.6 The build-version LAW across nornir's binaries

Every nornir binary embeds its identity in --version, from the SAME pure-Rust, zero-shell git stamp (build.rs reads .git/HEAD + the resolved ref off disk → NORNIR_GIT_HASH; lib.rs builds VERSION_LINE = v<pkg> (<git-sha>)). Beyond the main nornir bin and nornir-mcp, the auxiliary bins now comply too: nornir-server --version and nornir-ui --version print nornir::VERSION_LINE; nornir-viz-web --version prints its own git-stamped line (that crate is not a member of the nornir crate, so it carries its own minimal pure-fs build.rs stamping NORNIR_VIZ_WEB_GIT_HASH).

9. Cross-references

• nornir-release-improvements.md — the field log of imperfections and their fixes (items #1–#10). This manual documents the shipped state of #7 (bump-aware preflight), #9 (root-cause), and #10 (registry-state).
• release-design.md, release-doctor-design.md, release-graph-math.md, release-publish-traps.md, release-rehearsal.md — the design-level companions.