nordisk/holger

Holger — Promotion (dev → staging → release)

Job ledger (2026-07-12): every promotion is now recorded as a durable jera job (running → done|failed + timing) when a ledger is attached — see consolidation-jera-embed-design.md §1. The storage transition (seal_inner) is byte-for-byte unchanged; the ledger is opt-in and additive (PromotionEngine::with_job_ledger).

Status: DESIGN — ratifiable. Scope narrowed to the one feature we're building now: promotion. Grounded in current code (file:line). Supersedes the promotion sketch in artifactory-parity-design.md §DESIGN (c); the broader parity gap-list stays there as context. Decision owner: Rickard.

The headline: promotion is the holger-side counterpart to nornir's release sparring. nornir already cascades a release into /sparring and verifies it builds; promotion is what makes the result durable and auditable — it turns a green sparring rehearsal into an immutable, sealed, signed release generation, by a pointer move, not a rebuild.

Executive summary

Modern software isn't a binary you copy to a server — it's a dependency closure pulled from dozens of registries, built by CI, scanned for CVEs, attached to an SBOM, and shipped into environments that are increasingly regulated, audited, and air-gapped. The hard questions a team must answer today are no longer "where's the file": they are "is this exact artifact the one that passed the gate? what's in its dependency tree? did it come from clean, pushed source? can I prove that offline, in a secure facility, six months from now? and if it's bad, can I pull it back instantly?"

Promotion is the discipline that answers those questions: build an artifact once, then move that same immutable bytes through quality stages (dev → staging → prod) and across environments (test1 → test2 → driftmiljön), gated at each boundary, never rebuilt. Rebuilding for production ships something you never tested; copying blindly ships vulnerabilities and broken dependency trees. A team needs:

• Atomic releases — a multi-crate release lands all-or-nothing, so a failed cascade never strands half a release in production.
• No broken releases — an artifact can't be promoted into an environment where its dependency closure doesn't resolve.
• Clean provenance — only artifacts built from committed-and-pushed source, with a vetted (CVE/license-clean) dependency set, cross a boundary.
• Self-describing, verifiable units — the SBOM, the policy verdict, the dependency closure and the signature travel with the artifact, so its integrity and origin are provable offline, with no connection back to a build server.
• Reversible, auditable history — every release is kept and addressable; rollback is instant; "what shipped, when, who promoted it, and why" is always answerable.

This document designs promotion for Holger so these become properties of the release unit itself — an immutable, content-addressed, signed .znippy that is the registry — rather than fragile state living inside one server. §13 enumerates the concrete capabilities this unlocks; the rest of the document specifies the mechanics.

0. Why this is cheap (the one good reason to do it now)

znippy archives are already immutable + content-addressed (blake3 per chunk, random-access extract_file). The writable dev-mode backend already exists (RustRepoZnippy::with_writable_archive → put appends via znippy_common::meta_sink_append::append_files, server/repository/znippy-rust/src/lib.rs:43,349,368). And seal() — the finalizer that freezes a live store into a static snapshot — already exists in znippy (znippy-iceberg/src/seal.rs:69, pub fn seal(sidecar, warehouse, namespace, out) -> SealReport).

So promotion is ~80% substrate-complete. What's missing is small and specific:

1. holger never calls seal() — it's referenced in comments only (server/lib/src/repository.rs:50,87,228, lib.rs:181) but not wired. This is the core primitive to land.
2. There is no promotion verb — proto has RepositoryService / ArchiveService / AdminService (proto/holger.proto) but no PromotionService.
3. There is no archive manifest — the immutable generation carries no record of what's in it / who sealed it / what gate verdict it passed. Promotion needs that record to be the pointer it moves.

Everything else (CAS dedup, full policy gate, multi-ecosystem) is deferred — see §7.

1. Model — stages are repos, promotion is a pointer move

  rust-dev                 rust-staging                 rust-prod
  (writable znippy)        (writable znippy,            (SEALED immutable
   live ingest             quarantine / soak)            znippy, read-only)
        │   promote --to staging        │   promote --to prod
        └──────────────►────────────────┴──────────────►─────────►
              gate (warn)                      gate (block) + seal()

• A stage is a repository in the RON config — same RustRepoZnippy backend, three instances. rust-dev and rust-staging are with_writable_archive (writable, append-on-put); rust-prod is the read-only sealed archive (with_archive, is_writable()==false).
• Promotion does not copy bytes. The artifact bytes already live in the source archive, content-addressed. Promotion records the artifact's identity in the target stage's next generation and (at the prod boundary) seal()s that generation into an immutable snapshot. Bytes move once, at the original put; never again.
• This is exactly nornir's flow seen from holger: nornir release stage publishes the cascade into /sparring (= rust-staging) and build-verifies it; promote --to prod + seal() is the durable, gated finalization of that rehearsal.

Promotion record (the audited unit)

PromotionRecord {
  artifact_id:        ArtifactId,        // {namespace, name, version} — traits/src/lib.rs:7
  blob_sha256:        String,            // content address (= the thing that moves)
  from_repo:          String,            // "rust-staging"
  to_repo:            String,            // "rust-prod"
  from_generation:    u64,
  to_generation:      u64,               // the new sealed generation in the target
  gate_verdict:       Verdict,           // pass | warn | block (+ reasons) — §4
  promoted_by:        String,            // mTLS CN or OIDC sub — auth.rs identity
  promoted_at:        i64,               // unix ts (passed in; no Date::now in lib)
  archive_sha256:     Option<String>,    // set when the target generation is sealed
  signature:          Option<String>,    // set when sealing signs (deferred — §7)
}

2. The seal primitive (land this first — it unblocks everything)

Goal: a writable dev-mode znippy store (/sparring, rust-staging) can be frozen into a static read-only "drift" snapshot that the prod backend serves.

Wire seal() into the backend. Add to RustRepoZnippy (or a thin SealableBackend seam on the trait) a method:

// server/repository/znippy-rust/src/lib.rs — new
/// Freeze this writable archive into an immutable sealed generation at `out`.
/// Returns the archive sha256 of the sealed snapshot. No-op error if read-only.
fn seal_to(&self, out: &Path, ts: i64) -> anyhow::Result<SealOutcome>;

• For the arrow-ipc-direct writable archive (today's /sparring, append_files), sealing = finalize the manifest/footer + compute archive_sha256 + atomic-rename to out. The append path is already reboot-safe, so seal is a finalize, not a rebuild.
• For the iceberg-skade flavor, delegate to znippy-iceberg::seal(sidecar, warehouse, namespace, out) (seal.rs:69) which already returns a SealReport.
• After seal, the prod repo opens out read-only via the existing static path (with_archive, znippy-rust/src/lib.rs), so the serve hot path is unchanged.

Acceptance: a unit test that puts two crates into a writable archive, seals it, reopens the sealed out read-only, and serves both crates' sparse index + download — i.e. the existing cargo_publish_stores_and_serves test, then sealed.

3. The promotion verb (gRPC + CLI)

proto — new PromotionService (proto/holger.proto, alongside the three existing services):

service PromotionService {
  rpc Promote(PromoteRequest) returns (PromoteResponse);            // move one artifact (or bundle) across one boundary
  rpc ListStaging(ListStagingRequest) returns (ListStagingResponse); // what's sitting in a staging repo
  rpc PromotionHistory(PromotionHistoryRequest) returns (PromotionHistoryResponse);
}
message PromoteRequest {
  ArtifactId artifact = 1;     // or...
  string bundle = 2;           // optional release-bundle name@ver (atomic multi-artifact)
  string from_repo = 3;
  string to_repo = 4;
}

handler (server/lib/src/grpc.rs, new impl PromotionService for Arc<HolgerGrpc>):

1. authorize_write on the target repo (reuse the existing gate, grpc.rs:49; per-repo RBAC is W2/A4, deferred — for now any valid writer).
2. resolve source backend + target backend via FastRoutes / Holger.
3. confirm the artifact exists in from_repo (backend.fetch(id).is_some() / contains), capture its blob_sha256.
4. run the gate for this boundary (§4).
5. if pass (or warn + non-enforcing): record the artifact into the target's next generation; if the boundary is the prod boundary, seal_to() it (§2).
6. build the PromotionRecord, emit it to the audit sink (§5), return it.

CLI (holger binary — the server/cli side, plus mirrored in agent/cli if driven internet-side):

holger promote <name@version> --from rust-staging --to rust-prod [--bundle name@ver]
holger staging list --repo rust-staging
holger promote --history [--repo rust-prod]

promoted_at timestamp is supplied by the binary (CLI/server entrypoint), not the lib — the lib must stay Date::now-free (matches the existing constraint noted across the workspace).

3.1 SHIPPED — holger-server promote is a LOCAL verb (not a gRPC client)

The RPC above shipped with a real engine behind it, but for a while nothing in any binary called it: the only PromoteArtifactRequest constructor was a #[cfg(test)] helper, so an operator could not trigger a DEV→PROD promotion at all. The verb is now wired — as a local, in-process command in the exact style of replicate, not as a gRPC client (the holger CLI has no client path; it is entirely RON-config + local-file driven).

holger-server promote --from <dev.ron> --to <prod.ron> \
    --repo <source-repo> [--to-repo <dest-repo>] \
    [--namespace <ns>] --name <name> --version <ver> [--execute]

• --to-repo defaults to --repo (repos pair by name, as replicate does).
• Dry-run by DEFAULT. The gates still run and the verdict prints, but nothing is written; --execute seals. A promotion is irreversible, so the safe default matters more here than for replicate.
• Flow mirrors the handler so behaviour matches: fetch from the source repo → policy gate → property requirement → immutability check → seal.
• The PROD store and both gates come from Holger::promotion_engine() — the ONE derivation serve uses (<data_root>/holger-prod-store), so the local path cannot seal into a different store or under weaker policy. It is fail-closed: a configured-but-unreadable advisory feed refuses the promotion.

ACCESS CONTROL — the trade-off, stated plainly. The gRPC path enforces an ADMIN authz gate scoped to the destination repo and records promoted_by from the authenticated identity (mTLS CN / OIDC sub). The local path's only gate is filesystem permissions on the configs and the store, and promoted_by is the honest OS user ($USER, else $LOGNAME, else uid:<euid> — never anonymous, never a faked admin subject). This mirrors replicate, which is already local-only; a reader must not mistake the local verb for an access-controlled one. Use the served path when the promotion must be authorised. The POLICY gates (HOLGER_PROMOTE_ENFORCE, HOLGER_PROMOTE_REQUIRE_PROPERTY) are not relaxed locally — skipping them would make the CLI a way around the boundary the server enforces.

Refusals carry the frozen holger-promote-<n> codes (.nornir/holger-error-codes.md), so a script asserts the CODE, not the prose: holger-promote-1 already promoted, holger-promote-2 policy blocked, holger-promote-7 artifact not in the source repo. Proven end-to-end in server/cli/tests/promote_cli.rs.

4. The gate seam (design now, minimal impl)

Promotion runs a gate at the boundary (the Xray model: gate once on crossing; the prod repo then holds only pass-verdict artifacts, so serve stays hot).

Design the seam fully; implement a pass-through verdict first.

// server/lib/src/promote/gate.rs — new
struct PolicyGate { enforcement: Block | Warn | Off, /* thresholds — §7 */ }
fn evaluate(&self, blob_sha256, format) -> Verdict { pass | warn | block, reasons }

• Per-boundary config in RON: dev→staging = Warn, staging→prod = Block.
• v1 verdict = pass (gate is wired, thresholds are no-ops) — this lets promotion + seal + audit land and be tested end-to-end without dragging in nornir::security.
• v2 (deferred, A3): evaluate calls nornir::security::scan / cached vuln_findings for CVE + license, caches the verdict on (blob_sha256, policy_version, advisory_db_rev). That's the artifactory-parity-design.md §DESIGN (b) work — not in this slice, but the seam is shaped for it now so it drops in without touching the promote handler.

5. Audit (the record is the point)

The "auditability is the point" pitch demands that every promotion is logged. Minimal v1, expandable:

• v1: append each PromotionRecord as a JSON line to a promotion log under the data root (<root>/holger/promotions.ndjson), atomic O_APPEND — same durability pattern as the writable-archive append. Reboot-safe, greppable.
• v2 (deferred, A5): also historize to the nornir warehouse table holger_promotions (the parity doc's lineage plane). One emit point; the warehouse sink is added behind it later.

Promotion is the first holger event worth auditing — landing the audit log here seeds Gap J without a separate project.

6. Build phases (in order — each independently testable)

Deferred out of this slice (tracked, not now): real policy gate via nornir::security (A3/§4-v2); warehouse historization (A5/§5-v2); signing the sealed manifest (ed25519/cosign); per-repo RBAC on promote (W2/A4 — promote currently needs only "a valid writer"); retention/GC of old generations; multi-ecosystem (this slice is rust-only, matching the sparring use).

7. Acceptance test — the arrow-58 sparring rehearsal, made durable

Reuse the §O acceptance case end-to-end:

1. nornir release stage cascades the arrow-58 release into /sparring (= rust-staging) and build-verifies it (existing flow).
2. holger promote <each crate> --from rust-staging --to rust-prod → gate (pass) → seal() → one immutable rust-prod generation.
3. A stock cargo build against rust-prod's sparse index resolves the whole arrow-58 set from the sealed archive (read-only hot path).
4. holger promote --history shows the full promotion record set; the promotions.ndjson survives a process restart.

Green = promotion turns a sparring rehearsal into a durable, audited, immutable release with zero rebuild and zero crates.io risk.

8. Open questions for ratification

1. Seal granularity — does each prod promote seal a new generation (append + re-seal, N generations accumulate) or re-seal the whole repo? Lean: new generation per promote, old generations retained until retention (P-later).
2. Bundle atomicity — RESOLVED (see §10.1): the seal is the commit. A release bundle = one sealed generation; seal() is the all-or-nothing transaction. One generation per release. This is the capability cargo structurally lacks.
3. Where promote is driven from — holger (server-side CLI, secure side) vs agent (internet side). Lean: server-side, since prod lives secure-side; agent only stages.
4. iceberg-skade vs arrow-ipc-direct for the staging store under seal — both work; arrow-ipc-direct is the leaner default already wired for /sparring.

9. Appendix — promotion/staging feature matrix across the field

The full canon of promotion/staging functionality, mapped across the field, so we build against the whole doctrine and not just Artifactory's slice. Legend: ✅ full · 🟡 partial · ❌ none. Columns: Artf=Artifactory · Nex=Nexus staging · Harb=Harbor · k8sP=k8s image-promoter · npm=dist-tags · Deb=Debian dak · Bod=Fedora Bodhi · Holg→=Holger target.

A — Stage & promotion mechanics

B — Gates (what can block a promotion)

C — Lifecycle & bundling

D — Audit / provenance / events

E — Storage & integrity

F — Access

The one row nobody else has: airgap-carriable single signed release unit. Everyone else's "release" is a state inside a live server; Holger's sealed .znippy is the release as one carriable file — the differentiator, and why seal() is the keystone (§2), not an afterthought.

10. Sharpened mechanics (resolved decisions)

10.1 Atomic release — the seal is the commit (resolves §8 Q2)

A release bundle = one sealed generation holding all N crates; seal() is the transaction commit — all-or-nothing. Pre-seal half-states live only in the writable staging archive; prod is untouched until the seal lands. Cascade dies mid-way (reboot, a failed build-verify) → nothing sealed → nothing released → resume/retry, master clean.

This is the capability cargo structurally lacks: cargo publish is N independent, non-atomic calls; a mid-cascade failure strands you in a torn, half-published, undeletable state (versions are immutable, yank-only). Holger's seal makes the whole bundle appear atomically.

Boundary (be honest): atomicity holds within Holger. The eventual Holger→crates.io push (Tier-3 promote --brave) re-enters cargo's world — that hop stays best-effort topo-order; the limitation is crates.io's, not Holger's. So: airgap/private = atomic + gated + rollback-able; public crates.io = best-effort.

10.2 Build-info / provenance — it follows the artifact via the manifest

Build-info is associated metadata (git commit, rustc, build env, who/when, resolved deps), not bytes inside the .crate. It follows the artifact in two complementary places:

• (a) the archive manifest beside the blob — the manifest entry gains source_commit, rustc, build_env next to created_by. Travels with the sealed .znippy across the airgap; served by ArchiveManifest.
• (b) the nornir warehouse — the historized, queryable copy (holger_artifacts / holger_archives), the analysis plane.

"Why can't nornir embed it in the executable?" It can — but that's a different artifact for a different purpose. A .crate is source; there is no executable to embed into, so its provenance belongs in the manifest. For a built binary/firmware (the airgap-delivered executable), nornir can embed build-info into the binary at build time (linker section / built-crate constants / a SLSA provenance blob) so it travels even after leaving Holger. Two layers: registry provenance (manifest — this slice) vs binary-embedded provenance (nornir build-time — SLSA track, deferred).

10.3 SBOM — Holger can host it (it just doesn't yet)

No blocker; only unbuilt. nornir::security already emits CycloneDX (to_cyclonedx()). Holger hosts it the OCI-referrer way: the SBOM is a blob attached to the artifact's content address (keyed by blob_sha256), recorded in the manifest as an attachment, served by GetSbom. nornir produces; Holger stores + serves. The same referrer mechanism later carries the signed provenance and the policy verdict.

10.4 Retention without rewrite — deletion-vector overlay (Rickard)

Sealed archives are immutable — you can't delete a file without rewriting GBs. So a new generation is a thin overlay, not a rewrite:

Generation {
  base_archive_sha256: String,     // parent pointer → the root znippy id
  deletion_vector:     Set<...>,   // tombstoned artifact-ids / blob-shas
  new_blobs:           [...],      // only what this generation adds
}

Reads resolve through the overlay: tombstoned → 404; else fall through to the base. A "clone" is metadata-only and cheap; generations chain (git / iceberg snapshot style). Space reclaim is a later compaction that materializes a fresh base = root minus deletion-vector — until then nothing is rewritten. Aligns with znippy-iceberg's existing deletion-vector model; reboot-safe (append the overlay).

Free bonus: demotion/rollback becomes trivial — tombstone a promoted artifact-id in the prod overlay = instant demote, no rewrite (upgrades the matrix: Retention ❌→🟡, Demotion 🟡→✅).

10.5 Signing — CMS over the manifest (the set of blake3 hashes), not the bytes

znippy already computes blake3 per file/chunk on write (mandatory, non-optional). The manifest commits to every file's blake3. So sign the manifest digest once — that transitively signs all content without re-hashing the bytes:

• Fast path: CMS / PKCS#7 over the manifest, rooted in the same X.509 PKI that roots mTLS (one trust root for transport + artifact signing); or ed25519 for a lighter signature. The signed object = the manifest = the list of blake3 hashes.
• Verify = recompute manifest digest → verify CMS/ed25519 sig → blake3-verify-on-read covers the actual bytes (already mandatory). This is the cosign / in-toto model (sign the manifest, not the layers), but it reuses znippy's existing hashing, so it's near-free.
• The signature lands in the PromotionRecord + archive manifest (§1), making the §6 "manifest signing" deferred item specified, not TBD.

11. Two axes of promotion — stage (vertical) vs environment delivery (horizontal)

11.0 Terminology (resolved — supersedes "static / developer-mode")

"dynamic" replaces "developer-mode"/"dev-mode" everywhere in the holger docs; "static" stays (a.k.a. "drift").

11.1 The two axes (they compose; the seal is the pivot)

Promotion runs in two orthogonal directions:

1. Stage promotion (vertical) — within ONE dynamic Holger: rust-dev → rust-staging → rust-prod repos. Seal-at-prod = the commit (§2, §10.1).
2. Environment delivery (horizontal) — a Holger seals a generation and pushes the .znippy forward to the next environment's Holger. This is a chain, not a single hop: test1 → test2 → … → prod/drift. Each hop's target may be another dynamic Holger (e.g. test1 → test2) or a static drift node in driftmiljön (the airgap endpoint — USB/optical or a guarded link). Same mechanism; transport and target flavor vary along the chain.

The seal is the pivot between the axes. Vertical promotion's output (a sealed prod generation) is horizontal promotion's input (the unit delivered to the drift node). One mechanism feeds the other; the "airgap-carriable single signed release unit" (the §9 differentiator row) is exactly this handoff object.

11.2 How a read-only static node "receives a push"

A static Holger never writes artifacts (znippy is immutable by construction). So "dynamic pushes to static" = deliver a sealed generation + the drift node adopts it as its served generation. The drift node's only write is the generation-pointer swap (which archive it serves) — atomic, reboot-safe, auditable. Never an artifact write.

Two delivery modes (reuse the §10.4 generation chain):

• Standalone — a fully self-contained sealed .znippy (no base pointer). Bigger, but self-sufficient: best for a cold/fresh drift node or where the base isn't guaranteed present.
• Incremental overlay — {base_archive_sha256, deletion_vector, new_blobs} delivered against a base the drift node already holds. Smaller; grows the chain. Requires the base present.

11.3 The adoption gate — verify before swap (where §10.5 signing pays off)

The drift node must verify before it swaps. The §10.5 manifest signature (CMS/ed25519, PKI-rooted in the same CA as mTLS) is the adoption gate: the static node accepts a new generation only if the signature verifies against its trusted CA. Signing is therefore not polish — it is the trust gate of horizontal promotion across the airgap. (Vertical promotion uses the §4 policy gate; horizontal promotion uses this signature gate. Both run once, at a boundary.)

11.4 Relation to replication (subsumes the Gap I / W7 question)

Inter-instance promotion is the airgap-native form of replication: delivery of a signed, sealed unit — not live push-replication. A connected variant (drift node pulls, or the dev node pushes over a guarded one-way channel) is the same model with a different transport. This folds the "geographically dispersed Holgers" question (parity Gap I) under promotion's horizontal axis — same mechanism, transport varies. Tracked here, built after the vertical axis lands.

12. Cleansing — what crosses a boundary must be complete, vetted & provenance-clean

A promotion does not move a lone artifact. What crosses any boundary (vertical stage or horizontal hop) is a cleansed set: its dependency closure, vetted, and proven to come from clean source. Cleansing is the substance of promotion; the pointer move is just the mechanics.

12.1 Dependency cleansing (four checks)

When a crate is pushed forward (test1 → test2, or dev → prod), its dependency closure must be cleansed for the target:

1. Closure-complete — the full transitive dep set must exist in (or be promoted alongside) the target. Promote X ⇒ X's whole dep tree resolves in the target. The release bundle (§10.1) is the dependency-closed set, never one crate.
2. Gate-clean — every dep in the closure passed CVE / license / forbidden — not just the top crate. A clean X with a vulnerable transitive dep is not clean.
3. Source-normalized — strip back-references to the source environment. A crate resolved against test1's registry must resolve from the target's index in test2 (rewrite source.replace-with / Cargo.lock provenance). No pointer home.
4. Pruned — drop dev/build-only deps not needed downstream; tombstone forbidden / superseded / yanked versions so the target doesn't accumulate cruft (the arrow-58 case: promoting arrow 58 must cleanse arrow 56 out — exactly the §10.4 deletion-vector).

Already half-modeled: §O Tier-2's "cargo build the next dependent against the embedded holger sparse index" is the closure-resolves proof. Cleansing promotes that rehearsal step into a closure gate: refuse to seal/adopt a generation unless every promoted crate's index deps resolve within the target stage.

12.2 Dirty-tree filter (provenance-clean source)

A boundary also filters by source cleanliness: only artifacts built from a clean git tree (committed and pushed) are promotable. A dirty/uncommitted/ unpushed build has no reproducible provenance → not promotable. This reuses the §O release doctor dirty-tree detection (real local git status per repo, gix: clean | uncommitted | unpushed). The crate's source_commit (§10.2 manifest field) must reference a pushed commit; a dirty tree fails the filter and the artifact is dropped from the bundle.

12.3 Who does what

• nornir computes the cleansing — it owns the dep graph + closure, the topo order, the version-skew / forbidden matrix, and the dirty-tree status (dep_graph.rs, the warehouse, release doctor). nornir hands Holger a cleansed, closure-closed, topo-ordered, provenance-clean bundle.
• Holger enforces the gates + records provenance — the closure gate (deps resolve within the target), the dirty-tree filter (every entry's source_commit is clean+pushed), and it writes the resulting closure + provenance into the archive manifest (§10.2) so the cleansed set is auditable and travels with the sealed unit.

13. Beyond Artifactory — problems the substrate solves that JFrog/Nexus leave open

The point isn't parity. Content-addressing + immutable sealed generations + the deletion-vector chain + the nornir warehouse make a class of capability fall out for free that Artifactory/Nexus cannot reach without re-architecting their copy-based, DB-backed, online-first model. Each row is a problem they leave open.

The throughline: Artifactory treats a release as mutable server state you trust. Holger treats it as an immutable, content-addressed, signed, self-describing object — so atomicity, verifiability, history, diff, rollback, offline provenance, and tamper-evidence stop being features to build and become properties of the unit.

Promotion STAGES — the ladder (shipped 2026-07-22)

Class: design · Function: promotion/stages. Implementation: server/lib/src/promote/stages.rs, wired into PromotionEngine.

What was already right, and is untouched

holger had the hard half. PromotionEngine::seal moves an artifact into the immutable, content-addressed PROD store by content address, not by rebuilding it, and the ref it writes IS the immutability guard. /sparring is the release rehearsal, and a green rehearsal becomes a sealed generation by a pointer move. That primitive is correct and this work does not replace, wrap, or duplicate it (L5). Every stage hop below is one of those same seals.

What was missing: the model on top

Artifactory's promotion is not one transition, it is a ladder — named stages an artifact travels through, where each hop is recorded, queryable, gateable and reversible, and where the ladder itself says which hops are legal. holger had "DEV → PROD" and nothing between.

    sparring              staging               release
  (rust-sparring) ────▶ (rust-staging) ────▶ (rust-release)
      rung 0                rung 1                rung 2
                                              [gated]

A Stage is a NAME bound to a REPOSITORY. That is how Artifactory models it and it is already the shape seal(from_repo, to_repo, …) takes, so a StageLadder is a lookup table over the existing mechanism rather than a second mechanism.

Reversible WITHOUT mutating a sealed generation

This constraint shapes everything. A promotion must be undoable; a sealed generation must never change. Those are compatible exactly one way: a withdrawal is a new forward record, never a deletion.

withdraw appends a Withdrawn entry. The blob is not touched. The ref is not touched. The bytes sealed at that stage remain byte-identical and still verify — an airgap bundle already carried across the boundary does not become invalid because someone changed their mind on the other side. What changes is the answer to a question: current_stage stops reporting that rung and falls back to the one below. The history keeps both records, in order, so "promoted to release, then withdrawn for CVE-2026-0001, then re-promoted" stays legible forever.

That is the same shape as a deletion vector over an immutable generation, for the same reason: you cannot un-write an immutable store, so you write the correction after it.

Consequence worth stating plainly: after a withdrawal the coordinate is still sealed. Re-promoting the same coordinate still returns AlreadyPromoted. Immutability survives the undo — asserted by withdrawal_moves_the_answer_not_the_sealed_bytes.

Ordering guarantees

• An illegal hop writes nothing. The ladder is judged before the blob, before the ref, before anything touches disk (an_illegal_hop_is_refused_before_the_store_is_touched asserts no blob dir, no ref, no lineage entry).
• The record follows the fact. The lineage entry is appended only once the seal took, so the history never claims a promotion that did not happen. A lineage write that fails is non-fatal and logged loudly — the artifact IS sealed and saying otherwise would be the bigger lie.
• Current is a reduction, not state. current_stage replays the append-only entries in order and takes the highest rung still standing. Two readers cannot disagree about where an artifact is, because there is only one record and one reduction of it.
• A torn trailing line is tolerated — a crash mid-append must not make the whole history unreadable (same tolerance as the Arrow-IPC audit replay).
• Traversal-closed — the ladder name and the coordinate are both sanitized to one path segment before they reach the filesystem, like the promotion ref path.

Additive: no ladder configured ⇒ nothing changed

PromotionEngine::new(root) has an empty ladder. seal then judges nothing extra, writes no lineage, and is byte-for-byte the promotion holger has always done (without_a_ladder_promotion_is_exactly_what_it_was). The ladder is opted into with with_ladder(name, ladder).

Deferred, deliberately

• The RPC + UI surface. PromotionService.PromoteArtifact still takes (from_repo, to_repo) and now inherits the ladder check through the engine, but there is no GetStage / GetLineage / WithdrawArtifact RPC and no UI pane yet. The engine API (current_stage / stage_history / withdraw) is the seam those will call.
• RON config for the ladder. The ladder is constructed in code today; the natural home is the RON config graph beside auth (declarative policy, git-tracked, immutable in the static flavour — the same argument auth-rbac-design.md makes for roles).
• Cross-ladder queries ("everything currently at release") — the lineage is keyed per coordinate, so a roster query needs an index. Not faked.