nordisk/Skidbladnir

PROMOTE — the design, and the public contract

skidbladnir::promote, feature promote (+ promote-k8s for the live half). Guide: promote-guide.md. Walkthrough: promote-for-idiots.md.

Everything below marked CONTRACT is what a consumer builds against. It is written precisely because the first consumer (Njord's crates/deployer) is being refactored onto it by a different pair of hands than wrote it.

1. What this module is

Skíðblaðnir's other verbs are birth: pack → archive → fetch ║airgap║ → unfold → install → start. promote is ongoing operations on something already alive — roll a new image into a test rung, copy production's data down into it, raise a fresh generation and switch the host onto it, and never do any of that to production by accident.

It was extracted from a working tool. The split is:

The seam is AppData. Nothing on the left ever learns anything from the right.

2. Charter — the open question, stated not answered

Does this ship's charter stretch to "keeps running things in sync"?

The honest answer from building it: the code fits, and the charter is a judgement call that the code shape does not make. Everything promote needs already lives here — the content-addressed bundle, the zero-shell law, the kube-rs applier, the pure-Rust posture, the "a refusal names its remedy" habit. Nothing had to be bent. The bundle envelope is a thin layer over archive::Archive and is the strongest argument that this is the same ship: promotion moves a sealed payload between two places, which is what the ship does.

The argument against is equally simple and is not a code argument. "Skíðblaðnir" is the ship that carries the gods' war-gear across a gap and folds into a pouch. That is a transport and birth metaphor. Blue/green generations, ingress switching and rollout watching are operations, and a reader who knows the crate as "the airgap framework" will not look here for them.

So the module is built to be liftable: one directory, two features, no changes anywhere outside #[cfg(feature = "promote")], and no new heavyweight dependency. If the answer is "this belongs in its own crate", the move is a directory rename and a Cargo.toml — not an untangling.

Recommendation, for Rickard and Henrik to accept or overrule: keep it here while it is changing weekly and the ladder schema is settling; revisit once a second consumer authors a ladder. A versioned cross-repo contract for something this young costs more than it buys.

3. CONTRACT — the ladder schema (promote::schema)

This is a public, versioned config format. Any consumer authors one file in it. Njord's Njord-deploys/env/<customer>/<slot>/slot.yaml is one instance of this schema, not a format Njord invented.

apiVersion: skidbladnir.nordisk/promote.v1   # REQUIRED. Unknown → loud refusal.
meta:                                        # ladder-wide consumer facts
  customer: Example Corp
  owner: platform-team@example.com
slots:
  - name: dev                # REQUIRED, unique
    namespace: example-dev   # REQUIRED, unique
    hosts: [dev.example.test]# authoritative; a host not listed belongs to no slot
    role: development        # REQUIRED — see the vocabulary below
    guarded: false           # REQUIRED — never derived from the name
    promotes_to: [qa-a, qa-b]# the rungs CODE may move to next
    meta: {note: anything}   # carried, never interpreted

The shipped reference instance is promote::EXAMPLE_LADDER (a fictional consumer, deliberately not anyone's real hosts). It is parsed by a conformance test, so it cannot drift from the parser.

3.1 Roles are declared, never inferred

role vocabulary, with tolerant spellings folded onto the canonical value:

Folding spellings of the role field is reading a declared value tolerantly. It is not the same as inferring a role from a slot's name, which never happens anywhere in this crate. A slot called prod whose declared role is test is a test slot; a slot called harmless whose declared role is production is production.

guarded is a separate required boolean, because "what is this for" and "does the fence apply" are different questions. Any role may be guarded. The one tie between them: role: production with guarded: false is a validation error — that combination is ambiguous, and ambiguity about what production is, is the failure mode the whole module exists to prevent.

3.2 Order is a DAG, not a rank — and why

promotes_to names the rungs code may move to next. Edges, not a rank integer, because:

• two parallel test rungs that both feed one pre-production cannot be given distinct ranks, and giving them the same rank loses the fan-in;
• "is this a legal next hop?" is an edge query — with ranks it becomes "is rank(to) == rank(from) + 1?", which says yes to a hop between two unrelated rungs that happen to sit at the same depth;
• a rank in the file is redundant state that can drift from the edges.

A rank is still available where a viewer wants a linear layout: Ladder::rank computes topological depth from the edges. Computed, never authored.

3.3 The two directions

Edges point the way code travels — up, toward production. Data travels the other way: copying production's data into a test rung is normal; the reverse replaces production with somebody's test rows. Both are answered from the same edges:

ladder.check_promotion(from, to) -> Hop   // code, upward
ladder.check_data_flow(from, to) -> Hop   // data, downward

Hop is Legal | SkipsRungs(Vec<String>) | WrongDirection | NotOnLadder | SameSlot | UnknownSlot(String), and Hop::require(from, to, allow_skip) turns it into a Result. A skip is the only overridable case, and even when overridden the skipped rungs come back in the Ok value so the caller reports them. A wrong-direction move is not overridable at all.

3.4 Validation refuses ambiguity

Every check names the file and the field:

• apiVersion missing or unknown → refused, listing the supported version. No best-effort parse of an unknown version.
• slots empty → refused.
• duplicate name, duplicate namespace, or the same host on two slots → refused (an identifier must resolve to exactly one slot).
• role or guarded absent → serde parse error naming the field. Unspecified is never silently "safe".
• role: production + guarded: false → refused as ambiguous.
• a promotes_to naming an undeclared slot → refused, listing what exists.
• a self-edge, or any cycle → refused, naming the cycle.

4. CONTRACT — the registry and the un-lowerable fence

SlotRegistry::from_slots(Vec<Slot>) -> Result<SlotRegistry>   // the baseline, in code
SlotRegistry::new(Ladder) -> SlotRegistry
SlotRegistry::load_over(&baseline, config: Ladder, DemotionPolicy) -> Result<SlotRegistry>

4.1 The hole being designed out

The tool this generalises had:

pub fn assert_lab_only(t: &Target) -> Result<()> {
    if !is_prod_namespace(&t.namespace) { return Ok(()); }   // ← here
    …
}

A config file that marks the production namespace non-prod does not lower that guard, it deletes it: the early return fires, and every destructive verb is legal against production with no banner and no audit line. Deleting the entry entirely is the same attack and works even better.

4.2 What replaces it

Configuration is layered over a baseline the caller declares in code, and the merge is one-directional:

• config may raise a slot to guarded — always allowed;
• config may not lower a guarded baseline slot, and may not make one vanish by omission. A baseline guarded slot is matched against the config by any identity, so renaming the entry does not evade the check either.

DemotionPolicy::Refuse (the default) fails the load. ReportAndKeepFenced loads, keeps the slot fenced, and records a Demotion — which then appears in every guard refusal touching that slot and in Topology::notes, so it reaches a panel and not only a terminal that scrolled away.

There is deliberately no third policy. "Let the config win" is the hole; offering it as an option is offering the hole.

The merged ladder is re-validated, so a config cannot launder an ambiguous slot in through the merge.

4.3 One slot has many names

enum TargetRef<'a> { Slot(&str), Namespace(&str), Host(&str), Url(&str) }
registry.resolve(&TargetRef) -> Resolution<'_>   // Slot(&Slot) | Unknown

There is no &str overload. resolve matches across every identity a slot has — name, namespace, each declared host, and any generation namespace derived from the name or the namespace — so which variant you pick cannot make the answer wrong, only the message clearer.

This is the fix for the second bug: a raw --slot prod cleared a fence keyed on namespaces, because no namespace is literally called prod. Here prod is one of the production slot's identities, so it hits the fence.

Two deliberate asymmetries, both in the fail-safe direction:

• ambiguity resolves guarded. If an identifier matches both a guarded and an unguarded slot, the guarded reading wins.
• www. folds, for guarded slots only. www.example.test matches a guarded slot that only listed example.test. This is not an inference about safety — it can only ever make the fence refuse more. Nothing is ever concluded to be safe from a host's shape; a subdomain is its own slot unless the schema says otherwise.

5. CONTRACT — the guard

Guard::new(&registry)
    .with_override(Override::flagged())          // key one: the CLI flag
    .with_env(EnvSource::Process)                // key two: PROMOTE_ALLOW_PROD=1
    .allow_unknown()                             // read-only verbs ONLY
    .assert(&TargetRef::Slot("lab"))  -> Result<Cleared>
    .assert_all(&[…])                 -> Result<Cleared>   // all or nothing
    .assert_promotion(from, to, allow_skip) -> Result<(Cleared, Vec<String>)>
    .assert_data_flow(from, to)             -> Result<Cleared>

• Two keys, and only two keys. flag && env == "1". Exactly "1" — a leftover PROMOTE_ALLOW_PROD=0 or =false must not arm it. The refusal says which key is missing.
• Unknown is fenced by default (UnknownPolicy::Fence). A tool that acts on whatever string it was handed acts on a typo. allow_unknown() is explicit, greppable and documented as read-only territory.
• EnvSource::Value(Option<String>) makes key two injectable, so tests never mutate the process environment (the tool this generalises needed a Mutex around its own test suite for that reason).
• Every refusal is a Refusal implementing Error + Display, carrying target (URL credentials stripped), kind, reason, a remedy naming the next step, and notes carrying any demotion attempt.

5.1 Cleared is a capability, not a promise

Cleared has private fields and is constructible only by the guard. Every mutating operation in promote::release takes &Cleared. "Did anyone check the fence?" becomes a signature instead of a convention each op has to remember.

Cleared::covers(namespace) is true for the cleared slot's namespace and for any generation of it, and false otherwise; require_covers is the Result form. A clearance for lab cannot be spent on njord.

6. CONTRACT — AppData, the seam

pub type Fut<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;

pub trait AppData: Send + Sync {
    fn describe(&self) -> &str;
    fn quiesce<'a>(&'a self, slot: &'a Slot) -> Fut<'a, Result<Quiesced>>;
    fn resume<'a>(&'a self, slot: &'a Slot, q: &'a Quiesced) -> Fut<'a, Result<()>>;
    fn dump<'a>(&'a self, slot: &'a Slot, staging: &'a Path) -> Fut<'a, Result<DumpReport>>;
    fn load<'a>(&'a self, slot: &'a Slot, staging: &'a Path, q: &'a Quiesced)
        -> Fut<'a, Result<LoadReport>>;
    fn watermark<'a>(&'a self, slot: &'a Slot) -> Fut<'a, Result<Option<Watermark>>>;
    fn neutralise_after_load<'a>(&'a self, slot: &'a Slot, q: &'a Quiesced)
        -> Fut<'a, Result<Vec<String>>>;
}

Refinements against the original sketch, each from the real code:

• resume was added. The real restore wraps its load so that whatever happens the workload comes back up — a failed load must never leave an environment scaled to zero. That is a second call, so it is a second method. It is deliberately not Drop-based: an implicit resume on unwind is how an environment comes back halfway through a load.
• Quiesced is a token, and load requires it. The order is a signature rather than a convention. It carries the slot name (is_for) so it cannot be spent elsewhere, and an opaque resume_token for whatever the app needs back.
• neutralise_after_load returns Vec<String>, not (). The real one neutralises a queue of unprocessed outbound work and the operator must be told how many rows moved. An app with nothing to neutralise returns an empty vector — a statement, because the method still had to be called.
• describe() is stamped into the bundle manifest, so a bundle knows what produced it.

6.1 Async without an async dependency

The traits return Fut — a plain Pin<Box<dyn Future>> from std. Object-safe (the ship holds &dyn AppData), runtime-agnostic, and zero dependencies: no async-trait, no tokio in the promote core. A synchronous implementation returns Box::pin(async move { … }) and awaits nothing.

6.2 No silent partial success

DumpReport::new()
    .part(CapturedPart::complete("database", Some(41)))
    .part(CapturedPart::skipped("objects", "S3 credentials were not configured"))
    .watermark(Some(w))
    .app_meta(json!({ … }));           // opaque to the ship

report.completeness() -> Completeness  // Complete | Incomplete{parts} | Unknown
report.require_complete() -> Result<()>

• The application declares its parts. A part that is never declared cannot be silently absent, because the app said it would be there.
• CaptureState is Complete | Partial{captured, expected} | Skipped | Failed | Unknown — five states, because "not attempted", "attempted and failed" and "attempted, outcome unknown" are different facts.
• An empty report is Unknown, never Complete. That is the shape a dump that never ran has.
• Every non-Complete part must carry a detail; is_well_formed() catches a forged wire form that does not.
• items: Option<u64> — None is unknown, never zero.
• A part may be .optional(), in which case skipping it does not spoil the verdict — but it still shows in lines().
• LoadReport is the same shape and held to the same standard.

The specific bug: the original backup degraded to database-only when S3_* was missing and still exited 0, producing a bundle indistinguishable from a good one. Here that becomes CapturedPart::skipped("objects", …) → Incomplete → the verdict is stamped into the manifest → unseal refuses it.

6.3 The ship never guesses the payload

Watermark { mark: Option<String>, magnitude: Option<BTreeMap<String, i64>> }
bundle_wm.would_discard(target_wm: Option<&Watermark>) -> Discard

Discard is Nothing | NewerWrites{..} | RemovedRows{buckets} | Unknown{because}, and requires_confirmation() is true for everything except Nothing — including Unknown.

Two clocks, and the second is not decoration. A change timestamp only moves forward when rows are written; a deletion does not move it, it removes it. So a slot where somebody deleted two hundred rows since the bundle was sealed reads as "nothing happened" on the mark alone, and a switch destroys those deletions silently. magnitude is the cheap quantity that actually goes down.

Every path that cannot prove safety returns Unknown: no target watermark, a missing mark on either side, no comparable counts, or a bucket the target did not report. None from AppData::watermark means unknown, never "nothing newer".

7. CONTRACT — the bundle envelope

seal(&dyn Archive, &SealSpec, &DumpReport) -> Result<Sealed>
verify(&dyn Archive, bundle) -> Result<VerifyReport>
unseal(&dyn Archive, bundle, dest, LoadPolicy) -> Result<Unsealed>
load_manifest(bundle) -> Result<PromotionManifest>

Payload-agnostic, and a thin layer over the existing archive::Archive (so TarZstd and the znippy engine both work unchanged).

The PromotionManifest is a sidecar at <bundle>.promote.json, mirroring archive::manifest_path. It is beside the bundle rather than inside it because a manifest cannot describe an archive it is part of — including it would change the content and therefore its own integrity root.

It carries version (refused if unknown, never best-effort parsed), source_slot, taken_at, app, completeness, parts, watermark, app_meta (opaque), and the archive::BundleManifest with its root_sha256.

VerifyReport keeps two questions apart on purpose:

• intact() — do the bytes verify?
• is_usable() — intact and the capture was whole.

Conflating them is exactly how a partial bundle passes a checksum and gets called good. LoadPolicy::strict() requires both; accept_incomplete() and accept_corrupt() are explicit opt-ins, and the verdict still comes back in Unsealed::completeness().

8. CONTRACT — generations

Generation::new(slot, unix) -> Result<Generation>     // "<slot>-YYYYMMDD-HHMM"
Generation::is_of(base, namespace) -> bool
generations(slot, &[String]) -> Vec<Generation>       // newest first
sleep_plan(ns) / wake_plan(ns) -> Vec<GenerationStep>
switch_plan(slot, from, to_ns, host, service, port, tls) -> Result<Vec<GenerationStep>>
RAISE_LADDER: &[RaisePhase]

Plans are pure functions returning values. That is what lets the ordering — the part that has actually caused outages — be asserted in a test with no cluster in sight, and it is what makes a dry run structurally incapable of touching the world (see §10).

switch_plan's order is the contract: wake the target → point the host at it → drop the old route. Waking first closes the window where the host resolves to a scaled-down generation. Both routes existing for a moment is safe: a route object is namespace-local and can only point at services in its own namespace. The old generation is not put to sleep — switching back must be as fast as switching over, and a running generation is seconds from live.

RaisePhase is Rbac → Namespace → Tls → Storage → Secrets → Release → Data → SwitchHost. The ship names the ladder and its order; the caller climbs it. TLS sits early because it is a step that can refuse, and a refusal belongs where it costs least — an empty namespace, not a completed data load. SwitchHost is the commit point: before it a failure costs a namespace nobody saw; after it, an outage.

9. CONTRACT — release operations, and the honest word on Helm

trait ReleaseOps {                       // async via Fut, object-safe
    fn current_image(&Cleared-free)      // read
    fn set_image(&Cleared, …)            // write
    fn wait_rollout / scale / wait_replicas / workloads
    fn apply(&Cleared, ns, &ReleaseSource)
    fn host_owner / route_host(&Cleared, …) / drop_route(&Cleared, …)
}

rollout(&dyn ReleaseOps, &Cleared, &RolloutPlan) -> Result<RolloutOutcome>
run_generation_plan(&dyn ReleaseOps, &Cleared, &[GenerationStep]) -> Result<Vec<String>>

KubeReleaseOps (feature promote-k8s) implements it over kube-rs. No shell, anywhere.

RolloutOutcome keeps four cases apart, because they mean different things to the person on call:

The anchor is read before anything is patched; after the patch it is too late to ask. NoAnchor exists so a rollback that never happened is never reported as one.

9.1 Helm

The original shells helm upgrade, and its --dry-run still spawned helm version — a preview that touched the world. Neither is acceptable here.

But promote does not reimplement Helm, and says so rather than designing around it. A chart is Go templates plus Sprig plus .Capabilities plus lookup against a live cluster. There is no production-grade pure-Rust implementation of that stack, and writing one is a project, not a module.

So ReleaseSource has two sides:

• Manifests { docs: Vec<String> } — the native path. Server-side apply through kube-rs, with discovery so any kind resolves. Produce them however you like: this crate's k8s::KubernetesRenderer, or helm template run inside the pipeline that owns the chart.
• Chart(ChartRef) — a declaration, not an execution. It records which chart, release name and values a slot is supposed to be running. The ship carries it, compares against it, and apply refuses it with a remedy — "render the chart where it is owned and pass Manifests" — rather than shelling out or pretending.

What is generic and worth owning is the read side. stored_release_values(client, namespace, chart_name) pulls a Helm release's stored user-supplied values straight out of its sh.helm.release.v1.* Secret (base64 → gzip → JSON). That is exactly what helm get values answers, obtained with no Helm binary on PATH, no subprocess whose stdout carries the slot's secrets, and therefore runnable as a plain Kubernetes Job. Only the deployed revision is read, and only releases whose chart matches chart_name — a namespace routinely carries more than one release and the others are different applications.

10. Dry run is a pure function, not a flag

Every planning entry point (*_plan, RolloutPlan::describe, Hop, the whole schema layer) is pure: no cluster call, no subprocess, no filesystem read. A dry run is "produce the plan and print it". There is no code path on which a preview can touch the world, so the --dry-run-that-spawned-helm-version class of bug cannot be expressed in this shape.

11. CONTRACT — the viewer contract

registry.topology(&Observations) -> Topology     // serde, both ways

Topology { apiVersion, meta, nodes, edges, notes }, where a TopologyNode carries name, namespace, hosts, role, guarded, computed rank, promotes_to, consumer meta, and an Observation.

A panel needs nothing else and never parses a config file itself.

Observation { state, code, detail, observed_at, revision } with RungState:

The honesty rules, carried over from sockets:

• Absence is never health. A rung with no observation is NeverRun, not green — the projection does the lookup and defaults to NeverRun.
• Unknown is its own state. "We could not look" and "we looked and it was fine" are different facts and are never collapsed.
• A non-green rung always carries a machine-readable code — the constructors Observation::failed(code, …) / unknown(code, …) make a code-less failure unconstructible, and is_well_formed() catches one forged on the wire.
• No rendering decisions in the engine. No colours, no human labels, no layout. rank is here because depth is a fact about the DAG, not a rendering choice.
• A configuration that tried to lower the fence appears in Topology::notes, so it reaches the panel and not only a terminal.

11.1 Which viewer

No UI dependency enters Skíðblaðnir. This crate is path-depended by nornir, dwarves, korp and holger; adding an egui stack would force it onto four headless consumers. The panel lives in the consumer's own UI crate.

facett-pipeline should render this as-is, through a new adapter, and a dedicated panel is not warranted. The precedent is exact: facett_pipeline::RunaPipelineAdapter already renders both knut's Spark pipelines and the brokkr build DAG through one viewer. A promotion ladder is the same object — a DAG with per-node status — and Topology hands the adapter everything the viewer wants: nodes with stable ids, edges, a computed rank for layout, and a per-node state with a reason code.

Two things the adapter must decide, because the engine deliberately does not:

1. How the four RungStates look. NeverRun and Unknown must not be the same colour as Green, and ideally not the same as each other — the whole point of keeping them apart is that an operator can see the difference.
2. What to do with guarded. A fenced rung should be visually distinct; the engine only says guarded: true.

Topology::not_green() is the panel's "what needs attention" list, and it includes the never-run and the unknown by design.

12. Feature gating

promote     = ["dep:serde_yaml"]                              # pure core
promote-k8s = ["promote", "kubernetes", "dep:flate2", "dep:base64"]

• promote adds one dependency, serde_yaml — pure Rust, already in the tree, and needed so a ladder can be a slot.yaml. The async traits use a std Pin<Box<dyn Future>>, so no async runtime enters the core.
• promote-k8s implies kubernetes (kube + tokio + serde_yaml) and adds flate2 (gzip, rust_backend — no zlib to link into a distroless image) and base64, both only for reading a stored Helm release.
• With the features off, not one byte of the existing build changes: the module is #[cfg(feature = "promote")] pub mod promote; and the integration test is #![cfg(feature = "promote")].

13. Tests

tests/promote_contract.rs exercises the module through public exports only — an external consumer's view. If a name stops being public or changes shape it fails to compile, which is the honest early warning for a downstream break.

Red-when-reverted, each verified by actually breaking the code and counting:

14. What a consumer must do to build against this

1. Declare a baseline in code — at minimum every guarded slot, with its namespace and hosts. This is the floor a config file cannot go under.
2. Author a ladder file in the schema of §3, and load it with SlotRegistry::load_over(&baseline, ladder, DemotionPolicy::Refuse).
3. Implement AppData in the application's own repository, next to the code that knows the schema. Declare every part in DumpReport; return None from watermark when you cannot answer.
4. Route every destructive verb through Guard::assert* and pass the resulting Cleared down. The signatures will not let you skip it.
5. Keep every application-specific fact behind AppData — the outbox table, the change column, the table list, the object prefixes, the release-named ConfigMap/Secret, the mail policy. DumpReport::app_meta is the opaque channel for anything that must ride in the manifest.
6. Render charts where they are owned; hand promote ReleaseSource::Manifests.