nordisk/Skidbladnir

REFRESH — the live-service upgrade verb (Lifecycle::refresh)

The one command behind korp-server --refresh / dwarves-full --refresh: the whole build → install → migrate → restart dance as ONE idempotent, rollback-safe call. Materializes the super-plan §🔄 SERVICE REFRESH corrected design — signed-bundle reinstall + self-migrate, NOT an exe-swap — by reusing seams that already exist.

Why

Lifecycle shipped install/start/stop/status/uninstall but no refresh/upgrade for a LIVE service. Bringing a fresh HEAD onto oden meant a manual build → copy → restart dance every time. refresh is that dance, made one verb, idempotent and rollback-safe.

The seams it REUSES (no parallel machinery)

New code is only the glue (install::refresh_product) + the migration model (migrate.rs) + each product's --migrate-config verb.

The migration model — stamped + ordered CHAIN (src/migrate.rs)

Migration code lives IN THE PRODUCT; Skidbladnir owns only the ordering.

1. Stamp. Every install writes the product's current schema version into <data_dir>/.migration-stamp.json (a dotfile INSIDE the data root, so the uninstall law erases it with the data — a wiped product forgets its schema too).

2. Chain. The product declares an ordered list of schema versions, oldest → head (korp: ["korp-v0"]; dwarves: ["dwarves-v0"]). migration_steps(stamped, chain) diffs the on-disk stamp against it:

   • stamped == head ⇒ no hops — this is the idempotency guarantee.
   • stamped earlier ⇒ each consecutive (from → to) hop up to head.
   • stamped NOT in the chain ⇒ hard error (downgrade/corruption never silently "catches up").

3. Replay. replay_migrations runs each hop through a MigrateRunner. Production is ProductBinRunner — it execs the NEW generation's own <bin> --migrate-config from=X to=Y (direct argv, zero-shell: no sh -c, the same direct-exec a systemd ExecStart= does; the product runs its own Rust migration). The stamp is written after every SUCCESSFUL hop, so a failed hop leaves a resumable stamp (retry resumes precisely, no double-apply, no skip). Tests inject a RecordingRunner and assert the exact hops offline.

An absent stamp is treated as a brand-new data root (stamp head, migrate nothing) — an honest statement, not a paper-over; a product introducing the stamp onto a pre-existing deployment backfills the stamp first.

The engine — install::refresh_product (§🔄 exact order)

1. Verify-on-receipt the bundle (refuse to unfold a single byte of a tampered one).
2. Unfold the whole product into software/<newver>/<name> — current NOT flipped yet (data + log preserved).
3. Migrate — replay stamped → head against the preserved data using the INCOMING binary, while the OLD generation is still live.
4. Flip current atomically to the new generation.
5. Restart over D-Bus (daemon-reload + hard stop→start; zero-shell).
6. Health-probe the running service.
7. Roll back on an unhealthy probe: flip current back to the prior generation (atomic symlink swap) + restart. A rolled-back refresh is a SAFE outcome (the service is healthy on the prior gen), reported as such — not a fault. The additive forward-migration is honestly left in place after a binary rollback.

Restart + health are injected via the RefreshControl trait (SystemdRefreshControl in prod; scripted in tests) so the whole orchestration — including rollback — runs offline.

The product half

Each product owns its chain + a --migrate-config from=X to=Y verb that REALLY validates the hop (identity or adjacent-forward only; unknown/garbage refused — never a stub that fakes success). Today both korp and dwarves declare a single head, so the only legit hop is an honest identity no-op and a fresh install's refresh runs zero hops. --refresh (no arg) builds from HEAD via cargo build (a sanctioned build-host action, target on /scratch/cargo/<repo> per the oden disk law) + packs the bundle; --refresh <bundle> installs an appointed one.

Tests (all green, offline)

• migrate: chain math (v0→v1→v2), idempotency (stamped==head ⇒ 0 hops), unknown stamp refused, absent stamp = fresh, resumable failed hop.
• install::refresh_is_idempotent_and_flips_current — install v1, refresh→v2 flips current, re-refresh is a proven no-op.
• install::unhealthy_refresh_rolls_back — an unhealthy new generation rolls current back to the prior one, restarts on it, and leaves the forward-migration in place.
• orchestration::ron_migrate_step_replays_via_module — the RON Migrate step replays a chain through MigrateModule + a runner.
• korp migrate_config_validates_the_chain (+ proven LIVE: identity ok, bad version exits 1). dwarves mirrors it.

Live status (oden, 2026-07-29)

--refresh is code-complete, builds, and its engine is proven by the offline rollback/idempotency tests. A live full-server refresh of the running korp-server was NOT run: it is a root-owned system unit (/etc/systemd/system/korp-server.service, serving the map on :50051) and a second rustfs-embed server would collide on :9000. --migrate-config is proven live. The clickable proof (map serving tiles + korp UI + dwarves UI on the RTX 4090) is documented in the super-plan.

KUBERNETES REFRESH — the K8s parity (KubeRuntime::refresh, src/kube_runtime.rs)

The same verb, one substrate over. A product promotes a new software version IDENTICALLY on Metal (systemd current-flip + D-Bus restart) and on Kubernetes (Deployment rolling-update over the apiserver) — the same stamped, ordered migration model drives both. src/kube_runtime.rs is the K8s half; it reuses the product-owns-its-migration doctrine and the pure chain math of src/migrate.rs wholesale, changing only the substrate the five refresh steps run on.

The mapping — each systemd step to its K8s equivalent

The migration model is UNCHANGED — only the stamp's home moves

migrate::migration_steps (the pure stamped → head chain diff) is reused verbatim: stamped == head yields zero hops (idempotency), a stamp outside the chain is a hard error (no silent downgrade), and each hop is stamped forward on success so a failed run resumes precisely. The one difference from the systemd side is where the stamp lives: on Metal it is <data_dir>/.migration-stamp.json beside the data; on Kube it is a small ConfigMap the engine reads before and writes after each good hop (the migration Job mounts the product's PVC so --migrate-config reads and writes the SAME data root the running pods do). The Job image is the NEW image because migrations are a property of the incoming code — exactly as ProductBinRunner execs the NEW generation's binary on Metal.

The exact order (kube_refresh, §🔄 corrected design, K8s)

1. Anchor. Read the Deployment's current container image FIRST, before any mutation — a rollback needs to know where it is going back to, and the moment after the apply is too late to ask.
2. Migrate. Read the stamp; compute migration_steps. An absent stamp is a brand-new deployment (stamp head, run nothing). Each hop runs a migration_job on the new image; on success the stamp advances. A failed hop aborts BEFORE any delivery — the old Deployment is untouched and still serving (outcome MigrationFailed), the analog of a systemd migration failing before the flip.
3. Deliver + swap. SSA-apply rolling_deployment with the new image; k8s runs the RollingUpdate (maxUnavailable: 0, maxSurge: 1 — zero-downtime).
4. Health. Wait for the rollout to become Ready (updated == ready == available == desired).
5. Roll back on an unready rollout: re-apply the prior image's spec so k8s scales the prior ReplicaSet back up, and wait again. No readable anchor yields NoAnchor (nothing restored — never reported as a rollback); a failed rollback yields RollbackFailed (the environment may be DOWN — never collapsed into "rollout failed"). A clean RolledBack is a SAFE outcome (serving the prior image), reported as such.

Restart/health/apply/migrate are injected via the KubeRefreshOps seam — the K8s analog of the systemd RefreshControl + MigrateRunner injections — so the whole orchestration, rollback included, runs offline against a scripted mock.

Additive, gated, zero-shell

The engine (kube_refresh), the manifest renderers, and the injected-mock tests compile in the lean core with NO kube (the seam is sync, serde_json-only). Only the ONE production impl, kube_runtime::live::KubeRuntime, is behind the OFF-by-default kubernetes feature; it bridges each sync seam call to async kube-rs over its own tokio runtime (the same sync→async bridge unfold's Talos target uses), reusing the existing k8s_apply engine (apply_json_manifests + run_job) — no kubectl, no helm. korp consumes Skidbladnir with default-features = false; its kube feature opts into skidbladnir/kubernetes and nothing else changes.

Tests (all green, offline)

src/kube_runtime.rs: golden renders — the migration Job (new image + exact --migrate-config from=X to=Y argv + PVC mount + backoffLimit: 0 + restartPolicy: Never), the refreshed Deployment (new image + RollingUpdate + readiness/liveness), the HTTP-probe form, the no-probe form, and the rollback target (prior-image Deployment, everything-but-the-image identical so k8s reuses the prior ReplicaSet). The engine over a scripted KubeRefreshOps mock: a good refresh migrates-then-delivers and reports the hops (anchor read first, all Jobs before delivery, stamp advanced to head); stamped == head runs zero Jobs; an absent stamp stamps head; a failed migration Job aborts before any delivery and leaves a resumable stamp with the old image still live; an unready rollout rolls back to the prior revision; a failed rollback is its own outcome; a missing anchor says nothing was restored; a stamp outside the chain is refused; the dry run is pure.

Live status

Offline-proven end to end (the five-step orchestration, the migration-Job + stamp lifecycle, the manifest renders, and the rollback path). What still needs a live cluster is the same class the systemd side defers: the real rollout timing and pod-readiness cadence under the apiserver (the wait_rollout poll is #[cfg]-gated over kube-rs like k8s_apply::wait_ready, not unit-tested against a live control plane). The KubeRuntime production impl builds clean under --features kubernetes.

WATCH-REFRESH — the deferred sub-verb (Lifecycle::watch_refresh, src/watch.rs)

The always-fresh oden: poll git HEAD; refresh on change; no-op when unchanged. It adds no parallel machinery — it is a thin DECISION in front of the refresh engine above.

The decision (watch_refresh_once)

1. Read HEAD through gix (GitHead, zero-shell — never git rev-parse), peeling HEAD to its 40-hex commit id.
2. Diff it against the persisted WatchStamp (<data_dir>/.watch-refresh-stamp.json — same dotfile-inside-data posture as the migration stamp, so the uninstall law wipes it with the data).
3. Unchanged (stamp == HEAD) ⇒ honest, logged no-op; stamp untouched.
4. Moved (or first-ever, no stamp) ⇒ run the injected RefreshTrigger (build-from-HEAD → pack → refresh_product in production), then advance the stamp to the new HEAD on success only. A trigger error propagates and leaves the stamp UNADVANCED, so the next tick retries the same commit — restart-safe, no double-apply, no skip. seed_watch_stamp lets the deployer set the post-install baseline so the very first tick is a clean no-op instead of a (safe, idempotent) redundant refresh.

Timer, not an in-process daemon (the chosen mechanism + why)

The refresh engine is already idempotent + crash-safe with ALL resumable state on disk (the schema stamp + the watch stamp). A long-lived in-process loop therefore buys nothing and costs supervision, reboot-persistence, and a runtime held alive only to sleep — and on a crash it must re-read the stamp anyway. So the wired production mechanism is a systemd .timer + Type=oneshot .service (render_watch_units / install_watch_refresh_timer, installed over the SAME zbus path as everything else): each tick is one watch_refresh_once that reads the stamp, acts, advances it, and exits; systemd owns the cadence, restart-on-failure, and (Persistent=true) firing a tick missed while the host was off. The oneshot is Restart=no (the timer re-runs it) and carries no [Install] (the timer pulls it). A bounded watch_refresh_loop (injected Ticker; SleepTicker parks the thread — no busy-wait) is a documented foreground/dev alternative; a product that already runs a tokio runtime can drive the same atom from a tokio::time::interval.

Additive + gix behind a feature

The HEAD reader is an injected HeadSource seam, so the whole poll/diff/stamp decision + the timer renderers compile in the lean core with NO gix. Only the ONE production impl (GitHead) is behind the OFF-by-default watch-refresh feature (dep:gix, the lean revision/sha1 leaf). korp/dwarves consume Skidbladnir with default-features = false and do not enable it, so their builds are unchanged — a product opts in when it wires --watch-refresh.

Tests (all green, offline)

src/watch.rs: unchanged-HEAD no-op (trigger never runs); moved-HEAD invokes the refresh with the prior commit + the stamp advances, then a second tick is a proven no-op; first-ever tick refreshes with previous=None; a failing refresh leaves the stamp for a precise retry (recovering retry re-runs the SAME hop); the bounded loop refreshes only on change; the oneshot/timer renderers are asserted (Type=oneshot, Restart=no, OnUnitActiveSec, Persistent, Unit=, user-scope drops User=). Behind --features watch-refresh, a REAL gix test inits a repo, commits, and proves GitHead resolves the exact HEAD and sees a second commit as a MOVE.

Live status

Offline-proven end to end (the decision, the stamp lifecycle, the unit render, and the real gix HEAD read). What still needs a live host: the actual timer tick cadence under systemd (the .timer/.service install is #[cfg(unix)] over zbus, not unit-tested, exactly like install_systemd) and a full live build-from-HEAD refresh — both gated by the same root-owned-korp-server / :9000 collision that held back the live --refresh above.