nordisk/Skidbladnir

PROMOTE, for idiots

You have the same application running in several places — a test one, maybe a "nearly-real" one, and the real one. You need to move new code up through them without ever wrecking the real one, and sometimes copy the real one's data down so testing runs against something true.

That is all this is. Nothing below assumes you know Rust, Kubernetes or what a "slot" is.

The words

Slot (also: site, environment, rung). One running copy of your application. bench, staging, apex. Each has a namespace (where it runs) and one or more hostnames (what users type).

Ladder. The slots in order, with arrows showing where code goes next.

Guarded. A slot the tool refuses to touch unless you turn two keys. In practice: the real one.

Generation. A dated copy of a slot — staging-20260721-1432. Switching between generations is how you roll back in seconds instead of restoring a backup.

Bundle. A sealed file holding a copy of a slot's data.

The one picture

   bench  ──►  staging  ──►  apex          CODE goes this way (up)
     ◄────────────◄─────────────           DATA goes this way (down)
                                  guarded ─┘

Two rules fall straight out:

• Code goes up. Try it on bench, then staging, then the real one.
• Data comes down. Copy the real one's data into a test slot. Never the other way — that replaces the real data with test rows. The tool refuses it, and there is no flag to make it agree.

Step 1 — write down your ladder

One file. Copy this and change the names.

apiVersion: skidbladnir.nordisk/promote.v1
meta:
  customer: Contoso            # anything you like; the tool just carries it
slots:
  - name: bench
    namespace: contoso-bench
    hosts: [bench.contoso.example]
    role: development
    guarded: false
    promotes_to: [staging]     # from bench, code goes to staging

  - name: staging
    namespace: contoso-stg
    hosts: [stg.contoso.example]
    role: pre-production
    guarded: false
    promotes_to: [apex]

  - name: apex
    namespace: contoso
    hosts: [contoso.example, www.contoso.example]
    role: production
    guarded: true              # ← the real one

Things that will bite you

You must write role: and guarded: on every slot. Leaving one out is an error, not a shrug. That is deliberate: a slot that never said what it was should never be assumed harmless.

The name means nothing. Calling a slot prod does not make it production. role: production does. And a slot called harmless with role: production is production. The tool reads what you wrote, never what you called it.

role: production must have guarded: true. Anything else is refused as ambiguous.

List every hostname. A hostname you did not list belongs to no slot, and the tool will refuse to act on it rather than guess.

Do not number your rungs. The arrows are the order. The tool works out "depth" itself.

Step 2 — the two keys

To do anything to a guarded slot you need both:

1. the --allow-prod flag (whatever your tool calls it) on this exact command, and
2. an environment variable: PROMOTE_ALLOW_PROD=1

One alone does nothing. A flag on its own is a typo in a script; two are a decision.

It must be exactly 1. PROMOTE_ALLOW_PROD=true does not work. That is not fussiness — it is so a leftover line in your shell profile cannot quietly arm production.

If you get refused, read the message. It tells you which of the two keys is missing.

Step 3 — moving code up

promote bench → staging      ✅  one step, fine
promote staging → apex       ✅  one step (needs both keys — apex is guarded)
promote bench → apex         ❌  refused: "it skips staging"
promote apex → staging       ❌  refused: "that is the wrong way along the ladder"

The skip can be forced if you really mean it — and even then the tool hands back the list of rungs you jumped so it ends up in the log. You cannot make a skip invisible.

The wrong-direction one cannot be forced at all.

Step 4 — copying data down

copy apex → staging          ✅  normal
copy apex → bench            ✅  normal
copy bench → apex            ❌  always refused

Before it loads anything, the tool asks the target: have you changed since this backup was taken?

Three possible answers, and two of them stop you:

That last row is the important one. "I don't know" is not a green light. If the tool cannot prove the target holds nothing newer, it behaves exactly as if it knows there is something newer. Guessing in that situation is how a day's work disappears.

There is also a second check you might not expect. A timestamp only moves when someone writes. If someone deleted two hundred records, the timestamp does not move — it goes away with them. So the tool also compares counts, because a count is the thing that goes down when records vanish. If the target has fewer records than the backup, it stops.

Step 5 — backups that are honest

The most dangerous thing in the world is a backup that looks fine and isn't.

The old version of this tool had exactly that bug: if it could not reach the file storage, it backed up the database only, printed a small warning, and exited successfully. Three days later that file looks identical to a good backup.

Now the backup file carries a verdict of its own:

• complete — every part came through.
• incomplete — something did not, and it says which part and why.
• unknown — nothing was even declared. (A backup that never really ran looks like this. It is not treated as good.)

And when you go to restore:

$ restore --from backup.tar.zst

⛔ refusing to unseal backup.tar.zst: the capture that produced it was NOT whole.
   verdict: incomplete
   objects: skipped — object-store credentials were not configured

Note what it did not say: it did not say the file is corrupt. The file is perfect. Every byte checks out. The contents are only half there — and a checksum can never tell you that, which is why the verdict is written into the backup at the moment it is made.

You can override it. You have to say so out loud, and the verdict still gets printed afterwards.

Step 6 — rolling out, and rolling back

When you push a new version, four things can happen. Your tool should say which:

Rows 3 and 4 are not "it failed". They are different phone calls at 2am, which is why they are different words.

Step 7 — generations, and the fast way back

Instead of overwriting a slot, you can stand a new dated copy of it next to the old one and then point the hostname at the new one.

staging-20260715-0900     ← the old one, still there, still complete
staging-20260721-1432     ← the new one
stg.contoso.example  ──►  points at one of them

Rolling back is then: point the hostname back. Seconds. Nothing was destroyed, so nothing has to be restored.

The switch happens in a fixed order, and it matters:

1. wake the new copy and wait for it to actually be up;
2. point the hostname at it;
3. remove the hostname from the old copy.

If you point first and wake second, there is a stretch of time — seconds to minutes — where the hostname resolves to something that is not running. Users see errors. That is the whole reason the order is fixed and not a matter of taste.

The old copy is not shut down by a switch. Leaving it running is what keeps "switch back" fast. Shutting it down is a separate, later decision.

Step 8 — the screen that shows you all this

The ladder can be drawn as a picture with a colour per rung. Four states, and they are not three-plus-one:

Never-run is not green. Unknown is not green. If nobody has ever deployed to a rung, the screen says so instead of showing a hopeful colour. If the tool could not reach a rung — the machine was off, say — it says that, rather than either "fine" or "broken".

Failed and unknown always carry a short reason code (rollout-timeout, node-unreachable). You never get a red box with nothing in it.

The five things to remember

1. Write down role and guarded for every slot. Names mean nothing; what you declare is everything.
2. Two keys for the real one. A flag alone never works, and the value is exactly 1.
3. Code up, data down. Data upward is refused and cannot be forced.
4. "I don't know" stops you. Every time. That is the feature.
5. A backup that is byte-perfect can still be half empty. Read the verdict, not the checksum.

When it refuses you

Every refusal names what would make it work. If you hit one that does not, that is a bug in the refusal and worth reporting — a message that only says "no" is treated here as a broken message.

⛔ refusing: no slot is registered for namespace 'stagng'.
   Registered slots: apex, bench, contoso, contoso-bench, contoso-stg, staging, …
   Either you meant one of those, or this slot belongs in the ladder file — add it
   there (with its `role` and `guarded`) and it becomes usable.

That is what a good one looks like: what you typed, what exists, and what to do.