nordisk/dwarves

dwarf's CI: the dwarves forge is the build server

Since builder-64 was deleted (2026-10-27), the constellation's CI runs in dwarves itself: dwarves serve with the [ci] project in ci.dwarves.toml (crates/dwarves/src/forge_ci.rs, the Explorer lane in crates/dwarves/src/explorer.rs). There is no cron script and no shell runner. git.gunnar.rs has no Actions, so the .github/ and .forgejo/ workflow files of the repositories run nowhere.

What it does

• Poll. Every 15 minutes the forge fetches each polled repository, one at a time, with retry (git.gunnar.rs caps transfers per user). A pipeline whose Push(branch) head moved gets one run queued at low priority. A run queued for an older head runs the newest head.

• Run. One run at a time, on the Explorer's action queue. A run:

  1. waits while ~/locks/kvm-gate.lock is held by a gate;
  2. takes one of dwarf's build slots (~/.slots/slot.N, the same slots dwarf-slot.sh uses);
  3. puts every followed sibling at its branch head and the repository at the run's head, under ~/tgt/ci/src;
  4. runs the repository's own cargo xtask ci --quick (pipelines/ci-<branch>.runa.ron) in a fresh ci4 container (pinned by digest), as the forge's own user (keep-id). Its target is /backup/ci-target/<repo> (on /backup since lane CI-RED: the root array is the KVM gates' room); all targets together stay under 40 GB, and the least recently used target is removed first. The budget holds DURING the run too: a run whose target passes 40 GB, or that leaves the disk with less than 110 GB free (room for the KVM gates), is stopped red, its jobs killed and its target removed; and no run STARTS while the disk is under that floor or a gate holds its lock (it waits).

• Nightly. At 02:30 UTC, scorecard-gunnar, scorecard-holger and scorecard-heimdall run the OpenSSF Scorecard in local mode (pipelines/scorecard.runa.ron → cargo ci-verbs scorecard). The same run then copies the repository's OpenSSF Best Practices badge (cargo ci-verbs openssf-badge, projects 16084, 16086 and 16105): it fetches https://www.bestpractices.dev/projects/<id>/badge, accepts only an SVG under 20 KB with no <script>, no event attribute and no javascript: URL, and the forge (which checks it again) publishes it at badges/openssf-<repo>.svg (gunnar, holger, heimdall) in ci-status, committed only when the bytes changed. gunnar.rs's READMEs draw the badge from https://gunnar.rs/api/repos/nordisk/ci-status/raw?path=badges/…, its own origin, so no visitor's browser and no Gunnar contacts bestpractices.dev (BADGE-7).

• monetize (MON-OSSF). fuzz-monetize (04:00 UTC) fuzzes monetize's open crates (fuzz/, five targets); fuzz-monetize-impl fuzzes the private vendors' crate in monetize-impl (pipelines/fuzz-private.runa.ron → cargo ci-verbs fuzz --private monetize-impl: the private checkout read-only beside the run, publish = "verdict-only", so neither the code nor a crash input leaves dwarf; the forge does not fetch that checkout itself, keep ~/tgt/ci/private/monetize-impl at its main). coverage-monetize (05:00 UTC) runs monetize's cargo xtask build-guests, then the coverage verb over the open crates (pipelines/coverage-guests.runa.ron); deps-monetize proposes the weekly update (Monday 01:30 UTC); scorecard-monetize (02:30 UTC) is the Scorecard without a badge step (pipelines/scorecard-plain.runa.ron: monetize has no Best Practices project).

• The weekly bench (README-BENCH). The owner's standing rule (2026-10-31): a performance claim in a README comes only from a bencher run on real repositories. bench-gunnar (Saturday 07:10 UTC) and bench-holger (Saturday 09:10 UTC) run cargo ci-verbs bench-readme (pipelines/bench-readme-<repo>.runa.ron, ci/verbs/src/bench.rs):

  1. the run's checkout (with .git, without target/) goes to oden through a tar pipe over ssh, into ~/tgt/ci-bench/<repo>/src;
  2. there, in ci4 by digest and behind oden's build-slot script (~/bin/oden-slot.sh), the repository's own verbs run: cargo xtask bench-readme run --seeds /seeds --machine oden (the bencher on the real seeds, ~/bench-seeds/<repo> on oden, read-only), write (the README tables from the new result) and compare --threshold 25 (red on a regression past 25 % against the previous result of the same machine);
  3. the changed files come back (the new result JSON and the READMEs), and oden's target for the run is removed;
  4. green: one commit with no trailer (author Rickard Lundin), and propose.json names a NEW branch bench/readme-<date>-<sha8> and merge: <default branch>. The forge pushes the branch, then merges it with --no-ff in a temporary worktree on a fresh fetch of the default branch, as the proposal's author, and pushes without force (forge_ci::push_proposal, merge_proposal). A conflict or a refused push is red, and the branch stays pushed for a person. A regression is red, names it, and proposes nothing. A deps/ proposal is still never merged.

The way to oden: a step container has HOME=/tmp and no keys, so the rows set DWARVES_CI_BENCH_SSH = "ssh -F /home/rickard/.config/dwarves-ci/bench-ssh/config", and that directory is in DWARVES_RUNA_BINDS. setup.sh makes it: a key of the forge's own (id_ed25519, never copied anywhere), a config naming Host oden (100.115.167.95, user rickard, that key, its own known_hosts, BatchMode), and it prints the public key, which goes into oden's ~/.ssh/authorized_keys once. The seeds live on oden in ~/bench-seeds/gunnar (a bare git.git, later linux) and ~/bench-seeds/holger (the real-client package sets); the repositories' bench-readme run says what it needs there.

• Results. After every run the forge commits and pushes to nordisk/ci-status on git.gunnar.rs. STATUS.md (which is also the README.md) has one row per pipeline, says why a red run is red, shows each run's summary (the Scorecard table), and lists the history. Each run's files go in runs/<pipeline>/<stamp>-<sha8>/: run.json, forge.log, the repository's receipt.json and step-log tails, and for a Scorecard run scorecard.json and summary.md. Anyone can clone the repository with no account: git clone ssh://anonymous@git.gunnar.rs:2222/nordisk/ci-status.git.

• Red. A red run appends one line to ~/plan/ci-red.log. No e-mail is sent.

The CI turn at the KVM gate

Gates hand ~/locks/kvm-gate.lock to one another in the same second, so a CI run that only waited for a free lock waited for hours (2026-10-30). Now (lane CI-RED, SC-GATE-STARVE):

• While a queued CI run waits on a held gate lock, the forge asks for a turn: it writes ~/locks/kvm-gate.ci-turn and touches it every few seconds.
• When the lock comes free, the forge takes it for one CI run, at most 30 minutes, and logs CI-TURN-START / CI-TURN-DONE in ~/locks/kvm-gate.turn. Past 30 minutes it gives the lock back and the run goes on. After its turn, a waiting gate goes first (30 s).
• Gates take the lock through ~/bin/kvm-gate.sh <TAG> "<what, ~how long>" <command…> (ci/forge/kvm-gate.sh, installed by setup.sh). It waits while a fresh ask stands, gives the lock back at once if an ask came while it waited, and writes the <TAG>-START and <TAG>-DONE rc=<n> lines. A bare flock -o ~/locks/kvm-gate.lock still works, but it does not let the ask go first. An ask that was not touched for 120 s (the forge is gone) holds nobody.

CI provenance is not release provenance

A CI run writes no release provenance, and nothing it builds is a release. Releases are dwarves release-build (crate dwarves-slsa): builder https://git.gunnar.rs/nordisk/dwarves/slsa/builders/dwarves-server/v1, signed with the owner's YubiKey, SLSA Build L3. If a CI build writes provenance, its builder id is https://git.gunnar.rs/nordisk/dwarves/slsa/builders/dwarves-ci/v1. That provenance is signed with a software key and is SLSA L2 at most.

Set up, stop, start

# build (ci4, in a lane tree), then install once:
cargo build -p dwarves --no-default-features --features cli,server,runs-on-container --bin dwarves
sh ci/forge/setup.sh <target>/debug/dwarves

systemctl --user stop dwarves-ci          # stop the forge (its own processes only)
systemctl --user start dwarves-ci         # start it again
systemctl --user disable --now dwarves-ci dwarves-ci-podman   # turn CI off
journalctl --user -u dwarves-ci -f        # what it does

# run a pipeline now, without waiting for its push or its night:
touch ~/.dwarves-ci/warehouse/ci/ci/run-now/scorecard-gunnar

To see what a poll would do without running anything, set DWARVES_CI_DRY=1 in ~/.config/dwarves-ci/forge.env and restart the forge. The poll then fetches and logs DRY — would queue …, and it queues nothing.

A run's containers belong to the forge's own podman service (unix:///run/user/1001/dwarves-ci/podman.sock). They are named draupnir-runa-<forge pid>-<n>; a run the forge was stopped in keeps its container until it is stopped. To stop a run that is in progress, stop only those: podman --url unix:///run/user/1001/dwarves-ci/podman.sock ps --filter name=draupnir-runa-<pid>-.

The box's settings (forge.env)

The step image is ci-forge.Containerfile: ci4 (by digest) plus Debian's Chromium, so the quick sets that run browser tests in the step container (holger-ui, monetize) find it at $CHROME (lane CI-RED, 2026-10-30; ci4 alone left holger-ui-ci red with "chromium starts: No such file or directory"). forge.env pins it by its digest; setup.sh builds DWARVES_CI_IMAGE_TAG when the box lacks that digest. Release builds keep ci4.

A pipeline may name its own step image, by digest, with image = "…@sha256:…" in its [[pipelines]] row; the project check refuses a tag. heimdall-ci does (lane heimci, 2026-10-30): heimdall's cargo xtask ci --quick runs its scripts/ci.sh (fmt, clippy, tests, the libseccomp shim, cargo deny, fuzz smoke runs) in nordisk-cargo-toolchain:heim-ci, built from heimdall's own ci/ci.Containerfile. setup.sh builds that tag from the checkout when the box lacks the pinned digest. The shared step image is not changed for it.

A pipeline may name tools the box keeps for it, tools = ["<name>"]: the directory <warehouse>/ci/tools/<name> (or DWARVES_CI_TOOLS/<name>) is bound read-only at /opt/ci-tools/<name> in its steps; a tool the box lacks refuses the run by name. tunnr-drift (lane HEIM-TOOLING, Heimdall tooling plan step 6) is the syscall-profile DRIFT GATE: on every head of tunnr's main it builds each measured row ([[build]] in .nornir/profiles/<row>/measure.toml) and runs cargo bt measure gate --build against the row-pinned heimdall (tools = ["heimdall"], HEIMDALL_TOOL=/opt/ci-tools/heimdall/heimdall). A call a checked-in profile does not allow fails the run, named by row, program, call and test. It only checks; a profile changes by measure syscalls <row> and a reviewed commit. setup.sh builds the tool in heim-ci when the box lacks it and warns when its sha256 is not the rows' pin (builds of the same version differ between hosts). The probe needs no privileges: it runs in the plain step container (default seccomp, no-new-privileges).

gunnar-ci and holger-ci set TMPDIR=/backup/ci-tmp/<repo> (a bind in DWARVES_RUNA_BINDS): Gunnar's server keeps a 5%,2GiB disk floor, which on the root array (3.7 TB at 96 %) is 186 GiB, more than is free, so with TMPDIR there 240 of its tests were refused "storage is full" (lane CI-FIX, 2026-10-29).