nordisk/dwarves

ci/: the constellation's CI runner

ci/runner.sh checks every repository of ci/repos.txt out side by side from local bare mirrors, runs the constellation lock check, then each CI repository's own verb (cargo xtask ci --nightly|--quick, or script:<path>@<image> for heimdall) one at a time, and writes a receipt per repository ($CI_WORK/runs/<stamp>/<repo>/receipt.json) and the status page $CI_WORK/STATUS.md. ci/mirror.sh fills the mirrors from dwarf, which has the forge access; the CI host never fetches from a forge. Public repositories only.

CI on oden

oden has 32 cores and is shared with other sessions, so the pass stays within 20 cores: one CI job at a time, CARGO_BUILD_JOBS=16, nice 10. From dwarf, in a dwarves checkout:

bash ci/oden.sh                 # fill oden's mirror, then a full pass (runner.sh --nightly), detached
bash ci/oden.sh tunnr           # rerun one row (or several) after a fix: CI_ONLY
bash ci/oden.sh --wait          # the same, and wait for the end; prints STATUS.md
bash ci/oden.sh --status        # oden's STATUS.md
bash ci/oden.sh --log           # the tail of the newest pass's log

What it does:

1. mirror.sh with CI_HOST=rickard@100.115.167.95 CI_REMOTE_WORK=/home/rickard/lanes/ci and CI_LOCK_WAIT=1 (it waits for the dwarf mirror timer's pass instead of skipping).
2. src/dwarves on oden at its mirrored main (the runner and repos.txt come from there).
3. The images: the runner's pinned toolchain image (CI_IMAGE, by manifest digest) and heimdall's heim-ci (by image ID in repos.txt). A missing one is copied from build-1's builder store (podman save --format oci-archive <ID> | podman load, which keeps the ID) and tagged localhost/nordisk-cargo-toolchain:ci-pin-<id12>.
4. The pinned heimdall of tunnr's evidence steps, handed on as CI_HEIMDALL_TOOL: oden.sh reads the digest tunnr's evidence rows pin (.nornir/evidence/*/evidence.toml measure_row → .nornir/profiles/<row>/measure.toml [tool] sha256) off the mirrored head and names /home/rickard/lanes/ci/pins/heimdall-<sha16>. Copies there: 0.4.3 f093633c… (build-1 /work/tunnr-open/ovn-bench/heimdall-bin/heimdall) and 0.5.0 c0f3d57b… (oden, lane floor-tools). When tunnr moves its pin, copy the new binary there under its digest; without it the evidence:<row> steps are red, and oden.sh says so before it starts.
5. runner.sh --nightly in tmux session ci-oden-<stamp> on oden, with CI_WORK, CI_JOBS=16, CI_NICE=10, CI_SCCACHE_PORT=4237 (another session's sccache server holds the default port on oden), CI_STEP_TIMEOUT=4h (it bounds a repo's whole cargo xtask ci: holger's full set needs more than 2 h), CI_LOCK_WAIT=1 (a second start waits for the pass going, it never doubles it) and CI_ONLY for the rows named.

The runner's home ($CI_WORK/home) keeps trunk's tool cache (.cache/trunk, the wasm-bindgen the UI builds download from GitHub): a 504 from GitHub on the first fill is a red fast step; a copy of ~/.cache/trunk/wasm-bindgen-<version> there avoids it.

Output on oden: log /home/rickard/lanes/ci/logs/<stamp>.log, receipts /home/rickard/lanes/ci/runs/<stamp>/<repo>/receipt.json, status /home/rickard/lanes/ci/STATUS.md. Stop a pass: tmux kill-session -t ci-oden-<stamp> on oden (the session name is printed at the start).

The runner's settings (environment)

Every command a step starts gets neither of the runner's lock descriptors (8, 9): a daemon a step starts (the sccache server of a host build) cannot hold the next row back.