.nornir/cicd-market-inspiration.md · nordisk/dwarves

CI/CD market inspiration — what dwarves borrows (anchor: TeamCity)

Why this doc. dwarves is the nordisk TeamCity-in-Rust. Survey the CI/CD field — long-popular tools and new-and-beloved ones — and pull the genuinely good ideas, even from tools we dislike. dwarves' differentiators stay: pipelines are Rust xtasks (zero YAML), the DAG is presented by facett (facett-plandag/facett-brokkrview — a visual edge YAML tools don't have), and everything is remote-controllable + robot-tested in code.

Established tools

Tool	Beloved feature	Borrow → dwarves
TeamCity (anchor)	Build chains (snapshot + artifact deps), agent requirements, pre-tested / personal builds, investigations/mute	brokkr `[[pipeline.depends]]` = the chain; content-addressed artifacts flow along DAG edges; agent requirements → jera worker-group capability match; pre-tested = run the DAG on a worktree before landing; "investigation" = attach an owner + mute to a red node (funnel/plan-DAG already has assignment).
Jenkins	ubiquity, plugin ecosystem, scripted pipelines	lesson = don't be Jenkins (Groovy/plugin sprawl); our "plugins" are just Rust crates. Keep the declarative `brokkr.toml` seed + the programmable xtask.
GitLab CI	`needs:` DAG (out-of-order), dynamic child pipelines, runners + tags	ready-frontier drain (already sindri); dynamic pipelines = an xtask emits its DAG at runtime (native, since it's code); runner tags → jera group/label match.
GitHub Actions	matrix fan-out, reusable workflows/composite actions, self-hosted runner labels, environments + required reviewers	matrix = `BuildVariantSet` (target×feature×profile); reusable = brokkr pipeline templates; runner labels → jera worker-groups; environments + approval gates → dwarves release gates (manual/approval node before ship).
CircleCI	orbs (reuse), test splitting/parallelism, fast caches	test-splitting across jera agents (gatling within an agent, jera across agents); layer/action cache already in brokkr (`ActionCache`).
Buildkite ⭐	self-hosted agents + hosted control-plane — you own the compute, they own orchestration	THE model for dwarves: `dwarves-server` = control plane (schedule/observe), `dwarves agent` = your machines run the work. Maps 1:1 onto the jera-multi-machine build below.
Argo / Tekton	k8s-native DAG, `Task`/`Pipeline` CRDs, step-as-container	container/VM steps via draupnir (KVM/OCI/Redfish) submitted as jera jobs — get k8s-style isolation without owning k8s.
Azure DevOps (borrow selectively)	YAML templates (parameterized reuse), environments + gate checks, artifact feeds/retention	template reuse → brokkr pipeline templates; gate checks (approval / external-signal before a stage) → release gates; artifact feeds → holger (the registry) with retention. Skip the rest.

New & beloved

Tool	Why loved	Borrow → dwarves
Dagger ⭐	pipelines-as-code in a real language + content-addressed caching, runs anywhere	validates our thesis: Rust xtask pipelines, zero YAML. Steal the caching model (cache by input digest — brokkr already does content-addressed).
Earthly	repeatable containerized builds (`Earthfile`), strong caching	reproducibility = draupnir container step + content-addressed artifacts; byte-identical re-run is already the bench/roundtrip law.
Depot / Namespace / Blacksmith	fast remote builders + shared remote cache, ephemeral runners	remote builders = jera worker-groups on beefy boxes (Loki/Odin); shared cache = the content-addressed action cache served from the warehouse.
Woodpecker	lean, self-hostable, open Drone-style	the "small & self-hosted" ethos — dwarves is one binary + agents, no SaaS.

dwarves synthesis — the top features to build (mapped)

Distributed agents / worker-groups (jera multi-machine) — the keystone; sketch below. (Buildkite/GHA/TeamCity-agents/Depot)
DAG with needs + matrix fan-out — brokkr DAG + BuildVariantSet; sindri drains the ready frontier. (GitLab/GHA)
Build chains: snapshot + artifact deps across pipelines — [[pipeline.depends]], content-addressed artifacts along edges. (TeamCity)
Dynamic pipelines — an xtask emits its own DAG at runtime. (GitLab child pipelines)
Approval / environment gates — a manual/external-signal gate node before ship. (GHA environments / Azure gates)
Pre-tested / personal builds — run the DAG on a worktree branch before it lands. (TeamCity)
facett presents the DAG — facett-plandag/facett-brokkrview render the live pipeline (stations light up); the visual edge over every YAML tool. (our own)
Content-addressed cache + reproducible container steps — already seeded; extend with a warehouse-served shared cache. (Dagger/Earthly/Depot)

Two standing differentiators: pipelines are Rust xtasks (zero YAML); the whole thing is remote-controllable and robot-tested in code (self-driving, per the autotest law).

jera multi-machine — design sketch (the keystone build item, ties H1)

Today jera = local job ledger + process/VM/container runners; brokkr's AgentPool is in-process. Target = the Buildkite model: hosted control-plane + your own agents, over gRPC.

control plane:  dwarves-server  (sindri scheduler + brokkr DAG + jera ledger)
                      │  gRPC (bearer + dwarves-workspace header)
   ┌──────────────────┼───────────────────────┐
 dwarves agent      dwarves agent           dwarves agent      ← your machines
 --group loki       --group odin            --group nuc
 caps=[gpu,12c]     caps=[32c]              caps=[6c]

Agent register: each machine runs dwarves agent --group <name>; registers {id, host, group, capabilities[], cores, mem, capacity} with the server (mirrors the bench machine-roster discipline — record machine+cores+mem on every job).
Schedule (sindri drain): each ready step declares required {group, caps}; sindri matches it to a capable idle agent in the target group (+ the exclusive gate), dispatches over gRPC.
Bidirectional kill / lifecycle: the job lives in the ONE jera ledger; cancel job → kill remote work; agent/work dies → job → Killed (never a silent stall — RAGNARÖK).
Artifacts: flow between agents along DAG edges, content-addressed (artifact::content_id) so a downstream agent pulls exactly the produced inputs.
The seam that keeps it clean (Law #1 — jera owns it, shared by nornir/dwarves/holger): an Agent dispatch trait with a LocalAgent (in-process, single-machine — ship first) and a GrpcAgent (remote worker-group — add second) impl. sindri/jera schedule against the trait, so going distributed is adding an impl, not a rewrite — and it lands in jera (edda), not dwarves, so all apps get it. Read jera + gatling from source; never hand-roll threads.

Verdict: jera-multi-machine is the #1 dwarves build item and the biggest single lever — it's what turns dwarves from a single-box forge into a real CI/CD fleet, and (per the reuse-first law) it must be built into jera as the shared Agent/GrpcAgent seam, not bolted onto dwarves alone.