.nornir/daemonize-and-durable-cron.md

dwarves daemonize — `dwarves serve` + durable server-side cron

Class: design · Function axis: f0 the long-running forge daemon · f1 one shared server bootstrap · f2 durable server-side cron · f3 multi-target builds via nornir-build-thing. Strict: this file states only what the daemon IS — its parts and why they are shaped this way. Sibling: the systemd wrapping is service-lifecycle-L1.md; the client modes are dwarves-parity-spec.md.

dwarves is the CI/CD forge, and a forge has to stay up: hold the pipeline registry, answer gRPC, and fire scheduled pipelines on their own minute without anyone watching. This is that daemon — one server bootstrap reached through two doors (dwarves serve and the dwarves-server bin), with a durable cron sweep that fires each scheduled pipeline exactly once per matching minute, even across a restart.

The system as one — the function map (f0)

part	lives in	what it is
`f1` one server bootstrap	`crate::server::serve` / `serve_blocking`	binds the addr, creates the warehouse, resolves the bearer token, spins the cron thread, serves `dwarf.v1` until shutdown — the single source both doors call
`f1a` in-process door	`main.rs` `serve_cmd`	the `dwarves serve` subcommand (fat, `--features server`) → `serve_blocking`
`f1b` standalone door	`bin/dwarves-server.rs`	the thin bin — skidbladnir L1 verbs, then hands off to the SAME `serve`
`f2` durable cron	`server_cron::CronStore` / `sweep_cron`	a redb KV (`pipeline-key → last-fired minute`) that makes sindri's in-memory dedup survive a restart
`f3` multi-target build	`runner::RealRunner` → `nornir-build-thing`	each `Variant` step is one `build_variant` call — holger's 3 execs are the first consumer

Why one bootstrap (Law #1: never a twin)

Before this landed, the whole dwarf.v1 service, the bearer interceptor, the jera-Scheduler wiring and the serve-loop lived inside the dwarves-server bin, and dwarves serve could only print a pointer at it. That is a twin waiting to drift. The fix folds the entire boot into the lib (crate::server), so both entrypoints run the identical daemon — same forge, same cron, same jera Scheduler. The dwarves-server bin keeps only what is genuinely its own: the skidbladnir lifecycle verbs (--unit/--install/--start/--stop/--status), which act and exit before the shared serve boots.

serve(real) is async (tonic); serve_blocking(real) builds a multi-thread tokio runtime and drives it — the seam the synchronous CLI subcommand needs. real selects the cargo-building RealRunner over the hermetic DemoRunner (also flipped by $DWARVES_REAL_RUNNER).

Place in the nordisk constellation

The daemon reuses, never hand-rolls (Law #1):

jera ([jera::agent::Scheduler]) is the single placement authority. One Scheduler is created before the loop and shared by both the RPC handlers (Trigger/Webhook) and the cron sweep — so every path drains through one placement decision. LocalAgent runs it single-machine today; a GrpcAgent makes the same path multi-machine with no change to the server.
skidbladnir owns the systemd lifecycle — see service-lifecycle-L1.md. The daemon itself starts no unit files.
nornir-build-thing owns multi-target cargo builds — RealRunner resolves each Variant step to a build_variant call over a discovered BuildVariantSet, never a hand-rolled per-target invocation.
redb is the on-disk KV (mirroring jera's single-writer jobs.redb and skade's catalog.redb) — not a new engine.

The server interceptor deliberately mirrors nornir-server (same authorization: Bearer <token>, same dwarves-workspace metadata header, same serve-loop) as a clean extraction candidate. server.rs carries the standing TODO(single-source): lift the server + thin-client scaffold into a shared edda crate nornir-service consumed by nornir + holger + dwarves. Until that lands this is the honest stopgap, not a competing framework.

Durable server-side cron

sindri's enqueue_cron already dedups a cron pipeline to at most once per matching minute — but only in memory. The server rebuilds a fresh Sindri every sweep, so without help that dedup would be lost across sweeps and across a restart, and the pipeline would double-fire.

server_cron makes the dedup durable with the smallest possible addition — one redb table:

CRON_LAST : &str -> &str      // "project/pipeline"  ->  "YYYY-MM-DDTHH:MM"

Each sweep is seed → tick → persist:

rebuild a Sindri from the current registry,
seed_cron_last(store.load()?) — pull the persisted last-fired map back in,
tick_scheduled(now, …) — cron match → enqueue → drain through the jera Scheduler,
store.save(sindri.cron_last()) — write the updated map back.

Because step 2 reads from disk, a minute that already fired does not fire again — whether the sweep merely runs twice or the whole process restarted. The scheduling decision stays in sindri's pure cron_matches/enqueue_cron; server_cron adds only the KV. The loop ticks every 60s (not 300s) so minute-granular crons are never skipped, and runs on a std::thread because Sindri::tick is synchronous by design.

Warehouse / storage seam

Listen addr: $DWARVES_SERVER_ADDR, default 127.0.0.1:7900.
Warehouse root: $DWARVES_WAREHOUSE, else ~/.dwarves/warehouse, else .dwarves/warehouse.
Bearer secret: $DWARVES_SERVER_TOKEN, else ~/.dwarves/token if present, else empty = open (dev).
Cron state: <warehouse>/cron.redb, single-writer (the server owns it), a mutable short-lived record — never Iceberg/historized. Distinct from jera's jobs.redb.
Runs themselves historize through the existing engine/store into the warehouse (unchanged by this feature).

Tests (inject-assert, LAW 1)

server_cron::tests::durable_cron_fires_once_per_minute_across_restart pins the whole contract on a tempdir warehouse: a 03:00 nightly cron fires once and drains through the scheduler; a second sweep in the same minute with a fresh Sindri does not re-fire; after drop + reopen (a simulated restart) the last-fired minute is still on disk and still does not double-fire; a non-matching 04:00 minute is a clean no-op.
runner::tests::a_pipeline_builds_multiple_variants_one_per_step proves the 3-variant build path.
testmatrix cell dwarves/server-cron (sweep_cron emits under feature = "testmatrix" / test).
cargo test -p dwarves --features server → 77 lib + 5 cli_smoke green; dwarves serve live-verified to boot, answer Health, and create cron.redb.

No robotui scenario: the daemon and cron sweep are a headless gRPC + timer surface, not a UI. The UI surfaces (Forge/Funnel/Claude panes) carry their own robotui scenarios; see three-pane-shell.md and funnel-live-graph.md.

Source map

one server bootstrap → crates/dwarves/src/server.rs (serve, serve_blocking, DwarfSvc, auth_interceptor, default_addr/default_warehouse/resolve_token)
in-process door → crates/dwarves/src/main.rs (Cmd::Serve → serve_cmd)
standalone door → crates/dwarves/src/bin/dwarves-server.rs (L1 verbs → shared serve)
durable cron → crates/dwarves/src/server_cron.rs (CronStore, sweep_cron, CRON_LAST)
multi-target build → crates/dwarves/src/runner.rs (RealRunner, nornir_build_thing::build_variant)

dwarves daemonize — dwarves serve + durable server-side cron