nordisk/nornir

🧾 The unified job engine — every long-running op is a tracked job (design + guide)

One durable job ledger answers "what's running / did it finish / how long did it take" for every long-running nornir op (release, bench, test matrix, docs, coverage, arch-generate, airgap pack, knowledge scan, index build, vector embed, bakeoff, container runs). The op result still lands in its domain Iceberg table; the ledger tracks the op lifecycle, orthogonal to it. Engine: src/jobs.rs. Plan: nordisk_plan.md (JOB-ENGINE-SWEEP).

1. The model — JobHandle over a JobSink

A job is a single mutable redb row (<root>/jobs.redb, key = job_id, value = JSON JobRecord):

JobRecord { job_id, kind, target, workspace, status, ts_start_micros,
            ts_end_micros, elapsed_ms, detail_json, result_ref }

status is queued → running → done | failed (queued and running are the two non-terminal states — queued only ever appears for heavy/exclusive kinds waiting on the serial scheduler, see §3). The same key is overwritten in place as the job advances — no append-only "two-rows-then-coalesce" dance, no Iceberg snapshot per start/finish (that would be the small-files antipattern). result_ref links the job back to its domain row (e.g. test_results:<run_id>, architecture_wiring:<id>).

A [JobHandle] wraps one op's lifecycle:

• start(sink, kind, target, workspace, detail) → writes the running row (immediate, ungated).
• start_scheduled(sink, kind, …).await → the serial-scheduler entry (§3). Parallel kinds behave like start; exclusive kinds (bench) are admitted queued, then wait on the gate before flipping to running. JobStore::start_scheduled(…) is the fat convenience.
• finish(detail, result_ref) → terminal done.
• fail(&err) → terminal failed, carrying the error chain.
• fail_with_detail(detail) → terminal failed carrying caller JSON (the "ran-but-failed" case, e.g. a container that exited non-zero — there is no anyhow::Error, just an exit code + logs).
• Drop guard: a handle dropped without a terminal call records failed ("dropped without finish/fail") — so a panic or early ? never leaves a zombie running row.

The write side is a [JobSink] (cheap Arc clone): fat = redb over the local jobs.redb; thin = a closure that submits each record over gRPC (Telemetry.SubmitJobs) to the server, which owns its jobs.redb. The library never links tonic — the binary supplies the closure. This is the SAME fat-XOR-thin split the warehouse uses.

2. Reading the ledger

--json emits the raw Vec<JobRecord> — the SAME records the viz renders (CLI⟺viz parity).

3. Scheduling — heavy jobs run one-at-a-time (serial gate)

Some ops are box-saturating and only measure-true in isolation — a benchmark sharing the machine with another benchmark measures noise. The server is one long-lived process, so two benches launched together (e.g. two Ops.RunBench calls) would otherwise both run at once. The scheduler stops that without serializing everything.

Concurrency class (src/jobs.rs). Each kind has a JobClass via class_of(kind):

is_exclusive(kind) is the convenience predicate. Adding a heavy kind = one arm in class_of; no call-site change. The default is parallel — only genuinely saturating, isolation-demanding work is made exclusive.

The gate. A process-global tokio::sync::Semaphore(1) (a OnceLock, so every call site shares the one instance). JobHandle::start_scheduled(…).await:

1. Parallel kind → identical to start (writes running, returns immediately).
2. Exclusive kind → writes a queued row first (so the wait is visible in nornir jobs / the viz job list), then .awaits the semaphore. While another exclusive job holds the permit the row sits queued — not running, not invisible. On acquiring the permit it flips queued → running (re-stamping ts_start_micros so elapsed_ms measures real work, not the queue wait) and returns a running handle. The handle holds the permit until it terminates (finish/fail/Drop), at which point the next queued exclusive job is admitted.

So N benches submitted together produce one running + (N−1) queued rows, draining one at a time — the user sees the queue, and no two benches ever run concurrently.

Wiring. The server's Ops.RunBench (src/bin/nornir-server.rs) calls wctx.jobs.start_scheduled(kind::BENCH_RUN, …).await before any bench work — the canonical heavy start site. The fat CLI nornir bench run path records its bench job synchronously (start_job, no async runtime) and is each its own short-lived process, so a process-global gate there is a no-op; cross-process bench serialization is intentionally the server's job (the long-lived process that owns the warehouse + ledger). Thin nornir bench already routes to the server's Ops.RunBench, so it inherits the gate.

4. What's wired (the JOB-ENGINE-SWEEP audit, 2026-06-23)

Already tracked (server + most CLI): release run/undo, workspace fetch/republish, docs render/export/book, knowledge scan, bench run, bakeoff, index build, vector embed.

Gaps the sweep closed:

• No CLI reader → added nornir jobs (§2).
• Untracked local/fat CLI paths (their server twins had jobs, the embedded path didn't): nornir coverage (kind::COVERAGE), local nornir test run matrix (kind::TEST_MATRIX), nornir arch generate/svg (kind::ARCH_GENERATE), nornir airgap pack (kind::AIRGAP). Each wraps the existing op (start_job → finish on green / fail on error), never rewrites it.

4b. The "every server op in the jobs panel" sweep (2026-06-24)

Several server-side RPC handlers that do real work (mutate the warehouse or shell out) still had NO ledger wiring, so the op was invisible in the Jobs panel. Closed by wiring each handler jobs.start(kind, …) → finish/fail (best-effort — a missing handle never gates the op, mirroring the register-eager error handling):

New kinds added in [crate::jobs::kind]: SNAPSHOT, SYMBOL_SCAN, SECURITY_SCAN.

Proven through-the-server (LAW 4) + empty→0 (LAW 2) by tests/server_ops_jobs.rs: drives Mimir.SecurityScan, Docs.Render, Release.GateAll via gRPC against a live server monitoring this repo, asserts 0 jobs of each kind before and a fresh terminal job (read back through Viz.Jobs) after.

5. Container jobs — podman run … AS a tracked job

jobs::run_container(sink, &ContainerSpec, target, workspace):

• ContainerSpec { engine, image, cmd, run_flags } → argv <engine> run <flags> <image> <cmd…>. ContainerSpec::podman(image, cmd) is the podman run --rm convenience.
• Spawns the engine as a subprocess (exactly how nornir already shells out to cargo/git — no container library linked), waits, captures stdout/stderr (capped at 16 KiB/stream so a chatty container can't bloat the row), stamps pid + exit_code + logs into detail_json.
• exit 0 → done; non-zero/signal → failed (via fail_with_detail); the call still returns the ContainerOutcome so the caller reads exit_code. A spawn/wait error returns Err (engine missing) and the Drop guard records failed.

CLI: nornir container --image <img> [--engine docker] [--flag '-e FOO=bar'] [--target label] -- <cmd…> — prints the captured output and propagates the container's exit code (so &&/CI gating works). Uses the same fat-or-thin sink as start_job, so a thin client submits the row to the server.

This complements (does not replace) the bench/test fixture containers (skade bench-containers, plan point 7): those bring up long-lived service containers in Rust; this is the generic "run a one-shot container and track it as a job" seam.

6. Tests (inject-assert, fail-on-bug)

src/jobs.rs #[cfg(test)]:

• start_finish_fail_and_drop_zombie_guard, running_then_done_overwrites_in_place_no_duplicate, prune_drops_aged_terminal_and_caps_count_but_keeps_running, job_record_vec_json_round_trips (pre-existing).
• container_spec_argv_is_engine_run_flags_image_cmd, truncate_log_caps_and_marks, container_job_missing_engine_records_failed_and_errs (no engine needed), container_job_real_podman_echo_tracks_done, container_job_nonzero_exit_records_failed_keeps_outcome (podman-gated, REAL runs).
• Scheduler: classify_bench_exclusive_others_parallel (bench is Exclusive, the rest Parallel); exclusive_bench_jobs_serialize_second_observes_queued (two BENCH_RUN jobs serialize — the second is observed queued while the first is running, then runs after the first finishes); parallel_job_not_blocked_by_running_exclusive (a release_run starts running immediately despite a bench holding the gate). The two gate tests share the process-global semaphore, so a std Mutex serializes them within the test binary.
• Viz surface (src/viz/nornir_tab.rs): state_json_surfaces_queued_bench_behind_running_bench injects a running bench + a queued bench through the real job_entry map and asserts state_json["nornir"]["jobs"] counts 1 running / 1 queued and renders the waiting bench AS queued (the visible-wait UX), with the Jobs surface wired to Viz.Jobs.

tests/cli_jobs.rs (drives the real binary against an isolated $HOME + explicit --config):

• jobs_empty_then_reflects_seeded_records — inject records via the library, read them back through every nornir jobs flag path.
• container_command_runs_and_tracks_a_job, container_command_propagates_nonzero_exit_and_records_failed (podman-gated, real nornir container runs).

7. Retention

Terminal jobs older than KEEP_DAYS (30) are pruned opportunistically on each terminal write, and the total is capped at CAP (2000) rows (oldest terminal first). Running jobs are never pruned.