nordisk/nornir

The job ledger — design

Class: design · Function axis: f0 (the ledger as one system) + f1 the single-writer store · f2 lifecycle & the serial scheduler · f3 container jobs · f4 the read surface + CLI. Strict: this file states only what the ledger is. The why — redb-not-Iceberg, why every long op is a job, rejected alternatives — is in jobs-reasoning.md. Sibling: jobs-guide.md.

f0 — the job ledger (the whole)

Every long-running nornir operation (~1s–minutes) is a tracked job: release runs, workspace fetch/republish/populate, docs render/export/book, knowledge scan, deepscan, bench, test matrix, arch generate, index build, vector embed, bakeoff, airgap, coverage, security scan, warehouse compact/expire, and container runs. Each op's result still lands in its domain Iceberg table; the ledger is one orthogonal place that answers "what's running / did it finish / how long did it take?" — the op lifecycle, not its output.

The ledger is a single redb file, <warehouse root>/jobs.redb, distinct from skade-katalog's catalog.redb and the server's registry.redb. It follows the same fat-XOR-thin split as the warehouse: a fat process writes the local file directly; a thin client submits each record over gRPC to the server, which owns its jobs.redb. One writer owns the file; the 🧬 viz observes by reading the same redb read-only. The engine is the self-contained leaf crate nornir-jobs, re-exported as nornir::jobs.

Reuse map (what the jobs subsystem uses). The leaf crate depends only on redb + serde + anyhow + tokio (the exclusive gate) + chrono + uuid + tempfile — no tonic (the binary supplies the gRPC submit closure), no Iceberg. It is used by every op family across the CLI and nornir-server, and by the viz through Viz.Jobs.

f1 — the store: one mutable row per op (jobs.rs)

A job is a single redb row, key = job_id, value = JSON JobRecord (jobs.rs:207). The same key is overwritten in place as the job advances — no append-only two-rows-then-coalesce, no Iceberg snapshot per transition.

JobRecord { job_id, kind, target, workspace, status,
            ts_start_micros, ts_end_micros, elapsed_ms,
            detail_json, result_ref, parent_id }

• kind (jobs.rs:74) is a string, one per op family (release_run, bench_run, docs_render, container, …) — a new op needs no schema change.
• result_ref links the job back to its domain Iceberg row (e.g. bench_runs:<uuid>, release_events:<run_id>).
• detail_json / result_ref are free-form ("" when absent), so the record never needs a migration as new ops adopt the ledger.
• parent_id (jobs.rs:236, #[serde(default)]) nests a sub-job under its parent: a workspace populate is a workspace_populate parent with a workspace_clone child per member (streamed) plus a workspace_republish build child; readers fold children under the parent.

JobStore (jobs.rs:507) owns the redb file. open (jobs.rs:516) is the sole-writer path; open_read_only (jobs.rs:536) is lock-tolerant — if the live server holds the exclusive lock, it copies the file aside across a size-stable window (copy_redb_consistent, jobs.rs:723) and opens the snapshot, the same trick as warehouse::IcebergWarehouse::open_read_only. Single-writer discipline matches registry.redb.

JobSink (jobs.rs:280, cheap Arc clone) is the write side, in three forms: JobSink::redb (jobs.rs:286, fat — upsert into the local file, prune on terminal writes), JobSink::remote (jobs.rs:303, thin — hand each record to a closure that does the Jobs.Submit RPC), and JobSink::noop (jobs.rs:309, drop every record, so call sites stay unconditional). A sink write failure is logged and swallowed — a ledger hiccup never aborts the op it tracks.

f2 — job lifecycle & the serial scheduler (JobHandle, class_of)

A JobHandle (jobs.rs:326) wraps one op's lifecycle. status (jobs.rs:59) has two non-terminal states (queued, running) and two terminal (done, failed):

The Drop guard (jobs.rs:482) records failed ("dropped without finish/fail") so a panic or early ? never leaves a zombie running row.

The serial scheduler. Some ops are box-saturating and only measure true in isolation (a benchmark sharing the machine with another benchmark measures noise). Each kind has a JobClass via class_of (jobs.rs:178):

is_exclusive (jobs.rs:188) is the predicate; adding a heavy kind is one arm in class_of, no call-site change. The gate is a process-global tokio::sync::Semaphore(1) in a OnceLock (exclusive_gate, jobs.rs:198). start_scheduled (jobs.rs:393): a parallel kind behaves exactly like start; an exclusive kind first writes a queued row (so the wait is visible in nornir jobs / the viz), .awaits the permit, then flips queued → running in place — re-stamping ts_start_micros so elapsed_ms measures real work, not the queue wait — and holds the permit until it terminates. So N benches submitted together are one running + (N−1) queued, draining one at a time.

f3 — container jobs (run_container)

run_container(sink, spec, target, workspace) (jobs.rs:823) runs a container as a tracked job — the generic "run a one-shot container and track it" seam.

• ContainerSpec { engine, image, cmd, run_flags } (jobs.rs:753) → argv <engine> run <flags> <image> <cmd…> (argv, jobs.rs:779); ContainerSpec::podman (jobs.rs:768) is the podman run --rm convenience.
• Spawns the engine as a subprocess (exactly how nornir shells out to cargo/git — no container library linked), waits, captures stdout/stderr truncated to LOG_CAP_BYTES = 16 KiB/stream (jobs.rs:799), stamps pid + exit_code + logs into detail_json.
• exit 0 → finish (done); non-zero/signal → fail_with_detail (failed), and the call still returns the ContainerOutcome so the caller reads exit_code. A spawn/wait error returns Err and the Drop guard records failed.

f4 — the read surface + CLI

--json emits the raw Vec<JobRecord> — the SAME records the viz renders (CLI ⟺ viz parity).

JobStore::list(sel) (jobs.rs:615) returns rows newest-first, filtered by a JobSelector (All / Workspace / Kind, jobs.rs:248); the server applies a thin client's record via JobStore::submit (jobs.rs:606). Both write paths prune on terminal writes.

Retention

Terminal jobs older than KEEP_DAYS = 30 are pruned opportunistically on each terminal write, and the total is capped at CAP = 2000 rows (oldest terminal first). Running jobs are never pruned (redb_prune, jobs.rs:652).