nordisk/edda

jera agents — multi-machine worker-groups (the distributed-runner seam)

Class: design · strict "what it IS". Sibling: the job ledger (jobs) records that work ran; this module (jera::agent) is where it runs.

f0 — the whole

One shared runner layer for the constellation (nornir · dwarves · holger consume it — never a per-app twin, reuse-first law). An Agent is a worker in a named worker-group advertising AgentSpec{ id, group, caps, cores, mem_mb }. A Scheduler (the control-plane) matches a ready WorkUnit{ id, group, required_caps } to the first capable idle agent in the target group and dispatches it, returning a WorkHandle with bidirectional kill.

f1 — capability match (the schedule rule)

Agent::can_run(w) = idle ∧ agent.group == w.group ∧ w.required_caps ⊆ agent.caps. Scheduler::match_agent returns the first eligible agent in registration order (deterministic); dispatch returns None when none is eligible — the caller re-queues, never a silent drop (RAGNARÖK).

f2 — bidirectional lifecycle (WorkStatus)

Pending → Running → Done | Failed | Killed. Two kill directions, both real+tested:

• cancel → kill: WorkHandle::kill() sets the flag the runner polls → Killed.
• agent death → kill: Agent::go_offline() marks Offline and kills every in-flight unit → Killed. No orphaned work.

f3 — the two backends (the seam)

Going multi-machine = implementing GrpcAgent, not rewriting the scheduler. The Buildkite split: one control-plane (Scheduler, e.g. inside dwarves-server) + N <host> agent --group … workers registering their AgentSpec, then dispatch/poll/kill as gRPC calls. Everything above GrpcAgent stays byte-identical.

f4 — observability

Scheduler::state_json() exposes the fleet as data (id/group/caps/cores/mem/state) — feeds an "Agents" pane and a nornir-testmatrix cell at the consumer (nornir/dwarves), so the robot/matrix asserts the pool in code (remote-control-in-code law). jera stays lean (no testmatrix dep in this foundational crate; the cell lands where the pool is wired to a UI/op).

Tests (drive-in-code)

agent::tests: group+caps match→Done · no-eligible→None (no silent drop) · cancel→Killed · agent-offline→in-flight Killed · busy-not-rescheduled · GrpcAgent-offline-skeleton-never-scheduled · state_json-as-data. 7/7 green, pure std (no network).

The transport seam (remote path — loopback REAL now)

GrpcAgent no longer hard-codes Offline: it delegates state/dispatch/go_offline to an AgentTransport. Two impls:

• LoopbackTransport — in-process, runs the unit on a real LocalAgent behind the transport boundary. REAL + tested: grpc_agent_over_loopback_transport_runs_to_done (scheduler → remote agent → transport → Done) and ..._kill_propagates (client kill() crosses the seam → Killed). So the full distributed path — match, dispatch, bidirectional kill — is exercised in code, no socket, before any wire exists. GrpcAgent::loopback(spec, runner) builds one.
• tonic GrpcTransport — the ONLY remaining wire, a drop-in impl AgentTransport behind a future agent-grpc feature (keeps jera's lean default tonic-free). Register/dispatch/kill/status over gRPC; GrpcAgent::with_transport(spec, endpoint, grpc) swaps it in. Nothing above the seam changes — the Scheduler, capability match, and kill semantics are already proven against loopback.
• GrpcAgent::new(spec, endpoint) stays the safe unwired default (UnwiredTransport → Offline, never scheduled).

Remaining wire bits: the agent-grpc proto (register/dispatch/kill/status) + tonic client/server; a <host> agent --group … worker that registers its AgentSpec and runs units. (dwarves already consumes the Scheduler + ships the dwarves agent CLI over the LocalAgent path — commit 055da45.)

Reuse / boundaries

Pure std (threads + atomics), additive to jera's public surface. Provisioning of the container/VM a remote unit runs in still goes through draupnir (jera::{container,vm} → policy::JobPolicy) — the agent layer schedules where, draupnir fires the instance.

agent-grpc wire — LANDED (feature agent-grpc, 2026-07-13)

The remote transport is now real (the drop-in the LoopbackTransport proved):

• client GrpcTransport impl AgentTransport — connect(endpoint) → wrap in GrpcAgent::with_transport(spec, endpoint, transport); remote_state/remote_dispatch /remote_go_offline bridge sync→tonic via a private multi-thread runtime.
• server serve(addr, Arc<LocalAgent>) — a tonic AgentWorker over a LocalAgent (the worker box: <host> agent --group …).
• wire proto/agent.proto (jera.agent.v1: State·Dispatch·Poll·Kill·GoOffline), pure-Rust codegen (protox, no protoc), build.rs gated on CARGO_FEATURE_AGENT_GRPC.
• bidirectional kill crosses the wire; unreachable worker ⇒ Offline (never scheduled).
• lean stays lean: tonic/prost link ONLY under agent-grpc; nornir/dwarves/holger consume jera unchanged.

Proven in code (no manual run): grpc_dispatch_runs_to_done_over_the_wire + grpc_kill_propagates_to_killed_over_the_wire (in-process tonic on a loopback port). Scheduler/match/kill semantics unchanged — this only added the socket. Next: a <host> agent --group <name> worker binary that calls serve, and the control-plane registering GrpcAgents from a worker roster.

Real cross-machine runner — stages 1–3 LANDED (2026-07-14)

The seam above proved where work runs; these three additive stages make jera a real cross-machine runner (what runs, which fleet, which node ran it). Each is a green commit; the single-machine Scheduler/WorkUnit/LocalAgent API is untouched (L2 additive).

Stage 1 — payload-carrying WorkUnit (the keystone)

WorkUnit gained payload: Option<WorkPayload> (#[serde(default)], skip_serializing_if), so a unit now says what to run instead of relying on a fixed pre-bound closure. A payload-less unit decodes as None and keeps the legacy closure path — additive.

• enum WorkPayload { Process(ProcessSpec) | Container(ContainerRun) | Vm(VmBoot) } — self-contained + serde-ready (no draupnir type on the wire); the worker maps a payload back onto jera's own boot/container/vm runners at the far end.
• WorkPayload::policy() -> JobPolicy links each leg to jera's policy classification; WorkPayload::process(prog, args) is the ergonomic Process constructor.
• WorkUnit::with_payload(p) (builder) · WorkUnit::payload() -> Option<&WorkPayload>.
• Wire: WorkUnitMsg.payload_json (proto field 4, "" = legacy) carries the serde-encoded payload across the agent-grpc socket, reconstructed server-side; proven by grpc_dispatch_carries_payload_over_the_wire (the remote runner reads the exact ProcessSpec). Scheduler::dispatch emits a jera-agent functional row for payload units (observable at the consumer).

Stage 2 — durable node roster + heartbeat/liveness (agent::roster)

Roster is the durable list of fleet nodes (Loki 12c · Odin 32c · NUC 6c + Njord), each a NodeEntry { spec, endpoint, last_heartbeat_micros }.

• register(spec, endpoint, now) · heartbeat(id, now) -> bool · is_live(id, now) · live(now) / dead(now) · evict_dead(now) (emits a jera-roster row per eviction) · load/save (JSON durable store) · state_json(now).
• No clock coupling: every liveness method takes now, so "stale past the window ⇒ dead" is deterministically testable with no network. No new RPC — the existing agent-grpc State probe is the liveness ping; the consumer calls heartbeat after a successful probe.

Stage 3 — JobRecord.node (which machine ran it)

JobRecord gained node: Option<String> (#[serde(default)], non-breaking like parent_id). JobHandle::on_node("odin") (builder, re-upserts) stamps it; JobHandle::node() reads it back. A distributed job now records its machine; a local job stays None; a pre-field row decodes as None (no migration).

Stages 4–5 + wire auth — LANDED (2026-07-14)

The return leg, the secured wire, and resource-aware fleet placement — the pieces a serial HEAVY bench dispatched across the fleet needs to route heavy runs to Loki/Odin and record each machine's result centrally. All additive; the stage 1–3 API is byte-unchanged.

Stage 4 — worker result return (WorkResult → JobRecord.node)

WorkResult { job_id, unit_id, status, node, exit_code, result_ref, detail_json } is the terminal outcome a worker returns for a dispatched unit — serde-ready so it round-trips over the return wire.

• WorkResult::observed(job_id, handle, node) forms it from a terminal handle; with_result_ref / with_exit_code / with_detail are builders.
• to_job_record(kind, target, workspace, ts_start) maps it onto a jobs::JobRecord — status folded (Killed → failed), node stamped, timing filled — to hand to JobStore::submit (the existing thin-client Jobs.Submit write path, reused). The mapping lives in agent so jobs keeps no reverse dep on agent.
• Artifacts stay out of the record. Bulk output rides nornir-flight (do_put/do_get); result_ref carries its flight ticket (or a domain row ref like bench_runs:<uuid>) and the control-plane fetches the bytes by that ticket — so jera stays a lean leaf and never links Arrow (L5a).
• RED: dispatched_units_result_round_trips_back_and_records_against_its_node — the result survives serde AND lands in the ledger against the right node (odin).

Wire auth + worker entrypoint (secure the agent-grpc wire)

The agent-grpc wire was unauthenticated. It now composes the shared nornir-service bearer interceptor + serve loop (L5 — auth is not re-rolled):

• agent_grpc::serve_authed(addr, agent, token) — serves the AgentWorker behind nornir_service::auth_interceptor; empty token = open (dev), non-empty = every RPC must carry authorization: Bearer <token> or is rejected UNAUTHENTICATED before it runs work. serve() = serve_authed with an empty token.
• GrpcTransport::connect_with_token(endpoint, token) — sends the bearer on every RPC; a missing/wrong token reads the agent Offline (never scheduled) and any dispatch is a visible Failed, never a silent stall.
• agent_grpc::run_worker(bind, spec, token, runner) — the reusable <host> agent --group <g> entrypoint primitive. Per L5a the leaf ships it; the conductor (nornir/dwarves) wires the CLI flags and builds GrpcAgents from Roster::live(now) endpoints, picking the runner (its payload → job mapping).
• RED: wire_auth_rejects_unauthenticated_dispatch_and_admits_the_authed_one.

Stage 5 — resource-aware placement + fleet-exclusive gate

ResourceReq { min_cores, min_mem_mb, tags } is a WorkUnit's placement floor — WorkUnit::with_resources(r); match_agent/can_run only route to a node that clears it.

• ResourceReq::heavy() floors ≥ 8 cores (+ 16 GiB), so a HEAVY bench places only on Loki (12) / Odin (32), NEVER the 6-core NUC — the core count is the guard (the NUC also has 64 GiB, so memory can't discriminate). This is the "serial HEAVY on a quiet, capable box" law expressed as placement data.
• Fleet-exclusive gate: Scheduler gains a per-group single-permit semaphore. WorkUnit::exclusive() + Scheduler::dispatch_scheduled(w).await acquires the group's permit before dispatch and holds it (via ScheduledWorkHandle) until the handle drops — so at most one exclusive unit per group runs at a time across the fleet, even with several idle agents free. This is the scheduler-level generalization of jobs' process-global gate: a fleet-wide BENCH_RUN serializes across Loki/Odin, not just within one control-plane process. Non-exclusive units are never gated (delegate straight to dispatch). ResourceReq/exclusive are control-plane placement metadata — they do not cross the wire.
• RED: heavy_resource_req_never_places_on_a_nuc_class_node and exclusive_group_units_do_not_run_concurrently_across_the_fleet.

Deferred remainder (documented)

• Richer status on the wire. The agent-grpc StatusReply still returns only the bare WorkStatus; folding exit_code + result_ref into it (so WorkResult is assembled fully server-side from a real process exit, not just the poll status) needs the LocalRunner contract to return a result struct rather than bool — a larger change than a pure additive fold. Today the control-plane assembles the WorkResult around the terminal poll status + the node it dispatched to.
• Arrow artifact shuttle in jera. The flight transport is reused by reference (result_ref = flight ticket); a jera-side ArtifactShuttle wrapping nornir_flight::FlightConn was deliberately not added — it would pull Arrow into a leaf whose whole point is a lean dep tree. The control-plane (which already links nornir-flight) owns the byte transfer.