nordisk/nornir

H5 — Agent × model bake-off MATRIX (nornir bakeoff)

Design + guide + idiot-guide for the H5 epic: ONE standardized way to pit a task against a cross-product of agents × models, land the results in the warehouse, and watch which agent+model combo won in a viz Leaderboard pane that shows the agent × model grid. Mirrors the nornir test run→persist→view shape and the release_events / test_results warehouse-row pattern, and reuses the H1 MCP-exercising + C6 run→persist→live shape.

1. What it is

nornir bakeoff run --task "…" --agents a,b --models x,y runs ONE task across the agents × models cross-product, recording one agent_model_runs row per (task, agent, model) cell — its answer plus score, wall_ms, tokens/s, cost, mcp_tool_calls — into the warehouse under a fresh run_id, and prints the leaderboard. The agent is the caller/harness identity (claude, kilo, a local-LLM driver); the model is what it drove. The viz 🏆 Leaderboard tab reads the same table and renders the agent × model grid with the winning cell highlighted; CLI parity is total — the pane shows nothing nornir bakeoff leaderboard --matrix can't print.

A model-only bake-off is the 1×N matrix under a single local-LLM agent, so the pre-matrix usage (--models … with no --agents) still works unchanged.

nornir bakeoff demo                                          # seed a mock matrix
nornir bakeoff run --task "Fix the bug" --agents claude,kilo --models opus,mistral
nornir bakeoff run --task "…" --mock                         # offline canned cells
nornir bakeoff run --task "…" --models a,b,c                 # 1-agent (model-only)
nornir bakeoff leaderboard                                   # newest run, ranked
nornir bakeoff leaderboard --matrix                          # agent × model grid
nornir bakeoff leaderboard <run_id|model|agent> --json       # raw rows

The --prompt / --prompt-id flags are kept as visible aliases of --task / --task-id for back-compat.

nornir bakeoff run --task T --agents a,b --models x,y
└─ ModelCaller
   ├─[--mock]  MockCaller (canned ModelAnswer per (agent,model) cell) ─┐
   └─[default] OllamaCaller (POST {host}/api/generate) ────────────────┤
                                                                       ▼
   run_bakeoff_matrix: one AgentModelRunRow per (agent,model) cell (shared run_id + ts)
   └─▶ append_agent_model_runs → Iceberg agent_model_runs
      ├─▶ matrix_grid() ranks cells → MatrixGrid
      │   ├─▶ render_matrix / render_leaderboard (CLI)
      │   └─▶ 🏆 Leaderboard pane grid (state_json.matrix)
      └─▶ leaderboard() flat ranking
          └─▶ render_matrix / render_leaderboard (CLI)

2. The bake-off backend (src/warehouse/agent_model_runs.rs)

The only thing that touches an actual LLM / agent harness is the [ModelCaller] trait — now agent-aware (the matrix's first axis):

pub trait ModelCaller {
    fn call(&self, agent: &str, model: &str, prompt: &str) -> Result<ModelAnswer>;
}

ModelAnswer carries the economics: output, latency_ms, tokens_in/out, tokens_per_s, score, cost_usd, mcp_tool_calls. Two impls ship:

• OllamaCaller — POSTs {model, prompt, stream:false} to {host}/api/generate (--host, else $OLLAMA_HOST, else http://localhost:11434) using the already-vendored ureq client. It ignores the agent axis (ollama has no agent concept) but the row still records which agent column the cell was placed in. Reads response, prompt_eval_count, eval_count, eval_duration (ns).
• MockCaller — a (agent, model) → ModelAnswer map (with_cell(agent, model, …), or with(model, …) for the AGENT_UNSET shorthand). The --mock flag (and every test) drives this so a bake-off needs no running LLM or Claude. A cell absent from the map returns an error, which the matrix runner records as a red ok=false row — exercising the failure path without aborting.

run_bakeoff_matrix(caller, run_id, ts, task_id, task, agents, models) is the pure cross-product→call→row pipeline both the CLI and the tests share. It does NOT touch the warehouse (the caller appends the returned rows), so a test runs a full matrix with a MockCaller and asserts the rows before any I/O. Row order is the (agent, model) cross-product — stable + testable. run_bakeoff(...) is the single-agent shim (agents = [AGENT_UNSET]). tokens_per_s is taken from the backend when supplied, else derived from tokens_out / (latency_ms/1000).

The default model set (default_models()) is the embedding-model registry's ids (src/vector/embed_registry.rs); the default agent set (default_agents()) is a single local-LLM agent. When the vector feature is off, the model default falls back to the same ids as a literal.

3. Warehouse table agent_model_runs

Schema: iceberg_schema::agent_model_runs (field ids 1–16). One row per (run_id, agent, model, prompt_id) cell:

Fields 14–16 (agent, cost_usd, mcp_tool_calls) are OPTIONAL so the schema is migration-safe: rows written before the matrix carry none, and the reader defaults them back to agent = "-" (AGENT_UNSET), cost_usd = 0.0, mcp_tool_calls = 0. The serde struct also #[serde(default)]s them, so a thin client decoding a pre-matrix server's JSON still parses.

Partitioned by run_id: a bake-off writes every cell in one append (one batch = one partition), and the leaderboard reads a whole run back, so run_id prunes scans. Per-model / per-agent history reads do a residual filter.

Writer/reader (src/warehouse/agent_model_runs.rs, mirrors test_results.rs):

• append_agent_model_runs(wh, rows) — one Iceberg snapshot per run.
• query_agent_model_runs(wh, sel) — sel ∈ Run | Model | Agent | All, sorted by (ts_micros, run_id, agent, model, prompt_id) (Iceberg has no scan order).
• leaderboard(rows) — flat ranking, best first: ok cells outrank failed ones; within those, higher score, then higher tokens_per_s, then lower latency_ms, then lower cost_usd, then (agent, model) cell name (a stable deterministic order the CLI + viz agree on, so a test asserts the exact rank).
• matrix_grid(rows) → MatrixGrid { agents, models, cells } — the agent × model grid. Each (agent, model) coordinate aggregates its task rows (mean score/tok/s/latency/cost, sum of mcp calls, AND of ok) into a MatrixCell with its rank (same comparator as leaderboard, so the grid winner = the leaderboard winner). grid.winner() is the rank-1 cell.
• demo_matrix_rows(run_id, ts) — a realistic 3-agent × 3-model mock matrix (one failed local-llm/opus cell) for nornir bakeoff demo.
• rows_to_json / render_leaderboard / render_matrix — the JSON + human renderers (flat ranked table + the grid).

4. The viz Leaderboard pane (src/viz/leaderboard_tab.rs)

The 🏆 Leaderboard tab reads agent_model_runs read-only + lock-tolerant locally, or over the Viz.BakeoffResults gRPC remotely (decoding into the same AgentModelRunRow, so the grid renders byte-identically thin):

• Run list (left) — one row per bake-off run_id, newest on top, with the task handle, agents × cells count, and the winning agent/model cell.
• Matrix grid (right, top) — an egui::Grid with agents down the rows, models across the columns; each cell shows score #rank, the winner cell in gold (🥇), failed cells in red (✗), empty coordinates ·. Hover = full economics.
• Ranked cells (right, below) — the flat leaderboard() ranking, each row score · tok/s · latency · cost · mcp with an ok/✗ chip.

LAW #6 — the pane exposes exactly what it renders as data via state_json()["leaderboard"]: the run summaries, the flat leaderboard (with agent/model/cell/cost_usd/mcp_tool_calls), and the matrix grid (agents, models, cells with per-cell rank, winner) — folded into the app's NORNIR_VIZ_STATE dump and the $NORNIR_VIZ_TRACE stream.

5. Tests (LAW #1 — inject real values, assert real output)

• warehouse::agent_model_runs::tests

  • run_bakeoff_builds_rows_and_records_failures — single-agent path: two models + one missing; asserts kept/derived tok/s + the red error row.
  • leaderboard_ranks_ok_then_score_then_throughput — exact score→throughput order, failed cell last.
  • matrix_crosses_agents_and_models_and_records_a_failed_cell — a 2×2 matrix: asserts the full cross-product, the per-cell agent/cost_usd/mcp values, and that the absent local/opus cell is the red one.
  • matrix_grid_ranks_cells_and_picks_winner — a 3×3 matrix: asserts the sorted axes, that claude/opus (highest score) is the rank-1 winner, the cell ranks, and that the grid winner == the flat leaderboard winner == render_matrix's.
  • demo_matrix_is_populated_and_round_trips — appends demo_matrix_rows to a tempdir warehouse, reads it back, asserts 9 cells (1 red), the winner, that cost/mcp survived, and that the Agent selector filters by agent.
  • legacy_single_axis_rows_default_agent_unset — a model-only run lands under AGENT_UNSET with zero cost/mcp.
  • append_query_round_trip_groups_and_ranks — full mock bake-off round-trip.

• tests/viz_leaderboard_pane.rs::leaderboard_pane_ranks_matrix_by_score_then_throughput — injects a 4-cell agent × model matrix, selects the Leaderboard tab, asserts state_json()["leaderboard"] exposes the winning cell, the ranked board (with cost + mcp), AND the matrix grid (sorted axes + per-cell rank + winner) with the failed cell last.

• src/viz/remote.rs::tests::bakeoff_results_wire_roundtrip — asserts the matrix axis + economics survive the Viz.BakeoffResults JSON wire, AND that a pre-matrix server's JSON (no agent/cost/mcp) decodes to the AGENT_UNSET defaults.

None require a running LLM or Claude — the backend is mockable by construction.

6. CLI parity