nordisk/nornir

Bake-off codegen judge (CodegenJudge / AnswerJudge)

Design + guide for the compile-and-test quality scorer that finally fills the bake-off's missing score. Companion to .nornir/bakeoff.md (the H5 agent × model matrix). Also doubles as the funnel #45 task-decomposition accept/reject oracle.

1. The problem it solves

A bake-off (nornir bakeoff run) records every (agent, model) cell's answer plus its economics — latency, tokens/s, cost, mcp calls — into agent_model_runs. But for a real ollama run the quality score was left 0.0 (OllamaCaller has no judge), so real leaderboards ranked cells by speed alone: the fastest gibberish beat the slow-but-correct answer. The mock/demo paths hand-author plausible scores, which never reflect whether the code works.

The codegen judge replaces that 0.0 with a score earned by actually building and testing the answer.

2. What it is

A pure-Rust evaluator — it does not call an LLM. It takes an answer that was already produced and grades it:

1. Copy the target crate into a temp scratch dir (skipping target/ + .git/ so it's fast + clean).
2. Apply the answer: either overwrite a file with it (the answer is a full file body), or git apply it (the answer is a unified diff/patch).
3. cargo build the scratch crate.
4. Run the acceptance command (default cargo test; configurable, e.g. a specific --test or a doctest run).
5. Turn the outcome into a 0.0..=1.0 score + a structured [Verdict].

Scoring

The fraction comes from summing every test result: ok. N passed; M failed; … line cargo prints (unit + integration + doc suites each emit one).

3. The API

// The trait — one method, never panics on a bad answer (a bad answer is a 0.0
// Verdict, not an Err; Err is reserved for the judge's own infra failing).
pub trait AnswerJudge {
    fn judge(&self, answer: &str) -> anyhow::Result<Verdict>;
}

pub struct Verdict {
    pub compiled: bool,
    pub tests_passed: u32,
    pub tests_total: u32,
    pub score: f64,        // 0.0..=1.0 — becomes the bake-off row's score
    pub message: String,   // one-line explanation
}
impl Verdict {
    pub fn rejected(msg) -> Verdict;   // a 0.0 "never built" verdict
    pub fn accepted(&self) -> bool;    // compiled && all tests passed  ← funnel oracle
}

// The first impl: compile-and-test.
pub struct CodegenJudge { /* holds a CodegenSpec */ }

pub struct CodegenSpec {
    pub crate_dir: PathBuf,     // crate copied into scratch
    pub apply: ApplyMode,       // WriteFile { rel_path } | ApplyPatch
    pub accept_cmd: Vec<String>,// default ["cargo","test"]
}
// constructors:
CodegenSpec::write_file(crate_dir, "src/lib.rs")
CodegenSpec::apply_patch(crate_dir)
    .with_accept_cmd(vec!["cargo".into(), "test".into(), "--test".into(), "it".into()])

AnswerJudge is the seam so the funnel epic reuses the same evaluator; CodegenJudge is the first impl, and more (lint judge, benchmark judge, …) can slot in without touching the bake-off.

4. How it wires into the bake-off

agent_model_runs::score_rows_with_judge(&mut rows, &judge):

• For every OK row, judge row.output and overwrite row.score with the verdict's. Returns one JudgeReport per judged cell (for the CLI to print).
• Failed rows (ok == false, the model never answered) stay 0.0 — there's nothing to compile.
• A judge infrastructure error (e.g. the crate can't be copied) is non-fatal: the row keeps its prior score, the error rides in the report. The bake-off never aborts because the judge tripped.
• Pure — no warehouse I/O; the caller persists the re-scored rows. Non-codegen bake-offs (no spec) skip it entirely and keep whatever score the caller set.

run_bakeoff_matrix → rows (score 0.0)
└─ --judge-crate set?
   ├─[no]──────────────────────────────────────────┐
   └─[yes]─▶ score_rows_with_judge                  │
            CodegenJudge.judge(output)              │
            └─▶ row.score = verdict.score ──────────┤
                                                    ▼
                                          persist as-is
                                          └─▶ leaderboard ranks by REAL score, then tok/s

5. CLI

New flags on nornir bakeoff run:

# Score each model's answer by building+testing it against a fixture crate.
nornir bakeoff run --task "Implement add(a,b)" \
  --agents local-llm --models llama3,mistral \
  --judge-crate ./fixtures/addcrate --judge-file src/lib.rs

# Patch-style answers, gated on one integration test.
nornir bakeoff run --task "Fix the race" --mock \
  --judge-crate ./fixtures/racey --judge-patch \
  --judge-cmd "cargo test --test concurrency"

The run prints each cell's verdict, e.g.:

⚖ judging 2 cell(s) by compile-and-test against `./fixtures/addcrate` (cargo test)
  local-llm/llama3: score 1.000 — compiled; all 2 acceptance tests passed
  local-llm/mistral: score 0.000 — did not compile: error[E0308]: mismatched types

…and the leaderboard then ranks by real score first, tok/s second.

6. Doubling as the funnel #45 oracle

The funnel decomposes an idea into sub-tasks; #45 needs to decide whether a sub-task's produced code is accepted or rejected. That is exactly AnswerJudge::judge(produced_code) → Verdict::accepted() (compiled AND all tests passed). The same CodegenSpec (crate + accept cmd) that scores a bake-off cell is the per-sub-task acceptance gate, so the funnel and the bake-off share one evaluator and one definition of "the code is correct".

7. Tests (inject-assert LAW)

src/warehouse/codegen_judge.rs tests feed real answers to a real scratch crate and assert real scores:

• known_good_answer_scores_one — correct add body → build+test pass → 1.0.
• noncompiling_answer_scores_zero — type-error body → won't compile → 0.0.
• compiling_but_failing_answer_scores_partial — add off by one → 1/2 tests → 0.5.
• patch_that_does_not_apply_scores_zero — bogus diff → 0.0.
• plus pure scoring-math + count-parsing unit tests.

src/warehouse/agent_model_runs.rs — matrix_scored_by_codegen_judge_lands_real_scores: a MockCaller matrix whose cells emit a GOOD body, a NON-COMPILING body, and one absent (red) cell; run through score_rows_with_judge against a scratch crate. Asserts the rows land real, non-0.0 scores (good=1.0, non-compiling=0.0, red stays 0.0/red) and the leaderboard now ranks the GOOD cell first on its earned score.