nordisk/nornir

Generator backends — the generative-LLM abstraction (EPIC #39)

One Generator trait, three real implementations (candle, mistralrs, onnx) behind a factory, plus a test mock and an off-by-default ollama client. This is how nornir generates text locally without depending on ollama, and how the H5 bake-off (#46) and the funnel intake (#45) get a pluggable model call.

Source: src/warehouse/generator/ (mod.rs + candle.rs + mistralrs.rs + onnx.rs + ollama.rs).

The trait

pub trait Backend {
    fn id(&self) -> &str;        // spec id, e.g. "candle:qwen2-0.5b", "mock"
    fn available(&self) -> bool; // runtime probe of lib + model presence
}

pub trait Generator: Backend + Send + Sync {
    fn complete(&self, req: &GenRequest) -> Result<GenAnswer>;
}

• Backend is split out so a caller can list/probe a backend without driving generation.
• Generator: Send + Sync so the server / bake-off can share one across threads (each backend that holds mutable model state guards it behind a Mutex).

Request / answer

GenRequest { prompt, system, max_tokens, temperature, stop } — the inputs every backend needs; a backend ignores knobs it can't honor rather than erroring. Builder helpers: GenRequest::new(prompt).with_system(..).with_max_tokens(..) .with_temperature(..).with_stop(..).

GenAnswer { text, tokens_in, tokens_out, tokens_per_s, latency_ms } — mirrors the exact fields the bake-off's ModelAnswer needs. GenAnswer::into_model_answer() slots it straight into an agent_model_runs row (local generation is free → cost_usd = 0.0, mcp_tool_calls = 0, score = 0.0 left for a judge).

The factory

pub fn generator(spec: &str) -> Result<Box<dyn Generator>>

spec is <backend>:<model> (or bare mock):

Each real arm is #[cfg(feature = …)]-gated:

• feature off → Err("backend X is not compiled in — rebuild with --features gen-X"). A clear instruction, never a silent stub.
• unknown backend → Err("unknown generator backend X …").
• feature on → constructs the backend (cheap — heavy model load is lazy on first complete). It may still report available() == false if the model/lib isn't present yet.

available() runtime-probes so a missing backend degrades gracefully:

• candle: GGUF + tokenizer already in the HF/candle cache (offline check).
• mistralrs: the in-process engine's tokio runtime can start (model fetched lazily).
• onnx: the ORT shared lib is loadable and model.onnx + tokenizer.json exist in the model dir.
• ollama: the daemon answers /api/tags.
• mock: always true.

CLI / env (CLI-parity LAW)

nornir bakeoff run --task "…" --models a,b --gen candle:qwen2-0.5b
NORNIR_GEN_BACKEND=mistralrs:Qwen/Qwen2.5-0.5B-Instruct nornir bakeoff run …

Priority: --gen <spec> > $NORNIR_GEN_BACKEND (spec_from_env) > --mock > prior-art ollama caller. When a --gen spec is given the CLI prints the resolved backend id + its available() state, then drives the SAME selected model across every matrix cell via BoxGeneratorCaller (a ModelCaller adapter) through the unchanged run_bakeoff_matrix pipeline.

The three backends (real call paths)

candle (gen-candle) — pure Rust, no C

candle-core + candle-transformers. Loads a quantized GGUF Qwen2-family model (quantized_qwen2::ModelWeights::from_gguf) + its tokenizer.json (fetched once from the HF hub via hf-hub). complete runs a real loop: encode → prefill forward → LogitsProcessor::sample → autoregressive decode until the token cap / EOS (<|im_end|>) / a stop sequence. Presets: qwen2-0.5b (default) and qwen2-1.5b.

mistralrs (gen-mistralrs) — candle-based, in-process

The mistralrs crate. TextModelBuilder::new(model_id).with_isq(Q4K).build() builds the in-process engine (fetches weights itself); complete sends a RequestBuilder chat request (system + user turns, temperature + max-len) via model.send_chat_request, then reads choices[0].message.content + usage.{prompt,completion}_tokens. Async engine driven on a current-thread tokio runtime so the trait stays sync.

onnx (gen-onnx) — ORT (onnxruntime-genai-style)

The ort stack + tokenizers. Points at a model dir holding model.onnx + tokenizer.json (a HF Optimum/genai export; $NORNIR_GEN_ONNX_DIR overrides an empty spec). complete builds an ORT Session and runs the genai decode loop: tokenize → session.run(input_ids, attention_mask) → take the last-position logits → greedy argmax → append → repeat until cap / EOS / stop. Links onnxruntime (C++) — the same accepted exception as embed-ort.

ollama (gen-ollama) — NOT one of the three

Off by default. The prior-art {host}/api/generate HTTP client adapted to the trait, kept as a convenience (--gen ollama:<tag>). Pulls no new dependency (reuses the vendored ureq). The original OllamaCaller / ModelCaller / MockCaller in agent_model_runs.rs are untouched so bakeoff run without --gen behaves exactly as before.

How it feeds the rest

• Bake-off #46 (judge): GenAnswer → ModelAnswer → AgentModelRunRow. The codegen judge then scores each cell's output (score_rows_with_judge), so a real local generation gets a real compile-and-test quality score instead of being ranked by speed alone.
• Funnel #45 (intake): the planner's ModelCaller can be a GeneratorCaller wrapping any backend, so idea→component matching runs on a local model.

Tests (inject-assert LAW)

• mock_round_trips_request_into_answer — feeds a real GenRequest (prompt + system + max_tokens) and asserts the GenAnswer carries them back with derived, non-zero token economics.
• factory_errors_on_unknown_backend / factory_reports_uncompiled_backends — the factory returns the right impl, the "unknown backend" error, and the per-feature "not compiled" error (each spec gated by its own cfg!).
• generator_caller_bridges_to_modelcaller — the bake-off adapter produces a free, no-MCP ModelAnswer.
• spec_from_env_reads_the_var — $NORNIR_GEN_BACKEND resolution (blank = unset).
• Per backend (under its feature): constructs + reports available() without a model load; an #[ignore]d heavy test does a real model fetch + generate.