.nornir/scalable-bench-design.md

Design: scalable benches in the nornir bench API

Status: proposal (for the nornir team to implement). Authored from the skade-katalog consumer side, which has the first scalable bench (tpch_warehouse).

Goal

Let a bencher declare its own scalable dimension (e.g. TPC-H scale factor, row count, thread count) and let the nornir runner sweep it — running the bench at several scale points and recording one result per point — so docs can plot a scaling curve instead of a single number. The bench owns the meaning of "scale" (it knows its workload); nornir owns the driving (which points, tagging, persistence, rendering).

Key principle the consumer asked for: "the bench impl adds a scale function since it knows its own test; the nornir runner calls that to get the start values."

Non-goals

No change to how single-point benches work (must stay 100% backward compatible — the ~30 existing benchers implement only id() + run() and keep working).
nornir does not interpret scale numbers; it only passes them through and tags results. Units/semantics live in the bench.

API (additive to `nornir::bench::api::Bencher`)

/// A scalable dimension a bench can sweep. The bench owns the meaning.
#[derive(Clone, Debug)]
pub struct ScaleAxis {
    /// Short axis key, used to tag results and in config/CLI. e.g. "sf".
    pub name: &'static str,
    /// Human unit for docs. e.g. "TPC-H scale factor", "rows", "threads".
    pub unit: &'static str,
    /// The series the bench *recommends* if the runner/config gives none —
    /// the "start values". e.g. vec![0.01, 0.1, 0.5].
    pub default_points: Vec<f64>,
    /// Hint for series generation / plotting (log vs linear x-axis).
    pub log_scale: bool,
}

pub trait Bencher: Send + Sync {
    fn id(&self) -> &'static str;

    /// Single-point entry (unchanged; still the required method).
    fn run(&self) -> Result<BenchResult>;

    /// Declares a scalable dimension. `None` (default) ⇒ single-point bench:
    /// the runner calls `run()` exactly once, today's behavior.
    fn scale_axis(&self) -> Option<ScaleAxis> { None }

    /// Run at one scale point. Default ignores the scale and calls `run()`,
    /// so a bench that overrides only `scale_axis()` still works; scalable
    /// benches override this. No recursion: `run()` stays the required method.
    fn run_at(&self, scale: f64) -> Result<BenchResult> {
        let _ = scale;
        self.run()
    }
}

Backward compatibility: existing benchers implement run() only → scale_axis() returns None → runner behaves exactly as today.

Runner logic (`run_main_json` / nornir bench run)

for b in benchers:
    match b.scale_axis():
        None        => push(b.run())                         # unchanged
        Some(axis)  =>
            points = config_override(b.id()).unwrap_or(axis.default_points)
            for p in points:
                let mut r = b.run_at(p)?
                r.metrics.insert(axis.name, p)                # e.g. "sf" = 0.5
                r.name = format!("{}@{}{}", r.name, axis.name, fmt(p))  # tpch_warehouse@sf0.5
                push(r)

One BenchResult per scale point (the existing BenchRun.results: Vec<...> holds the series — no struct change needed beyond adding the axis metric).
Tagging: both a metrics[axis.name] = p entry (machine-readable) and a name suffix @<axis><p> (so existing name-substring docs filters still group the series). Pick one canonical form; suffix is friendliest to current docs filters.
A scale-aware bench failing at point p records a failure for name@<p> only; other points still run (same per-result failure handling as today).

Config / CLI overrides (the "start values" come from here or the axis default)

nornir.toml:

[repo.<name>.bench.scales]
# override the bench's default_points; key = bencher id
tpch_warehouse = [0.1, 0.5, 1.0]
tpch_q01       = [0.01, 0.1, 0.5]

CLI:

nornir bench run <repo> --scale tpch_warehouse=0.1,0.5,1.0

Resolution order per bench: CLI --scale > [bench.scales] in nornir.toml > ScaleAxis::default_points.

Docs rendering (nornir docs)

When a benches region's results share a base name and carry the axis metric, render a scaling table (one row per scale point) or a small curve, instead of a single row:

| sf  | build_s | query_s | total_s | ingest_rows_per_sec |
|-----|---------|---------|---------|---------------------|
| 0.1 |   ...   |   ...   |   ...   |        ...          |
| 0.5 |   ...   |   ...   |   ...   |        ...          |
| 1.0 |   ...   |   ...   |   ...   |        ...          |

The  marker already groups by name-substring; it just needs to detect the @<axis> series and pivot on it.

Gates / history

bench_runs persistence: unchanged — each scale point is a row, with the axis value as a column (from metrics), so regression gates can compare like-for-like scale points across runs (compare tpch_warehouse@sf0.5 to its prior @sf0.5).
required_results in nornir.toml may reference a specific point (tpch_warehouse@sf0.5) if a gate should require it.

Reference consumer implementation (skade-katalog, once the API lands)

impl Bencher for TpchWarehouse {
    fn id(&self) -> &'static str { "tpch_warehouse" }
    fn scale_axis(&self) -> Option<ScaleAxis> {
        Some(ScaleAxis { name: "sf", unit: "TPC-H scale factor",
                         default_points: vec![0.1, 0.5, 1.0], log_scale: false })
    }
    fn run_at(&self, sf: f64) -> Result<BenchResult> { /* build_ctx_scaled(sf, cores) + run 22 q */ }
    fn run(&self) -> Result<BenchResult> { self.run_at(0.5) } // default single point
}

Open questions for the nornir team

Name-suffix vs metric-only tagging as the canonical series key (this doc recommends both, suffix canonical).
Should default_points be Vec<f64> (numeric, simple) or a richer ScalePoint { value: f64, label: String } for non-numeric axes? Numeric is enough for SF/rows/threads; revisit if a categorical axis appears.
Per-point timeout / budget so a runaway high-scale point can't wedge a sweep.