mikronova/modgunn

Engine A vs Engine B — HEAVY benchmark

Head-to-head of the two swappable advisory-matching strategies behind engine::AdvisoryEngine: Engine A (linear scan) vs Engine B (inverted index). Same verdicts (asserted equal every run); the question is cost. The rationale is in .nornir/engine-reasoning.md; the harness is benches/engine_ab.rs.

Method

• Harness: benches/engine_ab.rs (custom, harness = false — measures whole-fleet throughput at full parallelism, not per-call latency).
• The whole synthetic fleet is scanned through each engine on znippy-zoomies' no-barrier fork-join gatling (gatling_for_each, one worker per core — the SAME path the shipped AdvisoryEngine::fleet / warehouse scan run; no rayon), one shared engine instance. Best of 3 timed passes per engine.
• Reproducible SplitMix64 corpus (no rand dep): advisories are a mix of exact-version / OSV-range / all-versions rows, ~⅛ withdrawn, ~1/16 KEV, across 4 ecosystems; artifacts include deliberate misses + GPL/unknown-license + unsigned rows so license/provenance/KEV/EPSS escalation all fire.
• Equivalence is asserted inside the bench: the run aborts unless Engine A and Engine B produce the identical total finding count. HEAVY only — no smoke variant.
• Run:

export CARGO_TARGET_DIR=/mnt/work4t/cargo-target/modgunn-engine
cargo bench --features scan --bench engine_ab

• Regression guard (Loki / nornir bench run modgunn). This head-to-head is the deep cargo bench sweep. The nornir Bencher harness (bench/examples/nornir-bench.rs) additionally ships a lighter, historized modgunn.fleet_scan_indexed arm — the same Engine B fleet path over the shared fixture, with the one-shot index build + bucket count recorded and a baked-in B ≡ A finding-count assert — so Engine B's throughput is tracked run-to-run alongside Engine A's modgunn.fleet_scan, not only in the manual heavy bench.

Machine

Results

Point 1 — artifact-scaled (1 M-artifact fleet, mid-size db)

advisories=25 000 packages=3 000 artifacts=1 000 000 cores=12

Identical findings: 1 667 658 (A == B). B index buckets: 10 477. Winner: Engine B — 195×.

Point 1 · re-run on oden (32-core) — 2026-07-26

advisories=25 000 packages=3 000 artifacts=1 000 000 cores=32 · host oden, box uptime 12:19

Identical findings: 1 667 658 (A == B — the same count as the 12-core run on the same SplitMix64 corpus: a cross-machine large-scale equivalence confirmation). B index buckets: 10 477. Winner: Engine B — 142×. Absolute B throughput ~2.8× the 12-core box (faster CPU + more cores); the A/B ratio is lower (142× vs 195×) because Engine A's linear scan parallelizes across 32 cores better than the already-saturated index, narrowing the gap while both get faster.

Point 1 · re-run on t14s (16-core) — 2026-07-27

oden is powered off, so Point 1 was re-run on t14s (16 CPUs / 46 GB). A new host is a NEW row — never a merge into the oden or 12-core tables above, and no number here is restated from either.

advisories=25 000 packages=3 000 artifacts=1 000 000 cores=16 · host t14s, box uptime 0,58 before → 17,67 after (16 gatling workers on 16 cores; no foreign process present on either side)

Identical findings: 1 667 658 (A == B) — byte-identical to BOTH the 12-core and the oden runs on the same SplitMix64 corpus, a third-machine confirmation of the A/B equivalence assertion. B index buckets: 10 477. Winner: Engine B — 362.8×.

Reproducibility. A second clean-start run (uptime 0,68 → 18,63) gave A 66.602 s / B 0.205 s / 325.2× — A within 3.5 %, B within 15 %, which is the expected spread for a 0.2-second measurement. It is cited as corroboration only, not as the row, because a foreign build appeared on the box near its end. Engine B's sub-second scan is inherently the noise-sensitive number here; Engine A's ~65 s is the stable one.

Two earlier runs on this host were REJECTED, not published. Both picked up a foreign rustc build partway through (load on a 16-core box: 1,12 → 37,17 and 0,67 → 45,24) and disagreed with each other by 30 % on scan time — they measured contention, not the engine. They are recorded here as discarded so the published row above cannot be confused with them:

Note how badly contention distorts the ratio specifically: Engine A's 60-second linear scan barely notices a competing build, while Engine B's 0.2-second indexed scan is stretched 3×. A shared box makes the A/B speedup look 3–4× worse than it is — which is exactly why a contended run must never be published as a host row.

Point 2 — db-scaled (large advisory corpus, higher index memory)

advisories=200 000 packages=80 000 artifacts=300 000 cores=12

Identical findings: 359 906 (A == B). B index buckets: 148 915. Winner: Engine B — 3 214×.

Reading it

• Engine B wins decisively, and the margin grows with the advisory-db size — 195× at 25 k advisories, 3 214× at 200 k. That is the linear engine's Θ(artifacts × advisories) versus the index's Θ(artifacts × hits_per_package) made concrete: at Point 2 the linear scan drops to 1 728 verdicts/s while B holds 5.5 M/s.
• Engine B's build cost is real but tiny and one-shot: 5.3 ms at 25 k advisories, 139 ms at 200 k — amortised over the whole fleet, negligible next to the seconds/ minutes it saves. It is not repaid for a one-artifact gate scan, which is why the default stays A.
• Memory is modest and dominated by the corpus, not the index (213–291 MiB peak); the usize-handle inverted index adds little over the advisory rows themselves.
• Correctness is not a trade-off: every run asserts A ≡ B on total findings, so B's speed costs nothing in results.

Verdict

Use Engine A for the lean per-artifact gate (tiny db, one artifact — the index never pays off). Use Engine B for every fleet / warehouse scan over the real CVE-warehouse corpus: modgunn warehouse scan --engine b. Above ~10⁴ advisories it is one-to-three orders of magnitude faster for identical output.

Manifest ingest — HEAVY parse-throughput benchmark

Pure-Rust, zero-shell parse of each ecosystem's manifest/lockfile into the shared sbom::SbomComponent model — the ingest side of manifests.rs. The question is raw parse throughput (components/sec), not a match. Harness: benches/manifest_parse.rs.

Method

• Harness: benches/manifest_parse.rs (custom, harness = false). One large synthetic manifest per ecosystem (one representative format each), generated untimed, then parsed through the public parse_manifest dispatch; best of REPEATS passes.
• Single-threaded per format (parsing is per-file); reports components/sec, input size, and peak process RSS. Host-stamped plain numbers.
• Run:

export CARGO_TARGET_DIR=/mnt/work4t/cargo-target/modgunn
cargo bench --features manifests --bench manifest_parse
# tunables: MODGUNN_BENCH_DEPS (default 50 000), MODGUNN_BENCH_REPEATS (default 5)

Machine

host=loki cores=12 build profile=bench (optimized) target dir=/mnt/work4t (PCIe4)

Results

deps_per_manifest=50 000 repeats=5 cores=12

Total: 300 000 components across six ecosystems; peak RSS 88 MiB.

Reading it

• Hand-parsed text formats dominate (gem 14.0 M, pip 9.4 M, go 5.6 M comp/s) — line-oriented parsing with no allocation-heavy document tree.
• Structured formats pay for their tree: TOML (cargo, via the toml crate) is the floor at 759 k comp/s, XML (maven, roxmltree) 2.1 M, JSON (npm, serde_json) 1.8 M — all still fast enough that parse is never the fleet-ingest bottleneck (the advisory match is — see Engine A/B above).
• Memory is flat (88 MiB for 300 k components) — the shared SbomComponent model is four owned strings, no per-ecosystem bloat.