nordisk/nornir

Mímir ingest on t14s — the no-GPU host row, and three bugs the run exposed

A t14s row. It is NOT merged with oden's. oden's number (workspace_nordisk/.nornir/mimir-ingest-measurement-2026-07-26.md, branch enforce-valkey9) is the authoritative ingest measurement and it already answers the question: ingest is GPU-embed-bound — embed = 165.1 s of a 170.5 s populate (96.9 % of wall) at cores_busy = 1.12/32. This document does not restate that, contradict it, or merge with it. It records what the same populate costs on a host with no GPU at all, and the three defects that running it surfaced.

Host

Bencher law: uptime was taken on both sides of every run and is quoted per-run below. The absolute embed number below is not publishable as a clean figure — the box carried another agent's cargo build for most of the window and the 1-min load moved from ~5 to ~33. It is quoted only as a lower bound, which is all the conclusion needs.

Per-phase wall (full-from-empty, 17 members)

Every non-embed phase together: ≈ 9 s. The embed is therefore >400× everything else combined, and on this host it does not finish at all inside an hour.

The one conclusion this host adds

oden says the embed is 165 s because it has a 4090. t14s says: take the GPU away and the same 17-member workspace does not complete its embed in an hour. The GPU is not an optimisation of Mímir ingest — it is a precondition. Any ecosystem-scale (crates.io) plan that assumes CPU embedding is dead on arrival, and any "populate on the developer's laptop" story is a >1 h story today.

Not measurable here, so not claimed: batch-size tuning, a second device, or a smaller model — all three are oden-side experiments. t14s has no GPU, so no number for them is invented in this document.

What the run exposed

1. The code index is EMPTY on every FAT workspace, and reported green — including oden

index::walker::walk_workspace runs follow_links(false). The documented FAT workflow (07-26 verification, "Server populate unblocked") symlinks each member into <ws>/git/<member>. So the tantivy walk skips all 17 members, indexes zero files, and the phase prints ⏱ index-build in 0.0s — indistinguishable from "nothing to do". Index::report_build_status then scored it green, because its predicate short-circuited on stats.scanned == 0. nornir search returns nothing, forever, with no diagnostic anywhere.

oden's own ingest table shows the same index-build (map) 0.0 s row. This was not a t14s artefact — it is the live server, and nobody caught it because the phase was silent.

Fixed. The phase now prints its counts, and a 0-file walk over present members prints the cause and records an index-build ERROR row in workspace_events so the populate roll-up cannot come up green over an empty index. Live output:

nornir-monitor: nordisk ⏱ index-build 0 file(s) (0 added, 0 updated, 0 unchanged, 0 errors) in 0.0s
nornir-monitor: nordisk ⚠ CODE INDEX EMPTY — index-build walked 0 files over 17 present
  member(s): 17 member(s) under …/nordisk/git are SYMLINKS and the workspace walker does not
  follow links (Skidbladnir, draupnir, dwarves, edda, facett, funnel-core, holger, knut,
  korp, korp-collectors, modgunn, nornir, ordning, skade, tunnr, znippy, znippy-zoomies)
  — materialize them as real checkouts. `nornir search` will return NOTHING.

Now fixed (eb65188). The walker itself was the last gap and it is closed: index::walker::walk_workspace now runs follow_links(true), so a symlinked FAT member descends like a real checkout and the code-index populates. It is loop-safe (walkdir detects symlink cycles → Err, dropped by flatten; broken/dangling symlinks likewise), and HARD_EXCLUDES + repo_scope still prune the followed content by name/depth. Regression walk_follows_symlinked_members_so_fat_workspaces_index proves it (RED got [] on follow_links(false), GREEN on the fix). The monitor's empty-index diagnostic no longer claims "the walker does not follow links" (now false) — a zero-scan over symlinked members is re-attributed to BROKEN (dangling) symlink targets. The Search tab's state_json["search"] also gained index_health (not_run|empty|populated|error)

• index_populate_hint (LAW 2 + LAW 6), so an empty code-index reads as a flagged "empty" naming the cause rather than a silent green.

2. The embed was all-or-nothing and unobservable

One embedder.embed(&all_texts) over every missing chunk of every member, then one append. On oden that window is 165 s and nearly harmless. On t14s it is an hour in which the process holds every input text and every output vector resident (7.3 GB RSS, growing), writes nothing, and prints nothing — an operator cannot tell working from hung. Both times the run was interrupted, the entire hour was lost.

Fixed by slicing the embed at EMBED_SLICE_CHUNKS = 2048: bounded peak memory, one append per slice (so a crash resumes from what landed — vectors are content-addressed), and a done/total · chunk/s · eta line per slice.

This is not a speedup and is not claimed as one. It changes throughput by nothing; ingest is device-bound. It is a durability + observability fix. Its cost is ≈20 extra Parquet appends per workspace populate, which warehouse compact already handles.

3. MCP served Mímir's knowledge graph COLD on every call

46500fc added the snapshot-keyed VIEW_CACHE and switched the gRPC Knowledge RPCs to load_preferred_cached. nornir-mcp's knowledge_query_json — the single chokepoint for five MCP tools (callers_of, callees_of, call_path, symbol_lookup, defined_in) — was left on the uncached load_preferred. So the surface the loop brief says to drive Mímir through rebuilt the whole KnowledgeView from a cold Parquet scan, including the stree posting lists, on every single tool call, while the gRPC twin answered the identical query from RAM.

The "46500fc made search warm" claim was therefore half true: warm over gRPC, cold over MCP. Fixed — MCP now shares the same snapshot-keyed cache.

Also shipped

• nornir warehouse reset [--root <dir>] [--yes] — the loop's missing empty step. Removes both catalog.redb and warehouse/, because removing only the data dir leaves a catalog that still names deleted files and the next open() fails. Dry-run by default; refuses while the catalog is locked; reports "already empty" rather than a green tick over a no-op.
• examples/vector_timetravel.rs:189 — the known embed(&[query.to_string()]) type error (&[String] does not coerce to &[&str]). Fixed; --all-targets no longer fails on it.

Revert-checks (both actually reverted, both actually went red)

Corrected loop guidance

The loop brief's step 1c — "fix the serial half, the embed/map pass, fan out with gatling" — is wrong and should be struck. There is no caller-side parallelism left to win: the syn-scan is already gatling-parallel and costs ~1–4 s, the embed is already batched, and on oden the whole 32-core box idles at cores_busy 1.12/32 while a single GPU does the work. The number to move is the embed device, not the core count.