dwarves — agent-found problems
Real problems found while working this repo that were not actioned in the change that found them, plus the measurements that rule other explanations out. Sister file to the constellation's other <repo>_agents_problems.md.
Reporting shape follows the workspace LAW: col-1 a 1–3 word unique subject, col-2 status (done · new · as dåligt · as bra), col-3 the technical detail.
2026-08-18 — dwarves-server OOM-killed the 62 GB dwarf box
Eleven kills over 2026-08-16/18, the last four at 22:43 · 23:23 · 00:20 · 01:15, each at anon-rss 49–53 GB / total-vm 61–73 GB / pgtables 106–127 MB, followed by 152 × systemd-journald: Under memory pressure, flushing caches and the host dying at 02:17:31 without completing a journal write.
| subject | status | detail |
| mimalloc purge | done | Term 1 of the OOM, and the one this repo owns. mimalloc = "0.1" compiles the v3 C sources (libmimalloc-sys/build.rs picks v3 unless the v2 cargo feature is set; nothing in the constellation sets it), and v3 raised the purge_delay default from v2's 10 ms to 1000 ms. At 1000 ms freed memory is never handed back: MEASURED on dwarf, a 2 GB heap of ~1 KB Strings dropped three times over stays at 2006 MB RSS forever; MIMALLOC_PURGE_DELAY=10 → 156 MB, =0 → 7 MB. RSS therefore stopped meaning "in use" and became the running MAXIMUM of every transient the process ever built. Fixed in crates/dwarves/src/alloc.rs, called from server::serve_all; guarded by crates/dwarves/tests/mimalloc_transient_peak.rs. On a REAL embedded-nornir republish (deep_scan = true over dwarves' own Cargo.lock closure — 357 dep crates, 13 344 files, 12 023 symbol scans) the tuning cuts PEAK RSS 12–20 % — 3.08 → 2.71 GB and 3.38 → 2.72 GB across two independent pairs — which is the number that matters, because production died AT its peak (anon-rss == systemd's memory peak at all eleven kills). The settled floor moved only 2.53 → 2.46 GB: see the next row for why. |
| deepscan batch | new | nornir/src/deepscan.rs:409 const BATCH: usize = 1_000_000;, used at :424 as present.chunks(BATCH). The comment above it says "Per batch of crates (bounds peak memory)" — with 2 265 crates in the nordisk closure there is exactly one batch, so it bounds nothing. Three full copies of the corpus are live at the flush: CollectSink.files: Vec<CrateFile> (:310, 66 478 files decompressed into content: String), WarehouseSink.pending: Vec<SymbolScan> (:326, the 60 467 scans the log prints), and the Arrow arrays ingest_symbol_scans_async clones every string into (nornir/src/warehouse/iceberg.rs:3432, accumulated in tables at :3451 before one ingest_multi at :3717) — plus sort_groups (:1138) making a sorted copy while the original is alive. This is the transient the allocator fix now returns; shrinking it is still worth doing. A real batch size (crates per flush, sized off available memory) belongs here. |
| working-set cache | new | Term 2 of the OOM, and it is NOT reachable from this repo. MEASURED: after one heavy republish the settled floor is ~2.5 GB and is insensitive to the allocator setting (2.53 GB as-shipped vs 2.46 GB tuned) — so that floor is LIVE memory, and at production scale it is very likely the larger of the two terms. nornir/src/warehouse/working_set.rs:88 PREVIEW_CACHE and :90 ARROW_CACHE are process-global Mutex<Vec<…>> statics warmed at the END of every republish (nornir/src/monitor.rs:1557 → warm_hot_tables, working_set.rs:139). WORKING_SET_CAP = 64 (:49) bounds entry count, not bytes, and there are only 18 hot keys — so the cap never fires and all 18 decodes stay resident indefinitely. HOT_PREVIEW_LIMIT = 5_000_000 (:85) rows × 15 HOT_LENS_TABLES (:57) of TablePreview.rows: Vec<Vec<String>> (one heap String per cell) is tens of GB on its own, and arrow_cached → scan_arrow → load_and_read_all (nornir/src/warehouse/iceberg.rs:1529) has no limit at all while HOT_ARROW_TABLES (:80) includes embeddings — the entire 768-float vector table for the workspace, held in an Arc in a static. preview_cached (:107) also builds the new decode before evicting the old, so each warm peaks at 2×. Cap it on BYTES. nornir deliberately has no crate dep on dwarves (nornir/Cargo.toml:671-680 — that edge was removed so the forge could compile nornir IN), so nothing in this repo can bound it; it has to land in nornir. |
| cache key identity | new | nornir/src/warehouse/working_set.rs:97,107,119 — the cache keys are (table, snapshot_id, limit) with no warehouse/workspace identity. With several served workspaces they evict each other every sweep, so the same tables are re-decoded from scratch on every sweep of every workspace. |
| deepscan idempotency | new | nornir/src/monitor.rs:1133 calls deepscan::deep_scan(&dirs, wh) with no SHA/skip ledger, unlike the knowledge scan at :504-590 which prints knowledge unchanged, skip scan. So the whole dep closure is re-appended under a fresh snapshot id on every republish and symbol_facts/call_edges/feature_gate_facts/test_inventory grow monotonically on disk — which is also what feeds the preview cache above. |
| embed pre-read | new | nornir/src/monitor.rs:1806-1828 builds inputs: Vec<RepoInput> over all present members in one gatling pool, each holding full file contents (nornir/src/vector/store.rs:505). The doc comment at :1786-1789 claims "each member's sources are DROPPED after its commit, so peak resident text no longer grows with member count" — the into_iter() drain at :1840 is correct, but the peak already happened at :1806, before the first member is embedded. The doc and the code disagree; the code is what runs. |
| scip result vector | new | nornir/src/monitor.rs:847-856 — the SCIP fan-out is bounded (scip_fanout, clamp(cores/8, 2..=8)) but resolved: Vec<Result<ScipScan>> is not: every resolved member's full occurrence row set (ScipRow, 6 String fields) is held until the serial pass drains it at :861. Inside a worker, ingest_index_file (nornir/src/knowledge/scip.rs:171) holds the raw index.scip bytes and the parsed protobuf Index and the emitted rows simultaneously. |
| republish locals | new | nornir/src/monitor.rs:593 symbol_scans and :607 git_heats are locals of republish_with that are never scoped or dropped, so every rescanned member's symbol tables stay resident across the SCIP phase, the index build, the (measured 1021 s) embed, deep-scan and arch-generate — they are only needed by the append loop at :626-647. |
| arch phase | new | nornir/src/monitor.rs:1243 arch_computed holds every member's full ArchGraph and its rendered SVG String at once; :1267 all_access accumulates every member's fn → table edges for the whole workspace before one append at :1310. |
| unit blast radius | new | The unit dwarves-server --install renders declares no MemoryAccounting=/MemoryHigh=/MemoryMax= and no OOMPolicy=, so a runaway in the fat process is a global OOM: the kernel picked victims box-wide (tailscaled and tmux: server both invoked the OOM killer, journald flushed caches 152 times) and the host died rather than the unit. Restart=on-failure with no StartLimitBurst then re-launched it straight back into the same climb — restart counter reached 11. NOT actioned because skidbladnir::service::ServiceSpec (Skidbladnir/src/service.rs:350) has no field for cgroup resource limits; adding one is a skidbladnir change and every nordisk service wants it, so it should land there once rather than as a dwarves-local drop-in. |
| health full scan | new | crates/dwarves/src/server.rs:262-265 — Dwarf.Health calls store::read_runs(&self.warehouse) and uses it only for .len(), i.e. it parses the entire run history off disk on every liveness poll. recent_runs (:245-256) has the same shape: read everything, row_to_pb everything, reverse(), then truncate(cap). Harmless today (the live warehouse has no brokkr/ at all) and it is a read, not a retention — but it is O(history) per poll for a UI that polls, and it will be the next thing to hurt once pipelines actually run. |
| action cache growth | new | crates/dwarves/src/cache.rs:128,149,168 — the action cache is an append-only JSONL: insert appends a line even when the input_id already exists (later lines win on reload), and nothing ever compacts it. In-memory size is bounded by DISTINCT input ids, so this is not the OOM, but the file grows without limit and ActionCache::load — called once per dispatched run — parses all of it. |
| proxy dial churn | new | crates/dwarves/src/warehouse_proxy.rs, the proxy_impl! macro: every proxied RPC does self.0.channel().await?, i.e. a fresh Channel::connect (new TCP + HTTP/2 handshake) to the embedded nornir per call, rather than one channel cloned per handler. MEASURED not to leak — 4 500 proxied Warehouse.Tables calls left RSS plateaued at ~80 MB and the fd count constant at 13 — so this is a latency/TIME_WAIT problem, not a memory one, and it was left alone as out of scope. |
| cron store rows | new | crates/dwarves/src/server_cron.rs:78-89 — CronStore::save inserts and never deletes, so a pipeline that is de-registered keeps its cron_last row forever. Bounded by the number of pipelines ever registered; noted for completeness, not a risk. |
| assemble cell red | new | crates/dwarves/tests/assemble_container_cell.rs does not compile on clean main (E0063: missing field 'params' in initializer of 'BuildArgs', twice) — verified by stashing all local work and building the target at cf72fdf, and still red at fb5a752. So a bare cargo test -p dwarves cannot even finish linking on main today. PRE-EXISTING and unrelated to the allocator work; not touched, because "fixing" a red I did not cause by editing someone's in-flight surface is how a real failure gets papered over. |
| silenced funnel | done elsewhere | crates/dwarves/tests/silenced_tests_guard.rs was RED at cf72fdf: funnel_persist_live.rs carries #![cfg(feature = "funnel-action")], which is not in the default closure and had no declared arm, so 2 test fns compiled to an empty binary and printed running 0 tests ... ok. Verified pre-existing by moving the new allocator cell off disk (identical single finding over 46 targets instead of 47). Fixed upstream while this work was in flight — cc8528a ("testmatrix arms: release_registry_robot + funnel_persist_live rescued") declares both arms, and the guard is green again on fb5a752. Recorded because the finding was real and the record of who fixed it is worth having. |
| variant roster | new | crates/dwarves/tests/variant_drive.rs:911 the_drive_roster_is_declared_not_invented is RED on clean main (both at cf72fdf and at fb5a752): .nornir/build-variants.toml declares SIX atoms (dwarves-server, dwarves, dwarves-cli, dwarves-ui, dwarves-ui-gpu, dwarves-solo) and the drive roster covers five. PRE-EXISTING and byte-identical with the new cell removed. The guard is right — an uncovered atom is a silent cap — and the missing entry is a declaration only its author can make honestly. |
| floors unresolvable | new | main at fb5a752 does not resolve at all in the documented side-by-side checkout layout, so no cargo command runs on it: 46b99a5 ("floors raised to the published API") set facett-pipeline = "0.1.16" (crates/dwarves/Cargo.toml:635) and nornir-service = "0.1.7" (crates/dwarves/Cargo.toml:700, crates/nornir-funnel-embed/Cargo.toml:54, crates/nornir-warehouse-embed/Cargo.toml:45), while the sibling checkouts are at facett 0.1.15 and edda's nornir-service 0.1.6. A path dep whose sibling is BELOW the declared floor is a resolution error, not a warning: error: failed to select a version for the requirement 'facett-pipeline = "^0.1.16"'. PRE-EXISTING and reproduced on fb5a752 with zero local changes. NOT touched: those floors were raised deliberately to match crates.io, so lowering them would revert someone's decision, and the honest fix is to bump the siblings. The allocator change was verified twice around it — fully green at cf72fdf (the tip when the work started) and again on top of fb5a752 with ONLY those three floors relaxed locally and never committed. |
| host disk full | new | Not a code bug, but it belongs next to this incident: / on dwarf is 100 % full (1.9 T, 12 MB free when hit on 2026-08-18). cargo target dirs across the constellation account for essentially all of it — nornir/target 273 G, holger/target 210 G, korp/target 174 G, knut/target 102 G, dwarves/target 100 G, and eight more. A release build of the fat server failed with No space left on device (os error 28) mid-link. Worth connecting to the incident: the last thing before the host died at 02:17:31 was journald flushing caches 152 times and then not completing a journal write — an out-of-space /var/log/journal produces exactly that, so the memory ratchet may not have been the only thing squeezing the box that night. Reclaimed only dwarves/target/debug/incremental (16 G, pure regenerable cache) to finish the verification; no other repo's target tree was touched — that is the operator's call, not an agent's. |
| constellation-wide | new | The same v3 purge_delay default applies to every nordisk binary that installs mimalloc: nornir/src/bin/nornir-server.rs:18 and nornir/src/bin/nornir.rs:11 both declare #[global_allocator] mimalloc::MiMalloc with no option tuning. A standalone nornir-server running the same monitor sweep will ratchet exactly the same way. The fix wants lifting into a shared leaf (edda) so one call site serves the whole constellation instead of being re-implemented per repo. |
What was ruled out, and how
The hypotheses that looked most likely up front were all eliminated by measurement, not by reading:
- sindri / the watch loop.
/home/dwarves/.dwarves/warehouse/contains exactly one file,cron.redb— there is nobrokkr/directory, sostore::append_historyhas never been called and not one pipeline run has ever executed. The in-memoryRegistry(crates/dwarves/src/server.rs:52,864) starts empty at every boot and is only ever populated by theRegisterPipelineRPC, soserver_cron::sweep_cronis skipped on every tick (server.rs:895,if !regs.is_empty()). The "watch loop" churn in recentgit log(a6e0892,e80629a) is indwarves-gui— the UI — not in the server. Also proves the--real/cargo buildand run-history hypotheses dead: no run means noRealRunner, noInMemBuildInfoSink, no history rows, no bench rows. - gRPC / grpc-web subscribers. 8 500 grpc-web
Dwarf.Healthcalls and 4 500 proxiednornir.v1.Warehouse/Tablescalls, each on a fresh connection, against a live fat server: RSS 68 → 80 MB and then flat, thread count constant at 14, open fds constant at 13. No per-connection retention. - Unbounded channels. There are none —
grep -rn "unbounded_channel\|unbounded("overcrates/returns nothing; the only channels arestd::sync::mpscshutdown signals in test/embedding helpers. - Overlapping sweeps.
nornir/src/server_grpc.rs:5932awaits itsspawn_blockinginside the tick loop and setsMissedTickBehavior::Skip, andctx.republish_lock(:6117) is a second belt, so a 20-minute sweep cannot stack behind a 60 s tick.nornir/src/monitor.rs:2415spawn_poll_loopis dead code — nothing wires it. - Captured child output. rust-analyzer's SCIP run pipes stderr only (
nornir/src/knowledge/scip.rs:265,.output()); the index itself goes to a temp file via--output(:250) and is read back at:171. The enormous payload never passes through a pipe. - What the reproduction does and does not show. The allocator term is demonstrated three ways and all three agree: a 2 GB standalone probe (0 % returned as shipped, 100 % returned at
purge_delay=10), the regression cell in-process (443 MB peak, 0 KiB returned as shipped vs ~300 MB returned tuned), and a real embedded-nornir deep-scan republish (peak 3.08 → 2.71 GB and 3.38 → 2.72 GB). What is NOT demonstrated is that the allocator term accounts for the whole 50 GB — the settled floor after a heavy sweep is the same either way, which is what points at theworking-set cacherow above as the larger term. Said plainly, because a fix that measurably lowers the ceiling is worth having and is still not the same thing as a fix for everything.