nordisk/nornir

Introspection warehouse DELTA — file-level incremental scan + inotify (LANDED)

The JetBrains-beating re-index lever, built in nornir (design audit: workspace_nordisk/.nornir/introspection-warehouse-efficiency.md §P0.1/§P0.2). A full symbols::scan_repo re-parses every .rs in a member on any change and re-emits the whole row set; JetBrains wins by re-indexing only the edited file. This closes that gap.

What landed

File-level incremental scan — src/knowledge/delta.rs

• Fingerprint manifest FileFingerprint { mtime_ns, size, hash } + Manifest { files: BTreeMap<rel_path, _> }. Cheap (mtime,size) gate → sha2 content-hash confirm. Keyed on the repo-root-relative path — the same string a scan writes into a row's file column, via the shared symbols::rel_path, so eviction can never silently miss rows.
• Pure differ diff(old,new) -> DeltaPlan { added, changed, removed, renamed }. Rename = a removed + an added path with byte-identical content-hash → served by relabel, not re-parse.
• Manifest::refresh builds the new manifest without reading unchanged files (mtime+size match ⇒ reuse old hash: a stat, no read, no re-hash); only moved files are read + re-hashed.
• SymbolDelta { manifest, scan } — full_build (cold) / rescan (hot). rescan = evict changed∪removed (Vec::retain, in place) → re-parse added∪changed only (symbols::scan_selected_files, serial) → merge (Vec::append, a move) → relabel renames. No per-row clone on the hot path.

Evict-on-change invariant: a changed file's removed symbols disappear because its rows are evicted before the re-parse merges the fresh set. Proven by delta_matches_full_rebuild (delta result == from-scratch full build, exactly).

inotify watch mode — src/knowledge/watch.rs

• Rides libc's inotify syscalls directly — no new crate (pure-Rust / reuse-existing charter). Feature inotify (off by default) + target_os = "linux"; elsewhere watch() returns WatchError::Unsupported and the caller keeps the timer-poll. Clean degrade, never a compile break or panic.
• Debouncer — a pure, unit-tested state machine coalescing an editor "save" burst into one .rs-only batch fired after quiet of silence.

Wired 2026-07-23 — it had no caller and no build

For a month watch() had zero callers and the inotify feature was named by no build variant, so the real inotify loop was compiled by nothing. Green tests the whole time, because the module's four tests only drive the pure Debouncer. Closed by:

• Caller — nornir watch <repo> (run_watch, src/bin/nornir.rs): one cold SymbolDelta::full_build, then each debounced burst drives SymbolDelta::rescan (re-parse only the changed files) and appends the merged scan as a fresh snapshot. SIGINT sets the stop flag so Ctrl-C ends the loop between deltas instead of mid-append. Without the feature the command reports exactly why and names the variant to build — it never pretends to watch.
• Artifact — the nornir-watch variant in .nornir/build-variants.toml (fat nornir + inotify, the live-tree node). Built on demand by nornir build --variant nornir-watch; nothing schedules that, so declaring it does not compile the loop.
• Build — the compile that does happen automatically is CI's cargo test -p nornir --features inotify --test inotify_watch_wiring line in .woodpecker.yml's build-test step. It is the only thing in the repo that turns the feature on without a human asking. inotify = [] pulls no dependency, so it costs a recompile of nornir's own lib+bin+test and nothing else.
• Guard — tests/inotify_watch_wiring.rs, whose per-test failure modes are spelled out in its module docs. In a feature-OFF build it runs the real binary and requires nornir watch to exist and to REFUSE, naming the feature and the variant; in the feature-ON build CI produces it additionally drives nornir watch --max-deltas 1 end to end over a written .rs file, and calls watch() directly to check the loop delivers the changed path. It also asserts the CI line above still exists — without it the two behavioural tests are compiled out and the file would silently shrink to config-text assertions over a dead subsystem.

Placement (Rickard, 2026-07-23). inotify is for a live tree — one that changes WITHOUT git. On a server the git transaction is the trigger and gix sees every change, so monitor::sync_once's poll stays the path there and the default variants leave the feature off. That is a deliberate narrowing of the original "inotify REPLACES the timer-poll, full stop" (workspace_nordisk/.nornir/populate-background-delta-design.md §2): the mechanism is the one that doc specifies — FS events → debounce → per-member delta — but it is mounted on the CLI live-tree entry point, not inside the monitor.

Bench + functional_status

• nornir.delta_rescan (src/bench_registry.rs, linked via register_bench!): 400-file crate, 1-file edit → delta_rescan_ms vs full_rebuild_ms + speedup_vs_full + files_touched.
• functional_status markers at the scan / evict / merge branches (--features testmatrix; #[inline] no-op otherwise).

SCIP — reused, not re-derived

ScipRow carries the same repo-relative path, so the identical evict-by-file / merge-by-file machinery applies to a ScipScan unchanged. SymbolDelta::evict_file is the exposed shared key. Resolved tables are still populate-gated, so this landing deltas the populated syn layer; routing the resolved layer through the same key is the remaining SCIP wiring.

Latency intuition

Delta-rescan ≈ O(1 file parse) + O(N stat) vs full-rebuild O(N file parse) — speedup_vs_full grows ~linearly with N.

Not done (risk-gated)

Row-level Iceberg MOR/equality-delete append (iceberg.rs:2060) — rewrites warehouse commit/schema semantics, bench-gated per the audit. The delta engine produces the exact changed row-set (the hard part); the MOR write path is the remaining follow-up. Until it lands, a delta appends the merged whole scan as a new snapshot (correct — readers take the latest snapshot per repo — but it writes N rows for a 1-file change, so the delta's win today is the parse, not the append).

Routing the monitor's timer-poll through watch() is explicitly NOT a follow-up any more — see the placement note above: a git-driven server does not need it.