nordisk/dwarves

Incremental republish — rebuild only the blast radius

Plan item F1 of THE_BIG_PLAN. The server's monitored-workspace poll loop used to rebuild the whole workspace index over all members on any git change. F1 makes the rebuild incremental: when only some members' HEAD moved, it rebuilds only the changed members ∪ their transitive dependents (the Mímir affected set), not the whole workspace.

Why

A monitored workspace can hold many member repos. Today a single commit to one leaf forces a full re-walk + re-index of every member — wasted work that grows with workspace size. But the poll loop already knows which members moved (it diffs each member's last_seen_sha), and the republish already builds a cross-repo dependency graph (cargo metadata --no-deps) to persist for the viz Dep Graph tab. F1 simply consults that graph: a change to a member can only affect that member and the members that (transitively) depend on it. Nothing else needs reindexing.

This is the graph-scoped sibling of the SHA-idempotency optimization already in republish_with step 1 (skip the knowledge scan of a member whose HEAD SHA was already scanned). F1 adds the same "rebuild only what moved" idea to the code-index rebuild (step 2), driven by the dep graph rather than per-member SHA equality.

The affected set

The blast radius of a change is computed by WorkspaceGraph::affected_by_change:

affected(changed) = ⋃ over c ∈ changed of  { c } ∪ dependents_transitive(c)

returned in build order (dependencies first). Edges point consumer → producer, so dependents_transitive(util) walks the reverse edges to find every member that would have to be rebuilt because util changed.

Worked example — the diamond app → liba → util, app → libb → util:

monitor::affected_set wraps affected_by_change and intersects the result with the present member checkouts — a changed member that has no materialized .git checkout (SSH-only, not yet clonable) or a graph node that isn't a real workspace member is dropped, so the index scope only ever names real member directories.

Wiring

poll sweep (sync_once / nornir-server sweep)
    │  rep.changed = members whose HEAD moved this tick
    ▼
republish(reg, root, ws, changed: Option<&[String]>)
    ▼
republish_with(.., members, ws, deep_scan, changed)
    │  1. build cross-repo dep graph (cargo metadata --no-deps) — up front
    │  2. scope = match (changed, graph):
    │        (Some(non-empty), Some(graph)) → affected_set(graph, changed, present)
    │        _                              → []   (= full rebuild)
    │  3. idx.build_scoped(&scope)          ← only the affected member dirs walked
    ▼
Index::build_scoped(scope) → walk_workspace(root, scope)

Index::build_scoped is the new entry point: build() delegates to it with the index's configured repo_scope; the incremental path passes the affected set instead. An empty scope walks the whole workspace — identical to an unscoped full build() — which is exactly the fallback behavior we want.

Fallbacks (always safe to over-build)

A scope of [] means "full rebuild". We fall back to it whenever scoping would be unsafe or meaningless:

• First publish — fat-mode nornir workspace add calls republish(.., None); there is no prior state to diff, so everything builds.
• Empty / wiped warehouse — the current_snapshot-is-empty path has no change-set (changed = None) → full rebuild so a cleared warehouse repopulates.
• Forced fetch — Workspaces.Fetch { force: true } passes None: an operator forcing a fetch means "rebuild everything", not just the diff.
• Dep graph unavailable — if cargo metadata hiccups and the graph can't be built, we can't know the blast radius, so we conservatively rebuild all members.

The graph is built once, up front and reused for the step-4 persist (it was previously rebuilt there), so incremental scoping costs no extra cargo metadata invocation.

What is not yet incremental

• The per-member knowledge scan (step 1) is already incremental by HEAD-SHA idempotency, independent of F1.
• The per-member Iceberg snapshot (step 3) still snapshots every present member each republish. The snapshot is keyed to each member's HEAD SHA and is cheap relative to the index walk; scoping it to the affected set is a possible follow-up but unaffected members simply re-snapshot to the same id.
• The graph is cross-repo, so a single-member workspace has no edges and no dependents — a change to it scopes to just itself (still correct, just no blast radius to compute).

Tests

monitor::tests::affected_set_is_changed_union_dependents_and_skips_unaffected (inject-assert, LAW 1): feeds a fake change-set + a real diamond WorkspaceGraph and asserts the affected set equals changed ∪ transitive dependents, that an unaffected sibling/dependency is skipped, that the result comes back in build order, and that a changed member with no checkout is dropped by the present-intersection.