nordisk/holger

Repository replication — design

Class: design · Function: replication (uses registry for the per-repo list()/fetch()/put() seam, the in-process Holger facade for the two-store adapter, and grpc for the RPC surface). The whole-system view is design.md; the parity gap this closes is feature-complete-gaps.md §9 (replication / HA — previously ❌) and artifactory-parity-design.md §I.

What this is

Repository replication answers the Artifactory/Nexus "replication" table stakes: copy a repository's artifacts from one holger store into another. It ships as server/lib/src/replication.rs — a pure function over two backend sets:

replicate_pairs(pairs: &[RepoPair], opts: &ReplicationOptions) -> ReplicationReport

A RepoPair is a (source_backend, dest_backend) pairing. The engine walks each source's list(), fetch()es the bytes, and — for every artifact the destination does not already hold by content id — put()s it into the writable destination. Its only I/O is the backend calls it is given (RepositoryBackendTrait::{list,fetch,put,is_writable}), so it is unit-testable with in-memory backends and reused verbatim by both surfaces below.

The three invariants

1. Read-from-source / write-to-dest, never the reverse. The source is only ever list()ed + fetch()ed; the destination is only ever fetch()ed (to probe presence) + put() (to copy). A source is never mutated. This is the direction the airgap agent's ConnectorTrait already moves bytes across a boundary — replication is the server-to-server automation of that motion.

2. Idempotent by content id. The dedup key is the artifact's content digest (lowercase-hex sha256 of the bytes — the same digest search.rs matches). Before copying, the engine fetches the destination's bytes for the same coordinate:

   • present and hashing to the same content id ⇒ skip (already there);
   • absent, or present with different content ⇒ copy (converging the destination onto the source). So a second replicate run over an unchanged source is a pure no-op — every artifact skipped, nothing written. This is the property the executor test asserts (copied()==0, skipped()==all).

3. Gated + fail-closed. execute defaults to false: a dry run reports what would copy (Outcome::WouldCopy) and writes nothing. An actual put() only happens under ReplicationOptions::executing(). A destination backend that is not writable fails that artifact closed (Outcome::Failed, reported in the report), never a silent drop — the report accounts for every source artifact examined.

The report

ReplicationReport carries one ReplicatedArtifact row per source artifact examined (both repo names, the content id, size, an Outcome, and a failure detail), plus derived tallies: examined(), copied(), skipped(), would_copy(), failed(), copied_bytes(), and is_complete() (true when nothing failed). truncated flags a limit cap hit.

The two surfaces (one engine)

• CLI holger-server replicate --from FROM.ron --to TO.ron [--repo R] [--execute] [--limit N] — the two-store form. Loads two RON configs as separate in-process Holger stores, pairs their repos by name (source repo foo → destination repo foo), and drives Holger::replicate_into. Both stores file-backed ⇒ laptop-testable, no network. Default dry-run; exits non-zero if any artifact failed.

• gRPC ReplicationService.Replicate(from_repo, to_repo, execute, limit) — the in-server form. Replicates between two repos on one running server (the destination must be writable). Admin-gated on the destination repo (like promotion), fail-closed; source==dest is rejected (InvalidArgument). Audited (AuditAction::Upload, identity + copied bytes + a copied/skipped/failed detail) and emits a holger-grpc/replicate functional_status cell.

The pairing math is the only difference: the CLI pairs many repos across two stores by name; the RPC pairs one explicit from_repo → to_repo within one store. Both build RepoPairs and call the same replicate_pairs.

Why this shape (design, not alternatives)

The engine is a pure function over the existing RepositoryBackendTrait seam — no new backend, no network client, no scheduler — exactly as the retention planner and the search engine are. That is what makes it laptop-testable against two file-backed stores and reusable across both surfaces without twinning (Law #5). Content-id idempotency reuses the sha256 digest the search engine already computes, so "already present" means the same bytes, and re-runs are provably no-ops. Alternatives (a push-daemon, a change-feed subscription, multi-site consensus) are HA concerns tracked separately in §9 and belong to a scheduler / skade-routed follow-up — see reasoning.md.

What is deferred

Continuous / event-driven replication (replicate-on-push), multi-site topologies, conflict policy beyond last-writer-converges, and S3/object-backend scale-out stay 🧭 in §9 (route scheduling through jera, historize through skade — never hand-roll a daemon). This increment ships the primitive: a gated, idempotent, one-shot source→dest copy the CLI and the RPC both drive.