traits/src/migrate.rs
//! **MIGRATE — the ONE-TIME move of an existing registry into holger.**
//!
//! An operator who already runs Nexus or Artifactory has one question before he
//! will run holger at all: *how do I get what I have into it?* Today the answer
//! is two commands and a `.znippy` file on a disk between them. This module is
//! the vocabulary for doing it in one pass, and it lives in `traits` — not in the
//! agent and not in the server — because BOTH surfaces have to agree on the
//! answers, and a classifier that exists twice is a classifier that will disagree
//! once:
//!
//! * `holger-agent --from nexus --to holger` (the CLI, over the wire), and
//! * `MigrationService` in the server (a UI driving a job that runs for hours).
//!
//! # A migration RUNS ONCE AND FINISHES
//!
//! This is deliberately NOT a mirror. A mirror is a standing relationship that
//! keeps converging; a migration is a move with an end. Everything here is shaped
//! by that: the plan is computed once up front, the run has a terminal state, and
//! nothing schedules a second pass. A continuous mirror is a different thing and
//! must not be built by widening this one.
//!
//! # Three things this module owns
//!
//! 1. **The format map.** A source repository's format string (`maven2`, `pypi`,
//! `rubygems`, `generic`, …) → holger's [`ArtifactFormat`] → the `ron_repo_type`
//! an operator writes. It delegates to [`ArtifactFormat::from_format_str`] and
//! [`ArtifactFormat::home`], so migrate gains a format the moment holger does
//! and invents no vocabulary of its own.
//! 2. **The creation plan + the collision refusal.** Target repositories are
//! created automatically, but a dry run must first list EXACTLY what would be
//! created, and creation is REFUSED when the name is already taken by a
//! repository of a different format. Silently migrating maven artifacts into an
//! existing `npm` repository of the same name is the one outcome nobody can
//! undo, so it is refused before a single byte moves — see
//! [`MigrationPlan::refuse_collisions`].
//! 3. **The sink seam.** [`MigrationSink`] is the two-question interface the byte
//! mover needs of a destination: *do you already have this?* and *store it*.
//! The CLI answers it over a `ConnectorTrait` (a remote holger); the server
//! answers it against its own local route table. One runner, two sinks, no twin.
//!
//! # Idempotence is a property of the sink, and it is spelled sha256
//!
//! [`MigrationSink::already_has`] is handed the digest the SOURCE published
//! ([`RemoteAsset::sha256`]). A destination that holds the same path with the same
//! digest is already migrated, so the asset is skipped and a re-run of an
//! interrupted migration resumes instead of duplicating. A destination that holds
//! the path with a DIFFERENT digest is not the same artifact, and the run converges
//! it. A source that published no digest leaves presence-by-path as the only
//! answer available — which is why a connector that knows a digest must publish it.
use std::fmt;
use serde::{Deserialize, Serialize};
use crate::{ArtifactFormat, RemoteRepository};
/// What migrating one source repository into holger would MEAN for the
/// destination's repository set.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum Disposition {
/// The destination already has a repository of this name AND a compatible
/// format. Artifacts land in it; nothing is created.
Existing,
/// No repository of this name in the destination — migrating creates one,
/// hosted + writable, in the mapped format.
Create,
/// **REFUSED.** The name is taken by a repository of a DIFFERENT format.
/// Creating would collide; migrating into it would mix two ecosystems in one
/// store under a backend that cannot index one of them.
Collision {
/// The format the destination's repository of this name actually has.
existing_format: String,
},
/// **REFUSED.** holger has no format for what the source calls this
/// repository, so there is nothing to create it AS.
UnmappedFormat,
}
impl Disposition {
/// True when this disposition stops the whole migration. Both refusals are
/// fatal to the RUN, not merely to the repository: an operator who asked to
/// move a registry is owed all-or-explicitly-nothing, not a half-move he
/// discovers later.
pub fn is_refusal(&self) -> bool {
matches!(
self,
Disposition::Collision { .. } | Disposition::UnmappedFormat
)
}
}
/// One source repository, what it maps to, and what it would cost to take it.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RepoPlan {
/// The repository's name in Nexus / Artifactory.
pub source_repo: String,
/// The format string the SOURCE used (`maven2`, `pypi`, `generic`, …), kept
/// verbatim so a refusal can quote what the source actually said.
pub source_format: String,
/// The repository name in holger. Same as `source_repo` today — migrate does
/// not rename, because an operator's build files already carry these names.
pub target_repo: String,
/// holger's `ron_repo_type` for the mapped format (`maven3`, `pip`, `raw`, …)
/// — the word an operator writes in RON. Empty for [`Disposition::UnmappedFormat`].
pub target_format: String,
pub disposition: Disposition,
/// Assets the source listed for this repository. A LISTING count: no artifact
/// content is read to produce it.
pub assets: u64,
/// Sum of the sizes the source's listing advertised. `0` where the source
/// advertised none — that is "not stated", not "empty".
pub bytes: u64,
}
/// The whole one-time move, as a preview. This is what a dry run prints and what
/// the UI's Discover shows.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct MigrationPlan {
pub repos: Vec<RepoPlan>,
}
/// Why a plan refuses to run.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PlanRefused {
/// One line per refusing repository, each naming the repository and the reason.
pub reasons: Vec<String>,
}
impl fmt::Display for PlanRefused {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "migration refused: {}", self.reasons.join("; "))
}
}
impl std::error::Error for PlanRefused {}
impl MigrationPlan {
/// Repositories that would be CREATED in the destination. This is the list a
/// dry run must print before anything is created — an operator gets to see
/// every repository his registry is about to grow, by name and by format.
pub fn to_create(&self) -> Vec<&RepoPlan> {
self.repos
.iter()
.filter(|r| r.disposition == Disposition::Create)
.collect()
}
/// Repositories that already exist and will simply be filled.
pub fn existing(&self) -> Vec<&RepoPlan> {
self.repos
.iter()
.filter(|r| r.disposition == Disposition::Existing)
.collect()
}
/// Every repository that refuses (collision or unmapped format).
pub fn refusals(&self) -> Vec<&RepoPlan> {
self.repos
.iter()
.filter(|r| r.disposition.is_refusal())
.collect()
}
/// Assets the whole plan would examine.
pub fn total_assets(&self) -> u64 {
self.repos.iter().map(|r| r.assets).sum()
}
/// Bytes the source's listing advertised across the whole plan.
pub fn total_bytes(&self) -> u64 {
self.repos.iter().map(|r| r.bytes).sum()
}
/// **The gate.** `Ok(())` only when nothing refuses. Call this before moving
/// a byte: a collision discovered halfway through has already mixed two
/// ecosystems in one store.
pub fn refuse_collisions(&self) -> Result<(), PlanRefused> {
let reasons: Vec<String> = self
.repos
.iter()
.filter_map(|r| match &r.disposition {
Disposition::Collision { existing_format } => Some(format!(
"'{}' exists in holger as format '{}', but the source calls it '{}' \
(mapped: '{}') — refusing to create or write into a repository of a \
different format",
r.target_repo, existing_format, r.source_format, r.target_format
)),
Disposition::UnmappedFormat => Some(format!(
"'{}' has source format '{}', which holger has no repository format for",
r.source_repo, r.source_format
)),
_ => None,
})
.collect();
if reasons.is_empty() {
Ok(())
} else {
Err(PlanRefused { reasons })
}
}
/// The human dry-run report: exactly what would be created, what would be
/// filled, what refuses, and the totals. ONE renderer, so the CLI's
/// `--dry-run` and anything else that prints a preview print the same words.
pub fn dry_run_report(&self) -> String {
let mut out = String::new();
out.push_str("MIGRATION DRY RUN — nothing is created and no bytes move\n");
let create = self.to_create();
out.push_str(&format!("\nWOULD CREATE ({}):\n", create.len()));
if create.is_empty() {
out.push_str(" (none — every source repository already exists in holger)\n");
}
for r in &create {
out.push_str(&format!(
" + {} (holger format '{}', from source format '{}') — {} assets, {} bytes\n",
r.target_repo, r.target_format, r.source_format, r.assets, r.bytes
));
}
let existing = self.existing();
out.push_str(&format!(
"\nWOULD FILL, ALREADY EXISTS ({}):\n",
existing.len()
));
if existing.is_empty() {
out.push_str(" (none)\n");
}
for r in &existing {
out.push_str(&format!(
" = {} (holger format '{}') — {} assets, {} bytes\n",
r.target_repo, r.target_format, r.assets, r.bytes
));
}
let refusals = self.refusals();
if !refusals.is_empty() {
out.push_str(&format!("\nREFUSED ({}):\n", refusals.len()));
if let Err(e) = self.refuse_collisions() {
for reason in e.reasons {
out.push_str(&format!(" ! {reason}\n"));
}
}
}
out.push_str(&format!(
"\nTOTAL: {} repositories, {} assets, {} bytes advertised by the source listing\n",
self.repos.len(),
self.total_assets(),
self.total_bytes()
));
if refusals.is_empty() {
out.push_str("This plan would RUN. Re-run without --dry-run to execute it.\n");
} else {
out.push_str("This migration plan is REFUSED and will not run until the refusals above are resolved.\n");
}
out
}
}
/// Map ONE source repository onto its holger disposition, given the destination's
/// current repository set.
///
/// `existing` is the destination's repositories, as holger reports them. A name
/// match is compared on the MAPPED format, not the raw string, so Artifactory's
/// `generic` and holger's `raw` are the same repository and not a collision.
pub fn plan_one(
source: &RemoteRepository,
existing: &[RemoteRepository],
assets: u64,
bytes: u64,
) -> RepoPlan {
let target_repo = source.name.clone();
let mapped = ArtifactFormat::from_format_str(&source.format);
let (target_format, disposition) = match mapped {
None => (String::new(), Disposition::UnmappedFormat),
Some(fmt) => {
let config_key = fmt.home().config_key.to_string();
let found = existing.iter().find(|e| e.name == target_repo);
let disp = match found {
None => Disposition::Create,
Some(e) => {
// Compare through the map: the destination may spell the same
// format with a different word than the source did.
match ArtifactFormat::from_format_str(&e.format) {
Some(existing_fmt) if existing_fmt == fmt => Disposition::Existing,
_ => Disposition::Collision {
existing_format: e.format.clone(),
},
}
}
};
(config_key, disp)
}
};
RepoPlan {
source_repo: source.name.clone(),
source_format: source.format.clone(),
target_repo,
target_format,
disposition,
assets,
bytes,
}
}
/// The destination of a migration, reduced to the only two questions the byte
/// mover asks of it.
///
/// Deliberately tiny. A `ConnectorTrait` (a remote holger over HTTP) and a live
/// server's own route table are different enough that sharing the whole connector
/// vocabulary between them would mean stubbing half of it; they are identical in
/// exactly these two operations, so that is what the seam is.
#[async_trait::async_trait]
pub trait MigrationSink: Send + Sync {
/// Is `path` in `repository` ALREADY the artifact the source published?
///
/// `want_sha256` is the digest the SOURCE published, lowercase hex, or `None`
/// when it published none. An implementation answers `true` only when it is
/// confident the stored bytes are the same artifact; when it cannot tell, it
/// answers `false` and the asset is transferred again (a re-upload is waste,
/// a wrong `true` is data loss).
async fn already_has(
&self,
repository: &str,
path: &str,
want_sha256: Option<&str>,
) -> anyhow::Result<bool>;
/// Store `data` at `path` in `repository`, creating nothing implicitly that
/// the plan did not already account for.
async fn store(&self, repository: &str, path: &str, data: &[u8]) -> anyhow::Result<()>;
/// Create a hosted, writable repository named `repository` in holger's
/// `format` (a `ron_repo_type` such as `maven3` / `pip` / `raw`).
///
/// Only ever called for a [`Disposition::Create`] row, after
/// [`MigrationPlan::refuse_collisions`] has passed. A sink that cannot create
/// repositories returns an error naming what an operator must declare — it
/// does NOT quietly proceed and let every asset fail one by one.
async fn create_repository(&self, repository: &str, format: &str) -> anyhow::Result<()>;
}
/// **The declaration for ONE repository, in RON, rendered ONCE.**
///
/// This is the text an operator pastes into the wired config before a migration
/// that would create the repository can run. It lives here, in one function, so
/// that the sink's refusal below, the CLI dry run and the console's Migrate page
/// all hand over the SAME words — a person who reads two spellings of the same
/// instruction has to work out which one is current.
pub fn ron_declaration(repository: &str, format: &str) -> String {
format!(
"( ron_name: \"{repository}\", ron_repo_type: \"{format}\", \
ron_data_dir: Some(\"<data-root>/{repository}\"), ron_upstreams: [], ron_in: None, \
ron_out: Some(( ron_storage_endpoint: \"<storage>\", ron_exposed_endpoint: \"<endpoint>\" )) )"
)
}
/// **The one refusal text for "this destination cannot create a repository".**
///
/// holger builds its route table when it wires its config and the table is
/// immutable for the life of the process (`FastRoutes` is a first-byte bucketed
/// array the hot path reads without a lock). So neither a holger reached over
/// the wire nor a holger migrating into itself can grow a repository mid-run.
/// Rather than let every asset of that repository fail one by one with a 404,
/// the sink refuses ONCE and hands the operator the exact declaration to add.
///
/// It lives here, rendered once, because both sinks must say the same words: an
/// operator who reads two different instructions for the same problem has to
/// work out which one is current.
pub fn cannot_create_repository(repository: &str, format: &str) -> String {
let declaration = ron_declaration(repository, format);
format!(
"repository '{repository}' does not exist on the destination and cannot be created \
while the server is running: holger wires its route table from its config at boot \
and it is immutable thereafter. Declare it and restart:\n \
{declaration}\n\
A dry run lists EVERY repository a migration would create, so the whole set can be \
declared in one pass before the move."
)
}
#[cfg(test)]
mod tests {
use super::*;
fn repo(name: &str, format: &str) -> RemoteRepository {
RemoteRepository {
name: name.to_string(),
format: format.to_string(),
repo_type: "hosted".to_string(),
url: None,
}
}
#[test]
fn an_unknown_target_name_is_created_in_the_mapped_format() {
let p = plan_one(&repo("libs-release", "maven2"), &[], 12, 340);
assert_eq!(p.disposition, Disposition::Create);
assert_eq!(p.target_repo, "libs-release");
// Nexus says maven2; holger's operator word is maven3.
assert_eq!(p.target_format, "maven3");
assert_eq!(p.assets, 12);
assert_eq!(p.bytes, 340);
}
#[test]
fn a_name_that_exists_in_the_same_format_is_filled_not_created() {
// Artifactory spells it `maven`, holger spells it `maven3` — the SAME
// format, so this must not read as a collision.
let p = plan_one(&repo("libs", "maven"), &[repo("libs", "maven3")], 1, 1);
assert_eq!(p.disposition, Disposition::Existing);
assert!(p.to_string_check());
}
#[test]
fn generic_and_raw_are_the_same_repository() {
let p = plan_one(&repo("drops", "generic"), &[repo("drops", "raw")], 0, 0);
assert_eq!(
p.disposition,
Disposition::Existing,
"Artifactory 'generic' IS holger 'raw'"
);
}
#[test]
fn a_name_that_exists_in_a_different_format_collides_and_the_plan_refuses() {
let p = plan_one(&repo("shared", "maven2"), &[repo("shared", "npm")], 5, 5);
assert_eq!(
p.disposition,
Disposition::Collision {
existing_format: "npm".into()
}
);
let plan = MigrationPlan { repos: vec![p] };
let err = plan
.refuse_collisions()
.expect_err("a collision must refuse the whole run");
assert!(
err.reasons[0].contains("different format"),
"{:?}",
err.reasons
);
// RED-when-broken: drop the collision arm in `plan_one` and this passes as
// Existing, which silently writes maven artifacts into an npm store.
}
#[test]
fn a_format_holger_cannot_serve_refuses_rather_than_guessing_raw() {
let p = plan_one(&repo("weird", "p2"), &[], 3, 3);
assert_eq!(p.disposition, Disposition::UnmappedFormat);
assert_eq!(
p.target_format, "",
"an unmapped format names no target format"
);
let plan = MigrationPlan { repos: vec![p] };
assert!(plan.refuse_collisions().is_err());
}
#[test]
fn the_creation_refusal_hands_over_a_declaration_that_names_both_words() {
let text = cannot_create_repository("libs", "maven3");
assert!(text.contains("ron_name: \"libs\""), "{text}");
assert!(text.contains("ron_repo_type: \"maven3\""), "{text}");
assert!(
text.contains("dry run"),
"it must point at the preview: {text}"
);
}
/// The console shows the declaration on its own, beside each repository it
/// would create; the sink shows it inside a refusal. If those two ever stop
/// being the same string, an operator pastes one of them and the other is a
/// lie. There is ONE renderer, and this is the assertion that keeps it one.
#[test]
fn the_refusal_embeds_the_very_same_declaration_the_ui_shows() {
let declaration = ron_declaration("libs", "maven3");
assert!(
declaration.starts_with("( ron_name: \"libs\""),
"{declaration}"
);
assert!(
cannot_create_repository("libs", "maven3").contains(&declaration),
"the refusal must quote the one renderer verbatim"
);
}
#[test]
fn the_dry_run_report_names_every_repository_it_would_create() {
let plan = MigrationPlan {
repos: vec![
plan_one(&repo("libs-release", "maven2"), &[], 10, 1000),
plan_one(&repo("pypi-local", "pypi"), &[], 4, 40),
plan_one(&repo("kept", "npm"), &[repo("kept", "npm")], 2, 20),
],
};
let text = plan.dry_run_report();
assert!(text.contains("WOULD CREATE (2)"), "{text}");
assert!(
text.contains("+ libs-release (holger format 'maven3'"),
"{text}"
);
assert!(text.contains("+ pypi-local (holger format 'pip'"), "{text}");
assert!(text.contains("= kept (holger format 'npm')"), "{text}");
assert!(
text.contains("TOTAL: 3 repositories, 16 assets, 1060 bytes"),
"{text}"
);
assert!(text.contains("would RUN"), "{text}");
assert_eq!(plan.to_create().len(), 2);
assert_eq!(plan.existing().len(), 1);
}
#[test]
fn a_refused_plan_says_so_in_its_report_and_never_says_it_would_run() {
let plan = MigrationPlan {
repos: vec![plan_one(
&repo("shared", "maven2"),
&[repo("shared", "npm")],
1,
1,
)],
};
let text = plan.dry_run_report();
assert!(text.contains("REFUSED (1)"), "{text}");
assert!(
!text.contains("would RUN"),
"a refused plan must never invite execution: {text}"
);
}
impl RepoPlan {
/// Test helper: the mapped format is non-empty for everything but an
/// unmapped source format.
fn to_string_check(&self) -> bool {
!self.target_format.is_empty()
}
}
}