src/roster.rs
//! **The honest roster: what each collector's state IS, and what has actually been
//! proven about it.**
//!
//! The owner's part (4), 2026-09-06: *"a collector whose parsing is proven on fixtures but
//! whose LIVE authenticated call has never run is UNTESTED ONLINE, and that is a
//! first-class state to report, not a red and not a lie. Say it in the roster with the
//! reason."*
//!
//! # Two questions, not one
//!
//! A roster that answers only "can this run?" is the roster this crate already had, and
//! it is the one that let a collector be reported as fine while nothing had ever
//! exercised it against its real endpoint. So a row carries two orthogonal answers:
//!
//! | axis | type | the question |
//! |---|---|---|
//! | startability | [`Readiness`](crate::requires::Readiness) | is the declared config filled in? |
//! | proof | [`LiveProof`] | has anything ever come back from the real endpoint? |
//!
//! They are genuinely independent, and every one of the four combinations occurs.
//! `polisen-events` is `Runnable` and proven online. `entsoe` is `NeedsConfig` and
//! UNTESTED ONLINE. `nordpool` is `NeedsConfig` for its licensed endpoint while its
//! parser is proven against a real payload from the provider's public one. Collapsing
//! the two axes into one traffic light is how the second fact gets lost.
//!
//! # The proof is DERIVED from evidence, never typed
//!
//! Exactly as [`Readiness`](crate::requires::Readiness) is. Nothing stores "this
//! collector works": a stored flag is a second opinion that goes stale the moment a key
//! is removed or a provider changes a field. [`LiveProof::of`] reads three pieces of
//! evidence a host already holds and derives the answer:
//!
//! * a **freshness mark** for `(collector, source)` — written ONLY after rows landed
//! from the real endpoint, so its existence is the strongest evidence available that
//! the live call has run;
//! * the **provider fixtures** committed for that collector, and of which kind — real
//! data, or only the provider's refusal document;
//! * whether the collector is **startable at all** right now.
//!
//! # UNTESTED ONLINE is not a failure
//!
//! [`LiveProof::is_red`] is `false` for it, deliberately. A collector whose parser is
//! proven against the provider's own payload and whose authenticated call has never been
//! made is in a perfectly respectable state — it is what every credential-gated
//! collector in this crate is in, and it will stay that way, because the owner is not
//! going to create the accounts. Painting it red would make the roster's red mean "we
//! have no key", which is not something anybody can fix by looking at the code, and a
//! red nobody can act on is a red everybody learns to ignore.
use serde::{Deserialize, Serialize};
use crate::provenance::{Access, ProvenanceManifest};
use crate::requires::{Readiness, Requirements, SecretPresence};
/// **What has actually been proven about this collector against its real source.**
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "proof", rename_all = "snake_case")]
pub enum LiveProof {
/// **Rows have landed from the real endpoint.** A freshness mark exists for
/// `(collector, source)`, and a mark is written only after a successful land — so
/// this is evidence of a completed round trip, not of an attempt.
ProvenOnline {
/// When the mark was written (RFC-3339 UTC), and what kind wrote it.
evidence: String,
},
/// **UNTESTED ONLINE** — the parser is proven against payloads the provider
/// published, and the live call has never run here.
///
/// The first-class honest state the owner asked for. `why` says which fixtures carry
/// the proof and what is missing for the live call, so the row is actionable
/// without being alarming.
UntestedOnline {
/// The sentence a roster prints.
why: String,
},
/// **Nothing has exercised this collector at all** — no provider fixture, no landed
/// run. Its parser may be covered by a synthetic fixture, which proves that the
/// parser accepts what its author expected and nothing more.
///
/// This one IS a red, and it is the only one that is: it names work nobody has done
/// yet, and somebody can do it.
Unproven {
/// What is absent.
why: String,
},
/// **The provider publishes nothing an anonymous reader can take — checked, and
/// dated.**
///
/// Bolagsverket answers an automated fetch with a CAPTCHA; SCB does not publish
/// företagsregistret's endpoint at all, because it is issued WITH the credential.
/// Neither is a gap in our work and neither can be closed by writing code, so
/// neither is a red — but leaving them indistinguishable from "nobody got round to
/// it" would send the next reader to find the same CAPTCHA.
///
/// Dated on purpose: a provider that publishes nothing today may publish an OpenAPI
/// next year, and an undated "impossible" is how a stale belief outlives its
/// evidence.
UnprovableWithoutAccount {
/// What was tried, what came back, and when.
why: String,
},
}
impl LiveProof {
/// The stable machine key (the wire, the CLI, a test).
pub fn key(&self) -> &'static str {
match self {
LiveProof::ProvenOnline { .. } => "proven_online",
LiveProof::UntestedOnline { .. } => "untested_online",
LiveProof::Unproven { .. } => "unproven",
LiveProof::UnprovableWithoutAccount { .. } => "unprovable_without_account",
}
}
/// The reason line, in the collector's own terms.
pub fn why(&self) -> &str {
match self {
LiveProof::ProvenOnline { evidence } => evidence,
LiveProof::UntestedOnline { why }
| LiveProof::Unproven { why }
| LiveProof::UnprovableWithoutAccount { why } => why,
}
}
/// **Is this a fault?** Only [`LiveProof::Unproven`] is — and even that is a gap in
/// our work, not in the collector. `UntestedOnline` is explicitly NOT red: see the
/// module note.
pub fn is_red(&self) -> bool {
matches!(self, LiveProof::Unproven { .. })
}
/// **Derive the proof from the evidence.**
///
/// `mark` is the freshness mark a host holds for `(collector, source)`, `fixtures`
/// the provenance manifest, `collector` the registry name.
pub fn of(collector: &str, mark: Option<&crate::freshness::ReadMark>, fixtures: &ProvenanceManifest) -> Self {
if let Some(m) = mark {
return LiveProof::ProvenOnline {
evidence: format!(
"rows landed from the live source; freshness mark `{}` written {} by \
the `{}` reader",
if m.mark.is_empty() { "(validators only)" } else { &m.mark },
if m.at.is_empty() { "(undated)" } else { &m.at },
m.kind,
),
};
}
let mine: Vec<_> = fixtures
.fixtures
.iter()
.filter(|f| f.collector == collector)
.collect();
if mine.is_empty() {
// Was this looked into and found impossible, or simply not done yet? The two
// read the same to a user and mean opposite things to whoever picks it up.
if let Some(u) = fixtures.unobtainable_for(collector) {
return LiveProof::UnprovableWithoutAccount {
why: format!(
"no credential-free payload exists — {} ({}, checked {}). This \
cannot be closed by writing code, only by an account.",
u.reason, u.provider, u.checked_utc,
),
};
}
return LiveProof::Unproven {
why: "no provider fixture is committed for this collector and no run has \
ever landed rows from it — any synthetic fixture in the tree proves \
only that the parser accepts what its author expected"
.to_string(),
};
}
// A refusal document is real provenance and a real guard, and it is NOT evidence
// that the collector can read the provider's DATA. Saying so is the difference
// between a roster and a scoreboard.
let data: Vec<&str> = mine
.iter()
.filter(|f| f.access.expects_rows())
.map(|f| f.file.as_str())
.collect();
let refusals: Vec<&str> = mine
.iter()
.filter(|f| f.access == Access::ProviderErrorDocument)
.map(|f| f.file.as_str())
.collect();
let dates: Vec<&str> = mine.iter().map(|f| f.fetched_utc.as_str()).collect();
if data.is_empty() {
return LiveProof::UntestedOnline {
why: format!(
"UNTESTED ONLINE — the only payload this provider serves without an \
account is its refusal document ({}), fetched {}. That proves the \
collector REFUSES a refusal by name rather than reading it as an \
empty result; it does not prove it can read the provider's data. \
The live authenticated call has never run here.",
refusals.join(", "),
dates.join(", "),
),
};
}
LiveProof::UntestedOnline {
why: format!(
"UNTESTED ONLINE — parse, schema, business key, freshness and graph \
mapping are proven against {} payload(s) the provider published ({}), \
fetched {}. The live authenticated call has never run here.",
data.len(),
data.join(", "),
dates.join(", "),
),
}
}
}
/// One roster row: both axes, and the reason for each.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct RosterRow {
/// `meta().name`.
pub collector: String,
/// Can it start? Every unmet requirement named.
pub readiness: Readiness,
/// Has anything come back from the real source?
pub proof: LiveProof,
}
impl RosterRow {
/// **The one line an operator reads**, both axes and their reasons.
pub fn line(&self) -> String {
let missing = match &self.readiness {
Readiness::NeedsConfig { missing } if !missing.is_empty() => {
format!(" — {}", missing.iter().map(|m| m.line()).collect::<Vec<_>>().join("; "))
}
_ => String::new(),
};
format!(
"{:<32} {:<12} {:<16} {}{missing}",
self.collector,
self.readiness.key(),
self.proof.key(),
self.proof.why(),
)
}
}
/// **The whole roster**, in registry order.
///
/// `secrets` is the host's live secret backend (korp's vault, or
/// [`EnvSecretStore`](crate::requires::EnvSecretStore)); `marks` the host's freshness
/// store, whose marks are the evidence of a completed live run; `fixtures` the committed
/// provenance manifest.
///
/// Every argument is a seam rather than something read here, for the same reason
/// readiness asks a `SecretPresence` instead of `std::env`: a host that swaps its vault
/// or its mark store changes what the roster says without the roster knowing that vaults
/// or databases exist.
#[cfg(feature = "parsers")]
pub fn roster(
secrets: &dyn SecretPresence,
marks: &dyn crate::freshness::MarkStore,
fixtures: &ProvenanceManifest,
) -> Vec<RosterRow> {
crate::registry()
.iter()
.map(|c| {
let name = c.meta().name;
let req: Requirements = c.requirements();
let params = serde_json::json!({});
let mark = marks.mark(&name, &req.source);
RosterRow {
readiness: req.readiness(secrets, ¶ms, true, false),
proof: LiveProof::of(&name, mark.as_ref(), fixtures),
collector: name,
}
})
.collect()
}