nordisk/holger

Prometheus /metrics — design

Class: design · Function: metrics (uses serve/exposed::http for the scrape door and each backend's list() for the artifact gauge, and the retention executor for the run counter). The whole-system view is design.md; the parity gap this closes is feature-complete-gaps.md §11 (ops / observability — previously only /healthz + /readyz, /metrics ❌) and artifactory-parity-design.md §K.

What this is

A read-only GET /metrics door on holger's existing HTTP/OCI gateway that emits Prometheus text exposition format 0.0.4 for real registry activity: request counts per verb, bytes served, 5xx errors, retention runs, and a per-repository artifact-count gauge. It is the full-tier operability win the gap doc names — self-contained, high-value, zero new dependencies.

It ships as server/lib/src/metrics.rs — a handful of AtomicU64 counters behind a process-global metrics::global() accessor, plus a pure render function:

Metrics::render_prometheus(repo_counts: &[(String, usize)]) -> String

The render does no I/O of its own — the per-repo gauge is driven entirely by the (repo, count) list the caller gathers — so it is unit-testable with an isolated Metrics instance and reused verbatim by the live door.

The exposed families

Counters are lifetime-of-process (reset on restart, as Prometheus expects); holger_repo_artifacts is a point-in-time gauge sampled on each scrape.

The chosen design: bump at the natural call sites, render on scrape

The gap doc frames it as "Prometheus /metrics (per-repo request / bytes / hit-miss counters) — self-contained, high-value for full operability." So:

• Registry (metrics::Metrics) — plain AtomicU64 fields, one bump method (record_request(verb, status, bytes) folds the verb count, the bytes total, and the 5xx error count into one call so a verb can never be counted without its bytes/errors) plus record_retention_run(). Lock-free, Relaxed ordering — a counter has no cross-field invariant, so the cheapest atomic add is correct.
• Process-global (metrics::global(), a OnceLock<Metrics>) — the emitting sites live on three unrelated planes (the HTTP door, the retention executor, and the gRPC services' record_audit choke point). Threading one Arc<Metrics> through all of them would be a large cross-cutting change for a purely additive observability feature, so a single global registry lets any site bump without wiring. The render path stays a pure function over a &Metrics snapshot, so tests use an isolated instance and the global is never in the assertion path.
• HTTP door — GET /metrics on the existing gateway, mirroring the GET /-/search / GET /-/sbom read doors. Reserved as the conventional /metrics (Prometheus scrapes that path). Unauthenticated and unaudited — it exposes only aggregate counters, never artifact bytes or identities, and a scrape is not an artifact operation. Content-Type text/plain; version=0.0.4; charset=utf-8.

Why the per-repo gauge is the one scrape-time I/O

Counters are just an atomic load, but holger_repo_artifacts needs the live artifact count per repository, which means calling each backend's list(). That is real I/O, so the door runs it under spawn_blocking (off the async reactor, exactly as the /-/search door does) and bounds each listing at METRICS_LIST_CAP (100k) — the gauge is a health signal, not a census. A backend that errors reports 0 rather than failing the whole scrape.

No hot-path regression (Law #2)

The only cost added to the request path is one Relaxed atomic add per request (the verb/bytes/error bump), issued at the same single exit point that already writes the audit row — no new lock, allocation, or syscall. The scrape's list() cost is paid by the (infrequent) scraper, off the reactor, not by artifact traffic.

gRPC-plane counting — one choke point, one shared classifier

gRPC is holger's primary protocol ("gRPC only by design"), so counting only the HTTP door left the main surface invisible in /metrics. Both doors now feed the same holger_requests_total{verb} family:

• The gRPC bump lives at the single choke point every RPC already exits through — HolgerGrpc::record_audit. Each RPC records exactly one AuditEvent (carrying its action, status, and bytes), so one bump there covers fetch/list/put/promote/search/sbom/property/replicate without threading a metrics handle into each handler. HTTP does not route through record_audit, so the two never double-count.
• The mapping from an audit action to a metrics verb is one shared function — Verb::from_audit_action(AuditAction) -> Option<Verb> (Law #5, no twin): a read (download or listing) → get, a store → put, delete → delete, a DEV→PROD promotion → its own promote verb, and Other → None (not request traffic, not counted). The HTTP door consumes the same classifier (it previously had a parallel inline match — collapsed).
• promote is a distinct verb (the Artifactory-headline operation earns its own counter); it is fed only by the gRPC promote RPC (there is no HTTP promote door).

Proxy hit/miss counting — one bump per read outcome

The proxy/group repo (proxy::ProxyBackend) chains a primary backend with an ordered upstream fallback list; every read exits through exactly ONE clean outcome, and each is bumped on the process-global registry as holger_proxy_requests_total{result="…"} (Law #5 — the same metrics::global() registry, a new label family, no twin):

• primary_hit — the primary served the bytes (a local cache hit): the fetch primary-Some and the handle_http2_request primary-non-404 branches.
• upstream_hit — the primary missed and an upstream in the chain served (pull-through): the fetch upstream-Some and the HTTP upstream-non-404 branches (this is also where the write-through cache-fill happens).
• negcache_hit — a live negative-cache entry short-circuited the upstream walk (a known-absent coordinate returned fast without re-querying).
• miss — the primary AND every upstream cleanly missed (a true miss, then remembered for the negative-cache TTL when enabled).
• upstream_error — an upstream ERRORED (connection refused, DNS failure, TLS error, a non-404 status) and was skipped so the walk could continue. Unlike the four above this is a per-upstream event, not a request outcome: one read over a three-upstream chain with two dead members bumps it twice, and the read still lands on upstream_hit or an error. It is the signal that a configured upstream is dead — without it a degraded chain is silently tolerated. Exclude it when computing hit-rate.

Summing the four request outcomes is total proxy traffic; (primary_hit + upstream_hit) / total is the cache hit-rate an operator watches. The bump is one Relaxed atomic add at each outcome point (no lock, no allocation) — dormant unless a repo is configured with upstreams, so a plain repo pays nothing.

Request-duration histogram — measured where the counter is

holger_request_duration_seconds is a lock-free Prometheus histogram over the HTTP door's request-handling latency: a std::time::Instant is captured at the top of exposed::http::handle and observed (observe_request_duration) at each door's exit right where record_request bumps the counter — so the histogram and the counter cover the exact same handled-request population (search, sbom, property-write, and the artifact record closure). Implementation:

• Fixed conventional web-latency buckets (DURATION_BUCKETS, 1 ms … 10 s) plus an implicit +Inf; one AtomicU64 per bucket (non-cumulative — the render cumulates), an observation count, and a nanosecond sum. One Relaxed add per field, no lock or allocation.
• Render emits the standard histogram triplet: cumulative holger_request_duration_seconds_bucket{le="…"} (monotone, +Inf == _count), _sum, and _count. Non-finite/negative observations are dropped so a clock quirk can't corrupt the sum.

Scope (honest): this observes the HTTP door. The gRPC plane's latency is a documented follow-up — the gRPC counter choke point (HolgerGrpc::record_audit) sees an AuditEvent with no duration, and the metrics design deliberately avoids threading a per-RPC start Instant into every handler. The right home for gRPC latency is a tower timing layer in the shared edda/nornir-service serve loop (Law #5 — it would time nornir/dwarves/holger uniformly), which is an edda change out of scope for a holger-repo slice.

What is deliberately NOT here (honest scope)

• gRPC-plane latency — see the histogram scope note above (needs an edda/nornir-service tower layer; the HTTP door is covered today).

Testing (Law #4)

• Duration histogram — metrics::tests::duration_histogram_buckets_sum_and_count (RED-when-broken, isolated Metrics: four observations land in distinct buckets, the cumulative le buckets are monotone with +Inf == _count, the _sum is exact, and a NaN/negative observation is ignored — a dropped observe, a wrong bucket, or a broken cumulation fails).
• Proxy outcomes — metrics::tests::proxy_outcomes_count_and_render_by_result_label (RED-when-broken, isolated Metrics: distinct counts per outcome land on their own slot, a hit is not counted as a miss, and each renders as its own result="…" series) and proxy::tests::proxy_outcomes_bump_the_global_metrics_family (drives the REAL ProxyBackend read paths — primary-hit / upstream-hit / miss / negcache-hit — and asserts each bumps the process-global family by >= N, so a dropped bump fails end to end).
• Registry (metrics.rs) — 7 unit tests over an isolated Metrics: request_counter_counts_exactly_n_per_verb (after N records the verb reads exactly N, and a different verb stays 0 — RED-when-broken: a bump wired to the wrong slot or dropped fails it), errors_count_only_5xx, retention_runs_increment, render_is_parseable_prometheus_reflecting_state (every non-comment line parses as name value with a numeric value, AND the live counter state is reflected — 2 GETs render …{verb="get"} 2, the supplied repo counts render as gauge samples, an untouched verb renders 0 not absent), label_values_are_escaped (an odd repo name stays parseable), plus the two gRPC-plane tests: audit_action_maps_to_the_expected_verb (RED-when-broken the shared classifier — a promote folded into put, or Other counted, fails it) and recording_via_audit_action_counts_the_grpc_plane_by_verb (records via the audit action exactly as record_audit does; download+listing both land on get, a promotion renders as its own promote series).
• The door emits a functional_status cell gated #[cfg(feature="testmatrix")] (holger-http/metrics, check metrics_served) so nornir test --features testmatrix sees the live surface.