.nornir/metrics-guide.md

Prometheus `/metrics` — guide

Class: guide · Function: metrics. What the feature is and how to drive it. The why lives in metrics-design.md; the gentle version in metrics-for-idiots.md.

Holger's HTTP/OCI gateway exposes a standard Prometheus scrape endpoint at GET /metrics, alongside /healthz and /readyz. It reports real registry activity so you can graph and alert on it. The request counters cover both doors — the gRPC service (holger's primary protocol) and the HTTP/OCI gateway — so a mostly-gRPC deployment still shows its traffic here.

Scrape it

curl -s http://your-holger:PORT/metrics

Sample output:

# HELP holger_requests_total Total requests handled (gRPC + HTTP), by verb.
# TYPE holger_requests_total counter
holger_requests_total{verb="get"} 1284
holger_requests_total{verb="put"} 57
holger_requests_total{verb="delete"} 3
holger_requests_total{verb="promote"} 9
holger_requests_total{verb="search"} 12
holger_requests_total{verb="sbom"} 4
# HELP holger_bytes_served_total Total response bytes served over HTTP.
# TYPE holger_bytes_served_total counter
holger_bytes_served_total 90173441
# HELP holger_errors_total Total 5xx HTTP responses.
# TYPE holger_errors_total counter
holger_errors_total 0
# HELP holger_retention_runs_total Total retention-executor runs.
# TYPE holger_retention_runs_total counter
holger_retention_runs_total 2
# HELP holger_proxy_requests_total Total proxy/group-repo reads, by outcome.
# TYPE holger_proxy_requests_total counter
holger_proxy_requests_total{result="primary_hit"} 903
holger_proxy_requests_total{result="upstream_hit"} 44
holger_proxy_requests_total{result="negcache_hit"} 61
holger_proxy_requests_total{result="miss"} 12
# HELP holger_request_duration_seconds HTTP request handling duration in seconds.
# TYPE holger_request_duration_seconds histogram
holger_request_duration_seconds_bucket{le="0.005"} 402
holger_request_duration_seconds_bucket{le="0.05"} 1183
holger_request_duration_seconds_bucket{le="0.5"} 1338
holger_request_duration_seconds_bucket{le="+Inf"} 1341
holger_request_duration_seconds_sum 74.913
holger_request_duration_seconds_count 1341
# HELP holger_repo_artifacts Current artifact count per repository.
# TYPE holger_repo_artifacts gauge
holger_repo_artifacts{repo="rust-dev"} 128
holger_repo_artifacts{repo="maven-prod"} 4471

Content-Type is text/plain; version=0.0.4; charset=utf-8 — the Prometheus text exposition format. Point a Prometheus scrape config at it directly.

What each family means

Family	Type	Meaning
`holger_requests_total{verb}`	counter	requests handled across BOTH doors (gRPC + HTTP), split by verb: `get` (reads/downloads + listings), `put` (writes/uploads), `delete`, `promote` (DEV→PROD promotions, gRPC-only), `search` (`/-/search` door), `sbom` (`/-/sbom` door)
`holger_bytes_served_total`	counter	total response bytes served over HTTP
`holger_errors_total`	counter	total 5xx responses (server errors)
`holger_retention_runs_total`	counter	retention-executor invocations (dry-run or executing)
`holger_proxy_requests_total{result}`	counter	proxy/group-repo reads split by outcome: `primary_hit` (served locally), `upstream_hit` (pulled through from an upstream), `negcache_hit` (a known-absent coordinate short-circuited), `miss` (absent everywhere). Plus `upstream_error` — not a request outcome but a per-upstream event: an upstream errored and was skipped so the walk could continue, so a non-zero value means a configured upstream is dead. Only a repo configured with upstreams emits these
`holger_request_duration_seconds`	histogram	HTTP request-handling latency (`_bucket{le}` cumulative counts, `_sum`, `_count`). Buckets span 1 ms … 10 s + `+Inf`. Measured at the HTTP door alongside the request counter; gRPC-plane latency is a follow-up
`holger_repo_artifacts{repo}`	gauge	current artifact count in each configured repository, sampled at scrape time

Prometheus scrape config

scrape_configs:
  - job_name: holger
    metrics_path: /metrics
    static_configs:
      - targets: ["your-holger:PORT"]

Semantics you can rely on

Counters are lifetime-of-process and monotonic — they reset to 0 on a server restart, which is exactly what Prometheus' rate() / increase() expect.
holger_repo_artifacts is a point-in-time gauge: it lists each backend on every scrape (bounded at 100k per repo), so it costs a little I/O — paid by the scraper, off the request hot path. A repo whose backend errors reports 0 rather than failing the scrape.
The endpoint is read-only, unauthenticated, and unaudited — like /healthz and /readyz. It exposes only aggregate counters, never artifact bytes or identities. Put it behind your network policy if you don't want it public.
A request is counted at the same point its audit row is written, so the counters and the audit log agree on what happened.

Useful queries

# request rate by verb over 5m
rate(holger_requests_total[5m])

# server error ratio
rate(holger_errors_total[5m]) / rate(holger_requests_total[5m])

# egress throughput (bytes/s)
rate(holger_bytes_served_total[5m])

# total artifacts across all repos
sum(holger_repo_artifacts)

# proxy cache hit-rate (local + upstream serves vs all reads)
sum(rate(holger_proxy_requests_total{result=~"primary_hit|upstream_hit"}[5m]))
  / sum(rate(holger_proxy_requests_total[5m]))

# 95th-percentile HTTP request latency over 5m
histogram_quantile(0.95, sum by (le) (rate(holger_request_duration_seconds_bucket[5m])))