nordisk/korp-collectors

RULE: Never add Co-Authored-By or any AI attribution to commits. Ever.

korp-collectors

The standalone LEAF of korp's open-data collector layer — one object-safe Collector trait + per-access-METHOD collectors (DCAT-AP-SE, PxWeb/JSON-stat2, ENTSO-E, generic REST/JSON, Trafikverket, OSM) parsing pure-Rust → Arrow, an Iceberg BatchSink (via skade), and the RegionSource fetch seam.

Why it is a leaf

It depends on neither korp nor facett, so BOTH korp and facett-demo can depend on it. That breaks the facett ⇄ korp dependency cycle: facett never depends on korp; both consume this leaf, which depends on neither.

Keep it a leaf. Allowed deps: arrow-array/arrow-schema 58, parquet 58, anyhow, async-trait, reqwest (rustls), serde/serde_json, governor, bytes, tokio; feature-gated: osm-katana (osm), skade (skade-sink), nornir-testmatrix (testmatrix). No facett-*, no korp dep — ever.

Features

parsers (default) · osm (osm-katana + RegionSource) · skade-sink (Iceberg runner) · testmatrix (nornir functional-status rows; off by default).

Laws

• Every crate builds AND has real inject/assert tests (no "didn't panic").
• RAGNARÖK: a collect landing 0 rows is RED unless CollectStats.reason says why.
• The RegionSource fetch test uses REAL OSM bytes (osm-katana parses ≥1 way), an in-process server, and a request counter — deterministic, no live network.
• Tests emit nornir_testmatrix::functional_status(...) rows under --features testmatrix (feature OFF = no-op, no nornir dep). Use the crate wrapper korp_collectors::functional_status(...); emit_registry_descriptors() emits one descriptor_declared row per collector (discovery-complete law).
• Every Collector DECLARES a typed source_target() → SourceTarget (the open-data endpoint it pulls FROM + where rows land: warehouse table / Arrow / a facett map scene). registry_descriptors() enumerates (meta, source→target) so the test-matrix iterates (collector, source, target) tuples.
• Real-or-fake samples: every collector has a sample::*_try_get_real_sample_data() that ATTEMPTS a short-timeout live fetch (validated against its own parser) and FALLS BACK to an embedded Synthetic fixture — tests + benches never hang and never fail for want of a network. KORP_COLLECTORS_OFFLINE=1 forces synthetic.
• Benches (benches/collectors_parse.rs, criterion, harness = false) measure per-collector parse/normalize throughput over that same offline-safe sample.

Direction taxonomy

Every Collector is Direction::Source; the BatchSink is Direction::Sink. This axis is orthogonal to SourceKind (the transport/format family).