nordisk/korp-collectors

korp-collectors — the open-data collector leaf

What it is. The standalone LEAF of korp's ingest layer: one object-safe Collector trait + per-access-method collectors that parse open data (pure-Rust → Arrow), land it via an Iceberg BatchSink (skade), and declare a typed source_target() so the test-matrix can iterate (collector, source, target). Depends on neither korp nor facett — both consume it; it breaks the facett⇄korp cycle.

Function axis

• f0 — the Collector trait (object-safe) + registry() of all collectors.
• f1 — per-method sources: dcat-ap-se · pxweb/jsonstat2 · eurostat-sdmx (SDMX-JSON 2.1) · entsoe (price/load) · entsoe_generation (A75 per-type) · kolada · smhi-metobs · smhi-weather (stations) · polisen · trafikverket-trainpos/trains/roads · osm (RegionSource) · osm-overpass (POI) · ais (boats) · marinetraffic-ais (AISStream) · opensky-flights + opensky-europe-bbox (planes) · gbfs-citybikes · bolagsverket-organisationer (company registry) · nordpool (electricity spot prices) · energy-charts-power (generation-mix parallel arrays) · usgs-earthquakes (GeoJSON seismicity) · wfs (OGC GML features) · rss-atom (RSS/Atom syndication feeds) · openaq (air-quality measurements) · gtfs-stops/routes/trips (transit feed zip) · gtfs-rt-vehicles (GTFS-Realtime protobuf vehicle positions) · custom REST/JSON.
• f2 — parse: parse_records / parse_records_bounded → Arrow RecordBatch (pure Rust).
• f3 — sink: BatchSink → skade Iceberg table (feature skade-sink); geo collectors also target a facett map scene (e.g. osm3d-flight for opensky).
• f4 — coverage: every collector emits a descriptor_declared testmatrix cell (emit_registry_descriptors()), and each has an inject/assert test + an offline-safe real-or-synthetic sample. Green ⟺ every declared source is exercised in code.

Collectors (Direction::Source)

Laws (this leaf)

• Leaf: no korp/facett dep — ever.

• RAGNARÖK: a collect landing 0 rows is RED unless CollectStats.reason says why. The verdict is computable, not a convention: CollectStats::ragnarok() → Rows | ExplainedEmpty | Red, with is_red() / ragnarok_check() as the gate and run_strict() as the one-call runner that returns Err on an unexplained empty. run()'s own annotation uses the recognisable RAGNAROK_UNEXPLAINED sentinel, so the runner can never launder a silent empty into an honest one.

• The registry sweeps (tests/collectors_registry_sweep.rs) assert the two laws over EVERY collector in registry() rather than the handful with a hand-written test:

  1. sample::for_collector(name) resolves an offline-safe sample for every registered name (RED when a collector lands without one);
  2. every collector fed an EMPTY-but-shape-preserving source (its own sample with the arrays emptied, so its records pointer still resolves) either errors or returns 0 rows with a reason — never Ok with 0 rows and no reason;
  3. every collector collect()s its own sample into ≥1 row — RED when a parse breaks. One functional_status cell per (collector, law).

• Real-or-fake samples: attempt a short-timeout live fetch, fall back to an embedded Synthetic fixture — tests/benches never hang, never fail for want of network (KORP_COLLECTORS_OFFLINE=1 forces synthetic).

• The RECORDED sample URL is credential-free: some live sample endpoints carry a token IN THE QUERY (ENTSO-E securityToken, Destatis GENESIS token), and SampleOrigin::Real{url} is rendered by label() into durable test-matrix rows, so sample::redact_url scrubs known secret query params (and userinfo passwords) at the single point the fetch URL becomes recorded identity.

• Auth resolves through the ASYNC seam: every collector calls cfg.auth.resolve_async().await?, never the sync resolve(). The sync form cannot serve AuthConfig::OAuth2ClientCredentials (the token is a network round-trip) and errors out; resolve_async delegates every other variant back to it, so using it always is free. Swept by tests/collectors_auth.rs.

• testmatrix cells emitted under --features testmatrix (OFF = no-op, no nornir dep).

Open work (see korp/.nornir/korp-worklist.md)

• ✅ Europe-extent bbox (lamin/lomin/lamax/lomax) opensky variant landed (opensky-europe-bbox).
• The UI proof: an Ingest/Tools tab that runs a collector as a jera job — ships ONE nornir-robotui scenario (headless + watch:0) + ONE testmatrix cell. Blocked on extracting a facett-korp pane from korp's welded main.rs (single-source P0).