.nornir/collectors/dcat.md

Collector: `dcat` (adopted — WAVE 1)

The DCAT-AP-SE discovery daemon. Crawls the whole Swedish public-sector open-data catalogue (dataportal.se / DIGG) into a (dataset, distribution) registry table that drives every downstream bulk/REST fetcher. Adopted to the daemon architecture (../collectors-daemon-design.md, §11) as the first warehouse-sink (Iceberg) daemon. Implementor: src/iceberg_daemons.rs (DcatDaemon), behind --features daemon. Rides the existing pure RDF parser src/dcat.rs::parse_dcat_rdfxml.

Config trait (Settings ▸ Collectors ▸ dcat)

enabled: bool
url:     String   // dataportal.se RDF/XML dump   (default https://admin.dataportal.se/all.rdf)
mode:    String   // rdf | sparql
table:   String   // Iceberg table                (default dcat_registry)

State machine (State trait)

Offline → Downloading(fetch %) → Indexing(parse RDF) → Online     // registry rows landed
        → Failed{reason}                                          // empty-is-RED / fetch/parse error

The State Facet renders: sink=iceberg, phase, table, rows_written, last_run_unix, error. Green in the infra matrix only when ≥1 registry row is landed.

Storage — Iceberg (warehouse-sink)

Rows land through CollectorCtx.warehouse (WarehouseSink) into dcat_registry (korp backs the seam with a skade Table / BatchSink::iceberg; tests with MemWarehouse). No S3 bucket — the S3 seam is unused by warehouse-sink collectors.

Acquisition — the fetch seam

The nightly RDF/XML dump is fetched through CollectorCtx.fetch (korp = the shared rustls HTTP core; tests = a canned Fetcher::from_map fixture, so the run-loop is proven with no live gov API). Fetch + parse both run off the async reactor via spawn_blocking (a multi-MB dump is CPU-bound — zero cpu-wait on the loop).

empty-is-RED

An enabled crawl that parses to 0 datasets is a loud Failed{reason:"…(empty-is-RED)"} (red in the matrix), never a silent green.

Tests (`src/iceberg_daemons.rs`)

dcat_daemon_reaches_online_and_sinks_registry — the synthetic DCAT-AP-SE RDF (2 datasets) drives fetch→parse→append→Online, ≥2 rows in dcat_registry, BIOS tuned in.
empty_source_is_red_not_silently_green — an RDF doc with no dcat:Dataset ⇒ Failed/red.
shared registry / schema / stoppable-loop tests.

Follow-on (mechanical)

Live mode=sparql (EU data.europa.eu) reuses dcat.rs::parse_sparql_results behind the same seam; CDC/incremental marks (only-new datasets) are the knut-bifrost checkpoint reuse.

Collector: dcat (adopted — WAVE 1)