nordisk/korp-collectors

Collector: osm-cut (design)

The map-data daemon. Keeps every configured region cut out of the OSM planet into the packed source korp's map is actually served from, striving to hold ONLINE. First implementor of the daemon architecture (../collectors-daemon-design.md). Behind --features osm-daemon (korp: collectors-daemon).

It was osm-tiles, and that was legacy

Rickard, 2026-09-02: "the collector osm tiles is legacy? should it not be the new 3 files from osm download?"

Yes. korp's map v2 deleted the tile pyramid — e66f846..1d53b1b removed the tiling gate, the tiled/monolith routes, src/map_tiles.rs (4 980 lines), the S3 tile store and 111.8 GB on disk; korp/tests/no_tiles_law.rs forbids the word in a user-visible string, and the client speaks CELLS. This collector kept cutting a z6/z9/z12 pyramid into a bucket nothing reads, and ~300 references pointed at it.

So it produces what the map IS served from: the lettered chain korp::pipeline publishes.

The letters are the contract between this crate and korp. korp::pipeline::Step and korp_collectors::state::Step are peers (this crate is korp-free by charter), and korp/tests/cut_letters_match_the_leaf.rs fails the build if they ever drift.

Purpose

planet.osm.{pbf,bz2} (declared seed, or torrented) → clip each configured region → drive a..through_step → the cut lands in <work_dir>/<region>/, which is the very directory korp-server verifies at boot (server_regions::verify_map_source).

Config trait

enabled:      bool
regions:      [String]     // multi-select from the region CATALOG (see State/Indexing)
planet:       String       // path to planet-*.osm.{pbf,bz2} (blank ⇒ seed/torrent)
work_dir:     String       // the cut ROOT, one dir per region
                           // (blank ⇒ $KORP_HOME/work — the source korp-server reads)
through_step: String       // a|b|c|d|e|f|g (default g — the whole chain)
clip_mode:    String       // auto (default) | polygon | bbox
poly:         String|{region:String}   // the region's REAL shape (korp passes assets/sweden.poly)
extract_dir:  String|{region:String}   // reuse an existing extract instead of converting
revision:     String       // bump to RE-CUT (part of the declaration)
expect:       {...}        // post-conditions the cut must SATISFY or the run FAILS
keep_generations: u64      // past extracts retained on disk (default 2, oldest pruned)
planet_digest: String      // sidecar (default) | blake3 | size
retry_secs:   u64          // retry cadence while NOT online (default 60)

Edited in ⚙ Settings ▸ Collectors ▸ 🗺 osm-cut. Before the planet exists, regions shows "planet not downloaded yet".

Migration. A live server persisted this collector as collectors."osm-tiles". osm_cut::migrate_collector_name is the ONE rename table; daemon::daemon_by_name and daemon::daemon_with_config route through it, and korp's KorpConfig::load_from rewrites the map key. An operator's enabled flag and region list survive the rename.

State machine (State trait)

Offline → Downloading(planet %) → Indexing(planet → region catalog)
        → Cutting(region, "<letter> <step>") → Online
                                     (Degraded / Failed{reason} reachable off any)

The State Facet renders: the current phase, the region and step letter it is on, the per-region RegionCutReport (one StepReport per letter: state · rows · bytes · wall_ms · note), the driver, the catalog and the last error.

• Online — every configured region complete through through_step, and every post-condition actually asserted something.
• Degraded — a step this build cannot run (Unavailable), or a post-condition that MEASURED but could not ASSERT (geography_asserted). Alive and honest.
• Failed — a convert that died, an unreadable extract, a post-condition that refused the cut. The daemon retries on retry_secs.

Who runs which step (LAW 5: one writer)

• a — acquisition lives HERE (design §9): an already-recorded planet, then a declared seed path (config.seed / seeds, through acquire::LocalSeed), then — under osm-daemon-download — librqbit. No $HOME search.
• b, c — osm_katana::convert, driven with the region's clip. One scan produces both passes; step c reads the artefact step b's convert wrote.
• d, e, f, g — korp's chain (regions::start_or_resume / pipeline), the very one korp-server drives at boot, injected through the CutDriver seam. This crate must never reimplement it: a second chain is the twin LAW 5 forbids, and the two copies would disagree the first time either moved. A build with no driver reports d–g as Unavailable naming the reason — never a skipped step that reads as done.

What it emits

No map RecordBatch. It writes FILES. What it does land, through the ordinary CollectorCtx.warehouse seam, is one summary row per (region, step) into osm_cut_steps:

So the cut shows up in the 🏬 Warehouse like every other collector, and the row says what it is: a receipt for work done, not a re-publication of the map.

Storage

• on disk, the cut — <work_dir>/<region>/: nodes.parquet, ways.parquet, relations.parquet, relations.resolved.parquet, networks/, the packed base table, address.parquet. Replaced file-by-file (never by a directory rename: work/<region> also holds steps e–g, and a directory swap would take the whole chain with it).
• on disk, the collector's own — ~/.korp/collectors/osm-cut/: the planet, the region catalog.json, and collector-state.redb (the completion ledger).
• the bucket seam is unused. CollectorCtx.bucket is present because the trait carries it; this collector PUTs nothing.

Acquisition — the download lives HERE

The librqbit planet fetch (moved out of korp/src/download.rs) is owned by this collector: resumable, caches the planet on disk, drives the Downloading phase's %. It runs inside spawn_blocking so osm-katana's own runtime is top-level (a nested runtime PANICS).

Region catalog (the pick-list)

After the planet exists, osm-katana enumerates its admin-region NAMES (stream_admin_region_names — bounded-memory streaming relation scan, gatling-parallel, NO in-RAM NodeStore) into a cached catalog.json. The cache is keyed on the planet that produced it, so a planet swap re-scans once rather than serving the old roster for ever. A pass with nothing to convert still refreshes the catalog off a planet already at hand — what is skipped is ACQUISITION, not the read.

The CUT is declared, not improvised

cut_declaration = region + clip_mode + the polygon's digest + revision. The same declaration always means the same cut, and changing it is what triggers a re-cut. It is stamped INSIDE the extract (collector-extract.json), beside the planet identity (collector-extract-planet.txt) — so an extract is stale when a NEW planet exists, not when a timer expires.

Post-conditions (expect)

Every one has been seen RED. A cut only returns Ok when all of them passed; a failure is Phase::Failed naming the check and its numbers.

geography_asserted is the marker that catches a pass which measured everything and asserted nothing — no clip declared, or no present expectation. It reports UNASSERTED, and the daemon turns that into Degraded rather than a green.

Safety properties

1. Headroom before the heavy step. /home hit 100 % three times on 2026-08-02.
2. Never a half-swapped extract. The convert writes into <work>/.extract-staging and is published file-by-file only once it reads COMPLETE (both parquet footers).
3. Cooperative stop between stages. A refresh tick can spend 31–52 minutes in a convert; the run gives up before publishing anything.
4. Generations. Each replaced file is archived …gen-NNNN-YYYYMMDD, oldest pruned beyond keep_generations.