nordisk/znippy-zoomies

osm-katana — reasoning

Class: reasoning · Function: osm-katana convert. Why the OSM → GeoParquet pipeline has exactly two passes with one barrier, and why that barrier is a property of the data model rather than a saturation bug. Shipped shape in osm-katana-design.md.

Why two passes at all

OSM is a reference graph, not a table. A way is a list of node IDs; a node carries the only copy of the coordinate. To emit a way's geometry you must dereference every node ID it names — and those nodes can sit anywhere in the file, before or after the way. So the converter cannot stream ways straight to disk in one linear scan: it must know the whole node index first. That is the entire reason for the split into pass 1 (harvest every node's coordinate) and pass 2 (resolve ways/relations against the harvested index). It is a consequence of the format, the same way gzip's single LZ77 window forces a serial fallback in lgz — the shape is dictated by the data.

Why the barrier between the passes is inherent, not a stall

Between the passes sits the node store build (external_merge_sort, node_store.rs:250), and pass 2 genuinely cannot begin until it finishes: the first way to resolve might reference the numerically-last node in the file, so no way is safe to resolve until the complete sorted index exists. This is a hard happens-before edge, not a scheduling accident. Calling it a "single-core stage" and hunting it as a saturation bug would be a category error.

What we did refuse to accept is a barrier that is also single-threaded. The sort itself fans across cores on gatling (node_store.rs), heaviest chunk first (LPT), so the barrier is a short all-core burst rather than one core grinding while eleven idle. The serial-hunter law applies within the stage even though the stage boundary is serial by nature: a barrier you must have is fine; a barrier that wastes cores while you wait is not.

Why the clip runs inside the workers

Region clip (--clip/--poly) is a per-node geographic test. The naive design runs it as a filtering pass over the coordinates; that is a second linear scan and a second place for one core to bottleneck. Instead the clip is folded into the pass-1 worker at the exact point coordinates are decoded (clip_keeps, lib.rs:401, used reader.rs:277) — every core that decodes a blob also clips it, no extra pass. And when no clip is set the if let Some branch folds away, so the common unclipped convert pays nothing for the feature (lib.rs:100-107).

Why compression lives on the workers, not the writer

The tempting design puts one writer thread at the end that receives rows, builds Arrow row groups, zstd-compresses them, and appends to the parquet file. That is what shipped first, and it pegged pass-2 at ~5 cores on planet: the workers finished their parse and blocked on collector_tx.send while the lone collector sat inside a parquet flush doing the compression for everyone (xml_reader.rs:2025-2037). Compression is the heavy per-row-group cost, and putting it on a single thread converts an all-core stage into a one-core queue.

The fix moves the row-group build and the zstd into the pass-2 workers via per-worker thread_local accumulators feeding a Clone + Sync ColumnEncoder (writer.rs:665). The workers hand the writer already-compressed EncodedRowGroups; the writer's job shrinks to stitching them in order and writing the footer. That is why Stage 4 can stay a single writer without throttling the pipeline — there is no expensive work left on it.

Why the final writer is still serial (and that's correct)

Parquet's file structure is inherently single-writer: row groups append sequentially and the footer indexes their byte offsets, so two threads cannot append concurrently without a lock that serializes them anyway. On top of that we want deterministic row order — the same input must always produce byte-identical output — which requires the collector to deliver segments in index order (Pass1TypedSink / Pass2TypedSink). A serial stitch is the honest shape for both constraints. The few boundary elements that straddle two segments are parsed on the collector via flush_stub (xml_reader.rs:1951 / :2080); this is a handful of rows per segment boundary, negligible next to the workers' bulk. Keeping the stitch cheap (no compression) is what lets it stay serial without becoming the bottleneck — the same discipline as the barrier: unavoidable serial work is fine only when it is also cheap.

Why one engine for four input formats

PBF, bzip2, gzip, and XML look nothing alike on disk, but the converter routes all four through the same gatling::run_typed engine with per-format codecs (PbfPass1TypedCodec, Bz2Pass1TypedCodec, GzPass1TypedCodec, Pass1ResolvedCodec). The engine owns the split/worker/ordered-collector machinery; a codec only teaches it how to find segment boundaries and decode one segment. bzip2 splits at independent block magics, PBF at blob frames (pbf_split makes ~2× workers coarse ranges for load-balance headroom, xml_to_pbf.rs:715-717), gzip at full-flush/bgzf boundaries or one segment if none exist. This is deliberate reuse: the parallel structure is written once and the awkward per-format decode is quarantined in the codec.

Honest saturation note

Both passes are all-core by construction; only the sort barrier and the writer stitch are serial, and both are so by nature. We do not claim a cores_busy number in these docs — the one weighted-LPT path with a concrete balance story (gatling_for_each_balanced, reader.rs:138) serves the separate load_ways_pbf helper, not the convert CLI. Real all-core throughput and peak-RSS numbers for osm-katana convert land in the README.md zoomies-katana mashup once the quiet-box planet run reaches the warehouse — recorded there against osmium / osm2pgsql, not fabricated here.