Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/znippy-zoomies

.nornir/osm-katana-design.md

osm-katana — design

Class: design · Function: osm-katana convert (OSM → GeoParquet). Strict: the shape we ship, every claim carrying a real file:symbol. How to run it in osm-katana-guide.md; why the pipeline has the shape it does in osm-katana-reasoning.md. The streaming engine is gatling-design.md.

What it is

osm-katana convert turns an OSM extract — XML (.osm), gzip (.osm.gz), bzip2 (.osm.bz2), or PBF (.osm.pbf) — into GeoParquet: three files, nodes.parquet, ways.parquet, relations.parquet, all-core on the shared gatling engine. The CLI handler is Cmd::Convert (osm-katana/src/bin/main.rs:183), which builds a ConvertOptions and calls osm_katana::convert (main.rs:193). The library entry is convert (osm-katana/src/lib.rs:483) → convert_inner (lib.rs:508), which dispatches by input extension and geometry mode (raw vs the default resolved) to a reader in osm-katana/src/xml_reader.rs:

input reader (xml_reader.rs)
.osm.pbf read_resolved_pbf (:2442)
.osm.bz2 read_resolved_bz2 (:2196)
.osm.gz read_resolved_gz (:2337)
.osm (XML) read_resolved_osm (:437)

Default worker count is available_parallelism() − 1 (lib.rs:488).

The pipeline: two gatling passes, one serial barrier

The convert is two gatling::run_typed passes over the same input with a serial sort barrier between them. gatling::run_typed(reader, codec, sink, n_workers, cfg) (gatling/src/gatling.rs) is the streaming typed engine: it splits the byte stream, each worker decodes + parses + encodes its own segment in parallel, and the TypedSink collector delivers segments in order on one thread. The four stages below map one-to-one onto the SVG above.

Stage 1 — Pass 1: nodes + coordinates (parallel, per worker)

Pass 1 extracts node records and their (id, lat, lon) coordinates. The PBF codec is PbfPass1TypedCodec (xml_reader.rs:1869); each worker runs pass1_parse_blob (xml_reader.rs:1579) → reader::blob_nodes_and_coords (reader.rs:60), which decompresses the blob with pbf_decomp::decompress (pbf_decomp.rs:23, a per-thread ZSTD DCtx held thread_local, pbf_decomp.rs:123) and delta-decodes dense + regular nodes via reader::parse_nodes (reader.rs:235).

The region clip (--clip/--poly) is applied inside the pass-1 workers at coordinate decode — clip_keeps (lib.rs:401, used at reader.rs:277) — so there is no extra serial scan (see the comment at lib.rs:100-107: "every core that decodes a blob also clips it").

A relation carries no coordinates, so the clip could not test it and every relation of the planet passed through (14 632 472 rows, 1.05 GB, on a Sweden cut whose resolve stage keeps 274 206 — measured 2026-08-31). Since 2026-09-01 every reader answers that with the relation clip prefilter: with a clip active, pass 2 is two sub-passes over the input (xml_reader::run_pass2, the one orchestration). The way sub-pass reports the ids of the ways it kept; those ids plus the node store are a reader::RelationClipOracle, and the relation sub-pass — opened where the relations start: the FRAME of the first relation blob for PBF (BlobPlan::first_rel_blob), the first <relation for .osm, the last <way block for .osm.bz2 (Bz2Skip::Ways), byte 0 for .osm.gz — keeps a relation when ANY member survived: a kept way, a node in the store, or (settled by a fixpoint on the collector, RelationClipOracle::settle) a relation that was itself kept. That is the admission rule relgeom::resolve_relations already applies, so the prefilter is a strict superset of what resolves. Without a clip there is one pass and every relation is written, byte-identical to before. The plain .osm reader lost its serial pass-2 sink for this: it runs the same Pass2TypedSink as the other three. The measured effect is in osm-katana-v2-design.md.

The split and call sites differ per format but keep the same shape:

The pass-1 collector Pass1TypedSink::process (xml_reader.rs:1999) streams packed 16-byte (id, lat, lon) records to node_coords.bin via CoordFileWriter::write_raw (node_store.rs:213).

Stage 2 — node-id → coordinate store (serial barrier, internally multithreaded)

After pass 1, store = coord_writer.finalize() (xml_reader.rs:2495 PBF / :2254 bz2 / :2390 gz). CoordFileWriter::finalize (node_store.rs:224) calls external_merge_sort (node_store.rs:250) — a sample sort — and writes an mmap'd Arrow-IPC NodeStore.

This is a hard barrier: way node-ref resolution (store.lookup_batch, reader.rs:131 / :432) must see the complete sorted node index before any way can resolve, so pass 2 cannot begin until the store is built. It is serial between the passes but not a single-core stage — the sort runs multithreaded on gatling (node_store.rs, heaviest-chunk-first sample sort / psort; the merge is gatling_for_each_balanced over the partitions and the STree layer fill is gatling_scanlines). It is logged as phase sort_and_stree (xml_reader.rs:2494, note "psort-samplesort").

Stage 3 — assemble geometry + encode Arrow (parallel, pass-2 workers)

Pass 2 resolves refs and builds the way/relation geometry. Codec PbfPass2TypedCodec (xml_reader.rs:1895); each worker runs pass2_parse_blob (xml_reader.rs:1634) → blob_ways_rels (reader.rs:70) → parse_ways_rels (reader.rs:335), resolving refs against the store, then encodes WKB geometry on the worker via CpuBackend.encode_wkb (reader.rs:86) — points through geometry::encode_point_inline (geometry.rs:11), area detection through geometry::is_closed_area (geometry.rs:21).

The Arrow row-group build and the zstd of full row groups also happen on the worker, through per-worker thread_local accumulators (WAY_ACC / REL_ACC / NODE_ACC, xml_reader.rs:1595) feeding writer::ColumnEncoder::encode (writer.rs:665) → EncodedRowGroup (writer.rs:648). The call site is the second gatling::run_typed (xml_reader.rs:2519 PBF / :2285 bz2 / :2415 gz). ColumnEncoder is Clone + Sync and is shared to the workers by build_pass2_sink (xml_reader.rs:2130).

Pushing compression into the workers was a real fix: the comment at xml_reader.rs:2025-2037 records that the collector-side flush had "pegged pass-2 at ~5 cores on planet … workers blocked on collector_tx.send while collector was inside parquet flush."

Stage 4 — write GeoParquet (serial in-order stitch, no compression on this thread)

There is one writer per table, but it only stitches the workers' pre-compressed row groups in index order: ParallelNodeWriter::append (writer.rs:692) / finish (writer.rs:702), built by parallel_node_writer (writer.rs:714), parallel_way_writer (writer.rs:748), parallel_rel_writer (writer.rs:767). Each sets the GeoParquet geo metadata (metadata::geo_metadata) and schema, and splits an ArrowWriter into the low-level SerializedFileWriter (into_serialized_writer, writer.rs:727).

The collector sinks Pass1TypedSink / Pass2TypedSink deliver segments in order, giving deterministic row order. Boundary elements that straddle two segments are parsed serially on the collector via flush_stub (xml_reader.rs:1951 node / :2080 way/rel) — tiny, serial by design. The stitch is serial because a Parquet append + footer must be single-writer with deterministic row-group order; it is kept off the hot path by having done all compression in the workers.

Correctness

The GeoParquet geo metadata and schema are written by the parallel_*_writer builders so each of the three files is a valid GeoParquet table. --geometry raw is rejected for PBF and for --clip/--poly (lib.rs:516, :522) — clip is a node-membership test that needs resolved coordinates.

What "the same output" means here — and how you check it

The output is reproducible as a row multiset, not as bytes, and the pipeline is built that way on purpose:

So sha256 nodes.parquet can never be the equivalence gate — it differs between two runs of the same binary. The gate is digest.rs, exposed as osm-katana verify --digest: each row is canonically encoded (presence tag + raw value bytes, per column in schema order) and FNV-1a hashed, and the per-row hashes are folded two ways over the Mersenne prime 2^61 − 1 —

column fold invariant to catches
ord positional polynomial, H(A ++ B) = H(A)·B^{len B} + H(B) row-group layout value change, drop, insert, reorder
set plain sum row-group layout and row order value change, drop, insert, duplicate

set is the gate. Because the combine rule needs only the right-hand run's row count, each row group is hashed on its own core (gatling_for_each_balanced over the row-group indices, LPT by compressed size) and the partials folded back — the digest is a parallel read, not a serial one.

osm-katana convert in.osm.pbf -o out-a && osm-katana verify out-a --digest
osm-katana convert in.osm.pbf -o out-b && osm-katana verify out-b --digest   # set must match

node_coords.arrow is byte-identical run-to-run (the merge writes fixed offsets), so cmp is the right check for it.

Whereabouts

Thing file:symbol
CLI convert handler osm-katana/src/bin/main.rs:183 (Cmd::Convert)
library entry / dispatch osm-katana/src/lib.rs:483 (convert) → :508 (convert_inner)
worker count default osm-katana/src/lib.rs:488 (cores − 1)
PBF reader osm-katana/src/xml_reader.rs:2442 (read_resolved_pbf)
bz2 / gz / XML readers xml_reader.rs:2196 / :2337 / :437
pass-1 worker transform xml_reader.rs:1579 (pass1_parse_blob)
pass-1 codec / sink xml_reader.rs:1869 (PbfPass1TypedCodec) / :1999 (Pass1TypedSink::process)
node decode / clip reader.rs:60, :235; lib.rs:401 (clip_keeps), used reader.rs:277
PBF split xml_to_pbf.rs:707 (pbf_split), engine :739/:751
coord file write node_store.rs:213 (CoordFileWriter::write_raw)
sort barrier node_store.rs:224 (finalize) → :250 (external_merge_sort) → psort
pass-2 worker transform xml_reader.rs:1634 (pass2_parse_blob)
ref resolve / WKB reader.rs:335 (parse_ways_rels), :86 (encode_wkb); geometry.rs:11/:21
worker Arrow encode writer.rs:665 (ColumnEncoder::encode) → :648 (EncodedRowGroup)
pass-2 sink build xml_reader.rs:2130 (build_pass2_sink)
in-order writer stitch writer.rs:692 (append) / :702 (finish); builders :714/:748/:767
boundary stub (serial) xml_reader.rs:1951 / :2080 (flush_stub)
tag JSON (canonical, sorted) tags.rs:19 (serialize_tags)
content digest / equivalence gate digest.rs:digest_file · CLI verify --digest
census of tables already on disk census.rs:census · CLI census / census --pyramid
the census closure identity census.rs:Partition::unaccounted / :balances
a percentage that carries its denominator census.rs:Ratio (the ONLY door to a %)
phase log + per-phase occupancy phase_log.rs:Phase · StopFlag
optimize row-group fan-out optimize.rs:gatling_collect
coord merge fan-out node_store.rs:merge_sorted_chunks_to_arrow
GeoParquet → Arrow IPC geo2arrow.rs:geo2arrow (ordered::run_ordered_sink)
verify row-group counters verify.rs:count_node_refs / :count_geometry
search-index row-group fan-out side_outputs/search.rs:gatling_features
the ONE sanctioned private thread xml_to_pbf.rs:blob_writer_loop (single blocking-I/O sink)

ROOT LAW #0 status (2026-07-22)

osm-katana holds zero std::thread::scope worker pools. Every fan-out in the crate — both convert passes, the coord merge and its reference twin, the parquet→Arrow converter, the verify counters, the search-index build, the optimize collector, and the row-group encode in writer.rs's tests — routes through gatling. The crate has no entries left in SCOPED_POOL_ALLOW (tests/rayon_free_law.rs), so a scope pool anywhere under osm-katana/src now trips the law test with no exception to hide behind.

The single exception is xml_to_pbf.rs's blob_writer_loop: ONE named, long-lived thread doing blocking file I/O off the gatling collector, fed by a bounded mpsc channel. That is IO-offload, not CPU fan-out — there is exactly one of it, it never scales with core count, it claims no units from a shared cursor, and it does no parse/encode. It is licensed by PRIVATE_POOL_ALLOW and commented at the call site.

Measuring it — per-phase core occupancy, not just wall time

Wall time cannot tell you which phase drops the box. Every phase opens a phase_log::Phase (phase_log.rs), whose done event carries the phase's REAL occupancy: cpu_cores (process CPU-seconds / wall-seconds over that phase's window), n_cores, busy_pct = cpu_cores / n_cores, and collector_pct (the fraction of wall the single collector spent inside process()). Phases run sequentially over one process-wide CPU counter, so this attributes occupancy per phase with no per-thread accounting. Read them with --log <file.jsonl> — or let the bench do it: examples/nornir-bench.rs parses that log back and publishes one row per phase (osm-katana_pass1 / _sort_and_stree / _pass2 / _convert).

Two signatures to read for:

The phase guard's teardown used to be part of the problem: the sampler thread only noticed the stop flag between 100 ms sleeps, and done() joins it before reading the clock, so every phase boundary charged up to 100 ms of dead time with the whole pool idle — four times per convert. StopFlag (Mutex + Condvar) makes that immediate.

Saturation state

Both passes fan across every core on gatling; the sort barrier (Stage 2) and the single-writer parquet stitch (Stage 4) are serial-by-design and exempt. The LPT weighting seen in the separate read_pass2_mmap mmap API — gatling_for_each_balanced weighted by blob byte length (reader.rs:138, comment reader.rs:130-136) — serves the load_ways_pbf helper, not the convert CLI (which uses run_typed). The SlotFill::Big choice (xml_to_pbf.rs:728-734) is what keeps slot turnover from starving the workers on planet-scale PBF. This doc records no fabricated cores_busy: honest all-core numbers land in the README.md zoomies-katana mashup on the next heavy-bench wave (the planet run); the measured per-phase occupancy on a 319 MB PBF is in benchmarks.md.

Known remaining drains (measured, not guessed — see benchmarks.md):

  1. sort_and_stree is the phase barrier and holds only ~4 of 32 cores: pass 1 must be fully drained before the tail chunk can be sorted, and pass 2 cannot start until the STree exists.
  2. Pass 1 is collector-bound — collector_pct ≈ 50 %. The single in-order sink copies every packed coord buffer into the sorter and writes the whole nodes.parquet (565 MB + 533 MB on one thread for the 319 MB input). Measured: the coord copy is not the cost — it already runs at memory bandwidth (3.83 GB/s; a hand-written memcpy was 0.5 % faster and was reverted, see benchmarks.md). The remaining ~50 % is the parquet row-group stitch + footer metadata, which is single-writer by format.
  3. Pass 2 re-reads and re-inflates every blob, including the node-only blobs it will find nothing in. Pass 1 already knows which blobs carry ways/relations; passing that set forward would let pass 2 skip them outright. Not attempted — the streaming codec's transform is handed a slot-relative slice and does not know the blob's absolute file offset, so the map has no key yet.

The pyramid stopped carrying what a reader cannot see (2026-08-31)

Three findings from a full Sweden cut on oden, each measured before it was acted on.

K16 — an untagged node is a vertex, not a feature

nodes.parquet was placed into the pyramid in full: nodes scanned: 103 488 583 · placed 103 488 583. In OpenStreetMap the overwhelming majority of nodes exist only to give a way its shape, and the ways that use them are already in the pyramid (or in a network table), so the vertex adds nothing a reader can see and nothing a classifier can name.

MEASURED whole-country, from the run's own scan ledger (no sampling):

SCAN LEDGER nodes: file 103 488 583 = placed 2 406 592 + untagged nodes 101 081 991
                   + NULL geometry 0 + undecodable 0 + no geometry extent 0
                   · rows read 103 488 583 · UNACCOUNTED 0

101 081 991 of 103 488 583 nodes — 97.7 % — carried no tags at all.

A correction, and how the wrong number was produced

An earlier revision of this section (and the commit that landed it) claimed "107 299 of 188 112 rows (57.0 %) over all nineteen z6 tiles". That figure is wrong, and the error is worth recording because it is invisible in the output. The census flattened the pyramid's paths when it copied the tiles — cp z6/<x>/<y>.parquet into one directory — so z6/33/15.parquet and z6/34/15.parquet overwrote each other and 7 of the 19 tiles survived. The run printed 7 and the table above it still said "nineteen".

A sample that silently loses two thirds of its population still produces a clean-looking percentage. The replacement number is taken from the tiler's own ledger, which counts every row of every file by construction and closes to UNACCOUNTED 0 — there is nothing to sample and nothing to get wrong.

korp's own map_networks module header had already measured the same thing from the other side — removing ways from the [6,9,12] pyramid shrank it by "only 5.54 % of bytes … because nodes dominate at those zooms".

The skip is ScanLedger::untagged_nodes, a ledger column and not a silent continue, because a row that leaves has to leave through a name — that is the whole of K15.

Two things it must not do, both guarded:

K18 — the fan-out cap measured tiles, and the cost is bytes

MAX_WAY_TILE_FANOUT is a fixed count of TILES, and a tile is a fixed fraction of the world — so the ground area it permits shrinks 4× per zoom step: ≈ 620 × 620 km at z12, ≈ 40 × 40 km at z15. The consequence is that the same feature is accepted at one zoom and refused at the next for no reason that has anything to do with the feature. MEASURED on the 2026-08-30 cut: z12 = 4 capped, z15 = 455 — coastline, river and railway runs placed in their CENTROID TILE ONLY, so the shore is served in one of the many tiles it crosses and vanishes exactly when a reader zooms in far enough to want it.

What the cap protects is the staging buffer, and that is measured in bytes. A long feature is cheap to duplicate because it is long and simple — Douglas–Peucker has already thinned a coastline run at z15 to a handful of vertices — while a dense polygon is expensive whatever its span. fanout_budget_tiles(per) = MAX_WAY_FANOUT_BYTES / per, clamped:

K17 — a ratio whose denominator is the wrong set is a different claim

The relation probe printed 274 206 rows, 225 799 carry assembled geometry (82.3 %), which reads as one relation in six failed and cost a reader an afternoon. Only an AREA relation is ever GIVEN geometry (RelationKind::Area): a type=route is an ordered set of ways, a type=restriction is a rule, a type=public_transport is a stop group. MEASURED on that same file: 226 967 area relations, 225 799 assembled = 99.49 %; the 48 407 without geometry are 14 588 routes, 14 412 stop areas, 5 633 turn restrictions, 2 463 waterway relations and so on. The real failure set is the ~1 324 areas that did not assemble (0.58 %).

The probe does not classify relations, so it now prints no ratio at all and names both populations instead. A percentage it cannot compute honestly is worse than none.

The geometry column says what it is (2026-08-31)

Every parquet this crate writes carried WKB in a bare BYTE_ARRAY. A reader learned it was geometry only by parsing the GeoParquet geo key in the footer — an out-of-band JSON convention, not the schema. Parquet's own GEOMETRY logical type exists now, and parquet 59.2.0 implements both ends of it, so the column can carry its own type.

shared::schema::geometry_field() is the ONE writer (LAW 5). It returns the field with the parquet_geospatial::WkbType arrow extension type, which is the only door through which ArrowWriter emits LogicalType::Geometry and runs the writer-side bounding-box accumulator that fills ColumnChunkMetaData::geo_statistics. Five production schema builders route through it — nodes_schema, ways_schema, relations_schema, tiles::tile_fields, network's table schema, graph's two, and optimize's repack.

parquet's geospatial feature is not on by default; a build without the feature line in Cargo.toml writes the old bare column and nothing complains.

What it buys, and what it does not

MEASURED 2026-08-31, geometry_columns_carry_the_logical_type, both arms on identical bytes: the declared field comes back Geometry(GeometryType { crs: Some("srid:0") }) with geo_statistics present and the bbox x (10.594, 24.178) · y (55.137, 69.06). The bare field comes back logical type: None, geo_statistics: NONE.

It cannot force a re-cut

TileRecipe's columns term is built from f.name() and short_type(f.data_type()), and an extension type is field METADATA that changes neither. Were that ever to stop being true, every cut pyramid on every box would be judged Stale — 1 635 971 tiles re-cut for a metadata edit. the_logical_type_does_not_move_the_recipe pins the name, the DataType, and that the field carries metadata at all, so the test cannot pass by the field being plain. Both guards seen RED with the bare field restored.

census — asking a question of tables that already exist (2026-08-31)

The 57.0 % correction above records what went wrong. This verb records why it could: there was no way to ask a question about a pyramid on disk except by hand-rolling a script, and the script that got hand-rolled lost two thirds of its population to a cp.

ScanLedger cannot be sampled wrong — but it only exists inside a tiling run. Once the tiles are written, the discipline was gone. census.rs is that discipline, applied to files that already exist:

osm-katana census ~/.korp/work/sweden                        # the four extract tables
osm-katana census ~/.korp/work/sweden/tiles --pyramid        # the pyramid, footer only
osm-katana census ~/.korp/work/sweden/tiles --pyramid \
    --tags --geometry --classes --json census.json

The property that makes it worth having

Every population sums to the file's own footer num_rows, and the books close. A Partition carries its denominator, Partition::unaccounted() is printed on every line, and a partition that does not balance prints no percentage at all and exits non-zero. A ratio over books that lost rows is precisely the machine that produced the wrong number.

Three further properties, each a direct answer to how the 57.0 % happened:

Every gate was seen RED

Five mutations of the shipped code, each killing exactly the tests it should and no others (tests/census_books_close.rs, 9 tests):

mutation tests killed
unaccounted() always returns 0 the three closure gates
note_nonconforming never increments the flattened-pyramid gate
Extent::intersects always true the misplaced-tile gate
the non-vacuity check removed the zero-files gate
a missing tags column fails CLOSED the fail-open gate

Sabotage::{DropTagged, DropPoints, InflateTotal} drop or mis-attribute a population inside the real counting loop, so the red arm travels the same code the green one does. It is reachable from a shell: OSM_KATANA_CENSUS_SABOTAGE=drop-tagged, which on the live z6 prints UNACCOUNTED 343937, suppresses every % on the line, and exits 1.

What it costs — MEASURED on oden, 2026-08-31, 32 cores

The live Sweden pyramid, /home/korp/.korp/work/sweden/tiles, page cache warm (korp-server was serving the same objects):

run files wall what was read
--pyramid 1 635 971 11.4 s footers only — no data page
--pyramid --tags 1 635 971 11.3 s + the tags column (646 MB compressed)
--pyramid --tags --geometry 1 635 971 23.7 s + geometry (34.53 GB compressed)
tables, --tags 4 3.5 s 127 720 403 rows

Seconds, not minutes, and the difference between the first two rows is the whole design: rows, bytes, row groups, column bytes and covering extents need no data page at all.

Fan-out is gatling (ROOT LAW #0), one unit per x directory, and each worker folds straight into its own GroupCensus. The pyramid is 1 635 971 files under 1 417 x directories; a Vec<PathBuf> over the leaves would be ~100 MB and the largest allocation in a process whose whole job is to hold nothing. 1 417 units over 32 cores is ~44 units each, so the longest single unit is a small fraction of a worker's total and gatling's shared cursor absorbs the z6-vs-z15 spread without a weight function.

What the first run found

MEASURED 2026-08-31, on the pyramid written at 06:15–06:17 UTC that day (mtimes from the census's own JSON), recipe token identical on all 1 635 971 files — one cut, not a mix:

ALL  files 1 635 971 · bytes 44 449 184 179 (44.45 GB) · rows 33 357 988
     geometry: LineString 9 052 208 (27.1 %) + MultiPolygon 16 961 280 (50.8 %)
             + Point 7 344 500 (22.0 %)                        · UNACCOUNTED 0
     tags:     tagged 28 959 168 (86.8 %) + untagged `{}` 4 398 820 (13.2 %) · UNACCOUNTED 0
     geometry × tags: every one of the 4 398 820 untagged rows is a LineString —
                      0 untagged Points, 0 untagged MultiPolygons.

Two figures reproduced exactly, which is what a validation is for:

⚠ A figure in osm-katana-v2-design.md that this pyramid does not support

That document quotes "sub-pixel areas shipped as Point (11 101 365 of 12 628 347 area placements, 87.9 %)", beside a centroid-only figure explicitly updated for the 08-31 recut. The pyramid that recut produced holds 7 344 500 Point rows in total, across all four zooms — 3 756 865 fewer than the claim needs, before any of them is asked whether it is an area. The claim cannot be true of the tables now on disk.

Which of the two is wrong is not determinable from the tables, and that is the point: the 87.9 % carries no date, no file list and no recipe token, so there is nothing to check it against. Every figure this verb prints ships with all three in its JSON.