osm-katana — design
Class:
design· Function:osm-katana convert(OSM → GeoParquet). Strict: the shape we ship, every claim carrying a realfile:symbol. How to run it inosm-katana-guide.md; why the pipeline has the shape it does inosm-katana-reasoning.md. The streaming engine isgatling-design.md.
What it is
osm-katana convert turns an OSM extract — XML (.osm), gzip (.osm.gz), bzip2 (.osm.bz2), or PBF (.osm.pbf) — into GeoParquet: three files, nodes.parquet, ways.parquet, relations.parquet, all-core on the shared gatling engine. The CLI handler is Cmd::Convert (osm-katana/src/bin/main.rs:183), which builds a ConvertOptions and calls osm_katana::convert (main.rs:193). The library entry is convert (osm-katana/src/lib.rs:483) → convert_inner (lib.rs:508), which dispatches by input extension and geometry mode (raw vs the default resolved) to a reader in osm-katana/src/xml_reader.rs:
| input | reader (xml_reader.rs) |
.osm.pbf |
read_resolved_pbf (:2442) |
.osm.bz2 |
read_resolved_bz2 (:2196) |
.osm.gz |
read_resolved_gz (:2337) |
.osm (XML) |
read_resolved_osm (:437) |
Default worker count is available_parallelism() − 1 (lib.rs:488).
The pipeline: two gatling passes, one serial barrier
The convert is two gatling::run_typed passes over the same input with a serial sort barrier between them. gatling::run_typed(reader, codec, sink, n_workers, cfg) (gatling/src/gatling.rs) is the streaming typed engine: it splits the byte stream, each worker decodes + parses + encodes its own segment in parallel, and the TypedSink collector delivers segments in order on one thread. The four stages below map one-to-one onto the SVG above.
Stage 1 — Pass 1: nodes + coordinates (parallel, per worker)
Pass 1 extracts node records and their (id, lat, lon) coordinates. The PBF codec is PbfPass1TypedCodec (xml_reader.rs:1869); each worker runs pass1_parse_blob (xml_reader.rs:1579) → reader::blob_nodes_and_coords (reader.rs:60), which decompresses the blob with pbf_decomp::decompress (pbf_decomp.rs:23, a per-thread ZSTD DCtx held thread_local, pbf_decomp.rs:123) and delta-decodes dense + regular nodes via reader::parse_nodes (reader.rs:235).
The region clip (--clip/--poly) is applied inside the pass-1 workers at coordinate decode — clip_keeps (lib.rs:401, used at reader.rs:277) — so there is no extra serial scan (see the comment at lib.rs:100-107: "every core that decodes a blob also clips it").
A relation carries no coordinates, so the clip could not test it and every relation of the planet passed through (14 632 472 rows, 1.05 GB, on a Sweden cut whose resolve stage keeps 274 206 — measured 2026-08-31). Since 2026-09-01 every reader answers that with the relation clip prefilter: with a clip active, pass 2 is two sub-passes over the input (xml_reader::run_pass2, the one orchestration). The way sub-pass reports the ids of the ways it kept; those ids plus the node store are a reader::RelationClipOracle, and the relation sub-pass — opened where the relations start: the FRAME of the first relation blob for PBF (BlobPlan::first_rel_blob), the first <relation for .osm, the last <way block for .osm.bz2 (Bz2Skip::Ways), byte 0 for .osm.gz — keeps a relation when ANY member survived: a kept way, a node in the store, or (settled by a fixpoint on the collector, RelationClipOracle::settle) a relation that was itself kept. That is the admission rule relgeom::resolve_relations already applies, so the prefilter is a strict superset of what resolves. Without a clip there is one pass and every relation is written, byte-identical to before. The plain .osm reader lost its serial pass-2 sink for this: it runs the same Pass2TypedSink as the other three. The measured effect is in osm-katana-v2-design.md.
The split and call sites differ per format but keep the same shape:
- PBF → wrapper
xml_to_pbf::pbf_gatling_run_typed(xml_to_pbf.rs:739, bodygatling::run_typedatxml_to_pbf.rs:751), invoked atxml_reader.rs:2480. The splitpbf_split(xml_to_pbf.rs:707) makes ~2×n_workerscoarse byte ranges (commentxml_to_pbf.rs:715-717); each worker then aligns to blob frame boundaries viapbf_io::align_and_collect. - bz2 → direct
gatling::run_typed(xml_reader.rs:2239), codecBz2Pass1TypedCodec(xml_reader.rs:1724): the split finds independent bzip2 block boundaries, workers decompress → XML, then VTD-parse. - gz →
xml_to_pbf::gz_gatling_run_typed(xml_to_pbf.rs:679), codecGzPass1TypedCodec(xml_reader.rs:1763): parallel DEFLATE where full-flush / bgzf boundaries exist, otherwise a single segment. - .osm XML → codec
Pass1ResolvedCodec(xml_reader.rs:478).
The pass-1 collector Pass1TypedSink::process (xml_reader.rs:1999) streams packed 16-byte (id, lat, lon) records to node_coords.bin via CoordFileWriter::write_raw (node_store.rs:213).
Stage 2 — node-id → coordinate store (serial barrier, internally multithreaded)
After pass 1, store = coord_writer.finalize() (xml_reader.rs:2495 PBF / :2254 bz2 / :2390 gz). CoordFileWriter::finalize (node_store.rs:224) calls external_merge_sort (node_store.rs:250) — a sample sort — and writes an mmap'd Arrow-IPC NodeStore.
This is a hard barrier: way node-ref resolution (store.lookup_batch, reader.rs:131 / :432) must see the complete sorted node index before any way can resolve, so pass 2 cannot begin until the store is built. It is serial between the passes but not a single-core stage — the sort runs multithreaded on gatling (node_store.rs, heaviest-chunk-first sample sort / psort; the merge is gatling_for_each_balanced over the partitions and the STree layer fill is gatling_scanlines). It is logged as phase sort_and_stree (xml_reader.rs:2494, note "psort-samplesort").
Stage 3 — assemble geometry + encode Arrow (parallel, pass-2 workers)
Pass 2 resolves refs and builds the way/relation geometry. Codec PbfPass2TypedCodec (xml_reader.rs:1895); each worker runs pass2_parse_blob (xml_reader.rs:1634) → blob_ways_rels (reader.rs:70) → parse_ways_rels (reader.rs:335), resolving refs against the store, then encodes WKB geometry on the worker via CpuBackend.encode_wkb (reader.rs:86) — points through geometry::encode_point_inline (geometry.rs:11), area detection through geometry::is_closed_area (geometry.rs:21).
The Arrow row-group build and the zstd of full row groups also happen on the worker, through per-worker thread_local accumulators (WAY_ACC / REL_ACC / NODE_ACC, xml_reader.rs:1595) feeding writer::ColumnEncoder::encode (writer.rs:665) → EncodedRowGroup (writer.rs:648). The call site is the second gatling::run_typed (xml_reader.rs:2519 PBF / :2285 bz2 / :2415 gz). ColumnEncoder is Clone + Sync and is shared to the workers by build_pass2_sink (xml_reader.rs:2130).
Pushing compression into the workers was a real fix: the comment at xml_reader.rs:2025-2037 records that the collector-side flush had "pegged pass-2 at ~5 cores on planet … workers blocked on collector_tx.send while collector was inside parquet flush."
Stage 4 — write GeoParquet (serial in-order stitch, no compression on this thread)
There is one writer per table, but it only stitches the workers' pre-compressed row groups in index order: ParallelNodeWriter::append (writer.rs:692) / finish (writer.rs:702), built by parallel_node_writer (writer.rs:714), parallel_way_writer (writer.rs:748), parallel_rel_writer (writer.rs:767). Each sets the GeoParquet geo metadata (metadata::geo_metadata) and schema, and splits an ArrowWriter into the low-level SerializedFileWriter (into_serialized_writer, writer.rs:727).
The collector sinks Pass1TypedSink / Pass2TypedSink deliver segments in order, giving deterministic row order. Boundary elements that straddle two segments are parsed serially on the collector via flush_stub (xml_reader.rs:1951 node / :2080 way/rel) — tiny, serial by design. The stitch is serial because a Parquet append + footer must be single-writer with deterministic row-group order; it is kept off the hot path by having done all compression in the workers.
Correctness
The GeoParquet geo metadata and schema are written by the parallel_*_writer builders so each of the three files is a valid GeoParquet table. --geometry raw is rejected for PBF and for --clip/--poly (lib.rs:516, :522) — clip is a node-membership test that needs resolved coordinates.
What "the same output" means here — and how you check it
The output is reproducible as a row multiset, not as bytes, and the pipeline is built that way on purpose:
- Row VALUES are reproducible.
tags::serialize_tags(tags.rs:19) collects into aBTreeMap, so thetagsJSON has a canonical sorted key order. It used to be astd::collections::HashMap, whoseRandomStateorder is seeded per process — two converts of the same input produced the same tag set in a different order, so every row differed. Sorted keys also pack better under the parquet zstd dictionary. - Row-GROUP layout is not reproducible. Row groups are Arrow-built + zstd-encoded on the workers (
writer.rs:665) and stitched by the collector, so their number and boundaries follow whatever segmentation that run produced. - Row ORDER is not reproducible either. At end of pass each worker flushes its partial row-group accumulator through
TypedCodec::finish_worker, and those tail batches are appended in worker-completion order.
So sha256 nodes.parquet can never be the equivalence gate — it differs between two runs of the same binary. The gate is digest.rs, exposed as osm-katana verify --digest: each row is canonically encoded (presence tag + raw value bytes, per column in schema order) and FNV-1a hashed, and the per-row hashes are folded two ways over the Mersenne prime 2^61 − 1 —
| column | fold | invariant to | catches |
ord |
positional polynomial, H(A ++ B) = H(A)·B^{len B} + H(B) |
row-group layout | value change, drop, insert, reorder |
set |
plain sum | row-group layout and row order | value change, drop, insert, duplicate |
set is the gate. Because the combine rule needs only the right-hand run's row count, each row group is hashed on its own core (gatling_for_each_balanced over the row-group indices, LPT by compressed size) and the partials folded back — the digest is a parallel read, not a serial one.
osm-katana convert in.osm.pbf -o out-a && osm-katana verify out-a --digest
osm-katana convert in.osm.pbf -o out-b && osm-katana verify out-b --digest # set must match
node_coords.arrow is byte-identical run-to-run (the merge writes fixed offsets), so cmp is the right check for it.
Whereabouts
| Thing | file:symbol |
| CLI convert handler | osm-katana/src/bin/main.rs:183 (Cmd::Convert) |
| library entry / dispatch | osm-katana/src/lib.rs:483 (convert) → :508 (convert_inner) |
| worker count default | osm-katana/src/lib.rs:488 (cores − 1) |
| PBF reader | osm-katana/src/xml_reader.rs:2442 (read_resolved_pbf) |
| bz2 / gz / XML readers | xml_reader.rs:2196 / :2337 / :437 |
| pass-1 worker transform | xml_reader.rs:1579 (pass1_parse_blob) |
| pass-1 codec / sink | xml_reader.rs:1869 (PbfPass1TypedCodec) / :1999 (Pass1TypedSink::process) |
| node decode / clip | reader.rs:60, :235; lib.rs:401 (clip_keeps), used reader.rs:277 |
| PBF split | xml_to_pbf.rs:707 (pbf_split), engine :739/:751 |
| coord file write | node_store.rs:213 (CoordFileWriter::write_raw) |
| sort barrier | node_store.rs:224 (finalize) → :250 (external_merge_sort) → psort |
| pass-2 worker transform | xml_reader.rs:1634 (pass2_parse_blob) |
| ref resolve / WKB | reader.rs:335 (parse_ways_rels), :86 (encode_wkb); geometry.rs:11/:21 |
| worker Arrow encode | writer.rs:665 (ColumnEncoder::encode) → :648 (EncodedRowGroup) |
| pass-2 sink build | xml_reader.rs:2130 (build_pass2_sink) |
| in-order writer stitch | writer.rs:692 (append) / :702 (finish); builders :714/:748/:767 |
| boundary stub (serial) | xml_reader.rs:1951 / :2080 (flush_stub) |
| tag JSON (canonical, sorted) | tags.rs:19 (serialize_tags) |
| content digest / equivalence gate | digest.rs:digest_file · CLI verify --digest |
| census of tables already on disk | census.rs:census · CLI census / census --pyramid |
| the census closure identity | census.rs:Partition::unaccounted / :balances |
| a percentage that carries its denominator | census.rs:Ratio (the ONLY door to a %) |
| phase log + per-phase occupancy | phase_log.rs:Phase · StopFlag |
| optimize row-group fan-out | optimize.rs:gatling_collect |
| coord merge fan-out | node_store.rs:merge_sorted_chunks_to_arrow |
| GeoParquet → Arrow IPC | geo2arrow.rs:geo2arrow (ordered::run_ordered_sink) |
| verify row-group counters | verify.rs:count_node_refs / :count_geometry |
| search-index row-group fan-out | side_outputs/search.rs:gatling_features |
| the ONE sanctioned private thread | xml_to_pbf.rs:blob_writer_loop (single blocking-I/O sink) |
ROOT LAW #0 status (2026-07-22)
osm-katana holds zero std::thread::scope worker pools. Every fan-out in the crate — both convert passes, the coord merge and its reference twin, the parquet→Arrow converter, the verify counters, the search-index build, the optimize collector, and the row-group encode in writer.rs's tests — routes through gatling. The crate has no entries left in SCOPED_POOL_ALLOW (tests/rayon_free_law.rs), so a scope pool anywhere under osm-katana/src now trips the law test with no exception to hide behind.
The single exception is xml_to_pbf.rs's blob_writer_loop: ONE named, long-lived thread doing blocking file I/O off the gatling collector, fed by a bounded mpsc channel. That is IO-offload, not CPU fan-out — there is exactly one of it, it never scales with core count, it claims no units from a shared cursor, and it does no parse/encode. It is licensed by PRIVATE_POOL_ALLOW and commented at the call site.
Measuring it — per-phase core occupancy, not just wall time
Wall time cannot tell you which phase drops the box. Every phase opens a phase_log::Phase (phase_log.rs), whose done event carries the phase's REAL occupancy: cpu_cores (process CPU-seconds / wall-seconds over that phase's window), n_cores, busy_pct = cpu_cores / n_cores, and collector_pct (the fraction of wall the single collector spent inside process()). Phases run sequentially over one process-wide CPU counter, so this attributes occupancy per phase with no per-thread accounting. Read them with --log <file.jsonl> — or let the bench do it: examples/nornir-bench.rs parses that log back and publishes one row per phase (osm-katana_pass1 / _sort_and_stree / _pass2 / _convert).
Two signatures to read for:
convert.busy_pctbelow every individual phase's ⇒ the pool drains between bursts — a barrier, not a slow phase.- high
collector_pctwith lowbusy_pct⇒ collector-bound; the workers are starving behind the single in-order sink.
The phase guard's teardown used to be part of the problem: the sampler thread only noticed the stop flag between 100 ms sleeps, and done() joins it before reading the clock, so every phase boundary charged up to 100 ms of dead time with the whole pool idle — four times per convert. StopFlag (Mutex + Condvar) makes that immediate.
Saturation state
Both passes fan across every core on gatling; the sort barrier (Stage 2) and the single-writer parquet stitch (Stage 4) are serial-by-design and exempt. The LPT weighting seen in the separate read_pass2_mmap mmap API — gatling_for_each_balanced weighted by blob byte length (reader.rs:138, comment reader.rs:130-136) — serves the load_ways_pbf helper, not the convert CLI (which uses run_typed). The SlotFill::Big choice (xml_to_pbf.rs:728-734) is what keeps slot turnover from starving the workers on planet-scale PBF. This doc records no fabricated cores_busy: honest all-core numbers land in the README.md zoomies-katana mashup on the next heavy-bench wave (the planet run); the measured per-phase occupancy on a 319 MB PBF is in benchmarks.md.
Known remaining drains (measured, not guessed — see benchmarks.md):
sort_and_streeis the phase barrier and holds only ~4 of 32 cores: pass 1 must be fully drained before the tail chunk can be sorted, and pass 2 cannot start until the STree exists.- Pass 1 is collector-bound —
collector_pct≈ 50 %. The single in-order sink copies every packed coord buffer into the sorter and writes the wholenodes.parquet(565 MB + 533 MB on one thread for the 319 MB input). Measured: the coord copy is not the cost — it already runs at memory bandwidth (3.83 GB/s; a hand-writtenmemcpywas 0.5 % faster and was reverted, seebenchmarks.md). The remaining ~50 % is the parquet row-group stitch + footer metadata, which is single-writer by format. - Pass 2 re-reads and re-inflates every blob, including the node-only blobs it will find nothing in. Pass 1 already knows which blobs carry ways/relations; passing that set forward would let pass 2 skip them outright. Not attempted — the streaming codec's
transformis handed a slot-relative slice and does not know the blob's absolute file offset, so the map has no key yet.
The pyramid stopped carrying what a reader cannot see (2026-08-31)
Three findings from a full Sweden cut on oden, each measured before it was acted on.
K16 — an untagged node is a vertex, not a feature
nodes.parquet was placed into the pyramid in full: nodes scanned: 103 488 583 · placed 103 488 583. In OpenStreetMap the overwhelming majority of nodes exist only to give a way its shape, and the ways that use them are already in the pyramid (or in a network table), so the vertex adds nothing a reader can see and nothing a classifier can name.
MEASURED whole-country, from the run's own scan ledger (no sampling):
SCAN LEDGER nodes: file 103 488 583 = placed 2 406 592 + untagged nodes 101 081 991
+ NULL geometry 0 + undecodable 0 + no geometry extent 0
· rows read 103 488 583 · UNACCOUNTED 0
101 081 991 of 103 488 583 nodes — 97.7 % — carried no tags at all.
A correction, and how the wrong number was produced
An earlier revision of this section (and the commit that landed it) claimed "107 299 of 188 112 rows (57.0 %) over all nineteen z6 tiles". That figure is wrong, and the error is worth recording because it is invisible in the output. The census flattened the pyramid's paths when it copied the tiles —
cp z6/<x>/<y>.parquetinto one directory — soz6/33/15.parquetandz6/34/15.parquetoverwrote each other and 7 of the 19 tiles survived. The run printed7and the table above it still said "nineteen".A sample that silently loses two thirds of its population still produces a clean-looking percentage. The replacement number is taken from the tiler's own ledger, which counts every row of every file by construction and closes to
UNACCOUNTED 0— there is nothing to sample and nothing to get wrong.
korp's own map_networks module header had already measured the same thing from the other side — removing ways from the [6,9,12] pyramid shrank it by "only 5.54 % of bytes … because nodes dominate at those zooms".
The skip is ScanLedger::untagged_nodes, a ledger column and not a silent continue, because a row that leaves has to leave through a name — that is the whole of K15.
Two things it must not do, both guarded:
- All three spellings of "no tags" count: a NULL cell, an empty string and
{}. A cut writes{}for a bare vertex and the monolith path has left the cell null; a test that covered one of them would have reported a fix that measured as nearly no change. - It fails OPEN on a missing
tagscolumn. A pre-tagsnodes.parquetcannot answer the question at all, and there a null cell means this file does not say, not this node carries nothing. Skipping on it would drop every node of such a file — the guard inverted. Caught by three existing fixtures; now a guard of its own.
K18 — the fan-out cap measured tiles, and the cost is bytes
MAX_WAY_TILE_FANOUT is a fixed count of TILES, and a tile is a fixed fraction of the world — so the ground area it permits shrinks 4× per zoom step: ≈ 620 × 620 km at z12, ≈ 40 × 40 km at z15. The consequence is that the same feature is accepted at one zoom and refused at the next for no reason that has anything to do with the feature. MEASURED on the 2026-08-30 cut: z12 = 4 capped, z15 = 455 — coastline, river and railway runs placed in their CENTROID TILE ONLY, so the shore is served in one of the many tiles it crosses and vanishes exactly when a reader zooms in far enough to want it.
What the cap protects is the staging buffer, and that is measured in bytes. A long feature is cheap to duplicate because it is long and simple — Douglas–Peucker has already thinned a coastline run at z15 to a handful of vertices — while a dense polygon is expensive whatever its span. fanout_budget_tiles(per) = MAX_WAY_FANOUT_BYTES / per, clamped:
- never below
MAX_WAY_TILE_FANOUT, so the change can only ever place MORE geometry than before — it cannot newly refuse anything; - never above
MAX_WAY_FANOUT_TILES_CEILING(512 × 512), so a mis-estimatedpercannot ask for an unbounded number of placements.
K17 — a ratio whose denominator is the wrong set is a different claim
The relation probe printed 274 206 rows, 225 799 carry assembled geometry (82.3 %), which reads as one relation in six failed and cost a reader an afternoon. Only an AREA relation is ever GIVEN geometry (RelationKind::Area): a type=route is an ordered set of ways, a type=restriction is a rule, a type=public_transport is a stop group. MEASURED on that same file: 226 967 area relations, 225 799 assembled = 99.49 %; the 48 407 without geometry are 14 588 routes, 14 412 stop areas, 5 633 turn restrictions, 2 463 waterway relations and so on. The real failure set is the ~1 324 areas that did not assemble (0.58 %).
The probe does not classify relations, so it now prints no ratio at all and names both populations instead. A percentage it cannot compute honestly is worse than none.
The geometry column says what it is (2026-08-31)
Every parquet this crate writes carried WKB in a bare BYTE_ARRAY. A reader learned it was geometry only by parsing the GeoParquet geo key in the footer — an out-of-band JSON convention, not the schema. Parquet's own GEOMETRY logical type exists now, and parquet 59.2.0 implements both ends of it, so the column can carry its own type.
shared::schema::geometry_field() is the ONE writer (LAW 5). It returns the field with the parquet_geospatial::WkbType arrow extension type, which is the only door through which ArrowWriter emits LogicalType::Geometry and runs the writer-side bounding-box accumulator that fills ColumnChunkMetaData::geo_statistics. Five production schema builders route through it — nodes_schema, ways_schema, relations_schema, tiles::tile_fields, network's table schema, graph's two, and optimize's repack.
parquet's geospatial feature is not on by default; a build without the feature line in Cargo.toml writes the old bare column and nothing complains.
What it buys, and what it does not
MEASURED 2026-08-31, geometry_columns_carry_the_logical_type, both arms on identical bytes: the declared field comes back Geometry(GeometryType { crs: Some("srid:0") }) with geo_statistics present and the bbox x (10.594, 24.178) · y (55.137, 69.06). The bare field comes back logical type: None, geo_statistics: NONE.
- Interop. DuckDB, Spark and skade read the geometry from the schema. No
geoblob to agree on first. - A second source for the prune. The chunk bbox exists whether or not a covering column was appended. korp's
mapquery.rsdocuments the hole this closes: a freshly-cutways.parquethas no covering bbox and is therefore "served slowly and honestly". - No speed.
geo_statisticslives onColumnChunkMetaData— the same granularity korp'smapquery::row_group_intersectsalready prunes at through the fourbbox.*leaf statistics — andparquet's page index (src/file/page_index/) carries no geospatial variant, so there is no page-level spatial prune to reach for. Nothing on any read path gets faster. - The
covering.bboxcolumn STAYS. It is what korp culls per ROW with, before any WKB is decoded. Deleting it to reclaim its 19.04 MB ofroads.parquet(6.36 % compressed) would push that cull onto the geometry column — 146.50 MB compressed, 487.90 MB uncompressed. Measured on~/.korp/work/sweden/networks/roads.parquet, 2 277 368 rows.
It cannot force a re-cut
TileRecipe's columns term is built from f.name() and short_type(f.data_type()), and an extension type is field METADATA that changes neither. Were that ever to stop being true, every cut pyramid on every box would be judged Stale — 1 635 971 tiles re-cut for a metadata edit. the_logical_type_does_not_move_the_recipe pins the name, the DataType, and that the field carries metadata at all, so the test cannot pass by the field being plain. Both guards seen RED with the bare field restored.
census — asking a question of tables that already exist (2026-08-31)
The 57.0 % correction above records what went wrong. This verb records why it could: there was no way to ask a question about a pyramid on disk except by hand-rolling a script, and the script that got hand-rolled lost two thirds of its population to a cp.
ScanLedger cannot be sampled wrong — but it only exists inside a tiling run. Once the tiles are written, the discipline was gone. census.rs is that discipline, applied to files that already exist:
osm-katana census ~/.korp/work/sweden # the four extract tables
osm-katana census ~/.korp/work/sweden/tiles --pyramid # the pyramid, footer only
osm-katana census ~/.korp/work/sweden/tiles --pyramid \
--tags --geometry --classes --json census.json
The property that makes it worth having
Every population sums to the file's own footer num_rows, and the books close. A Partition carries its denominator, Partition::unaccounted() is printed on every line, and a partition that does not balance prints no percentage at all and exits non-zero. A ratio over books that lost rows is precisely the machine that produced the wrong number.
Three further properties, each a direct answer to how the 57.0 % happened:
- One accumulator produces both the population size and every ratio. The bug's signature was a table saying nineteen above a run that read seven.
filesandrowsand every bucket come out of oneGroupCensus, so they cannot disagree — and the verb does the walk itself, keying every file by its path relative to the root, so there is no copy step to lose a tile in. - A flattened directory is REFUSED, not sampled. In
--pyramidmode a file must sit atz<Z>/<X>/<Y>.parquet; acp z6/*/15.parquet flat/leaves a15.parquetat depth 1 and the census reportsNONCONFORMING 1and exits 1. - Path identity is checked against content identity. A tile moved to a legal-looking address keeps a legal-looking layout, so the layout check cannot see it — but the file's own
covering.bboxfooter statistics can. A file whose extent does not intersecttile_bounds(z, x, y)isMISPLACED. The test is intersection and not containment on purpose: a feature is placed in every tile its bbox overlaps and the stored geometry is the whole feature, so a healthy tile routinely spills past its own bounds. 0 misplaced of 1 635 971 on the live cut.
Every gate was seen RED
Five mutations of the shipped code, each killing exactly the tests it should and no others (tests/census_books_close.rs, 9 tests):
| mutation | tests killed |
unaccounted() always returns 0 |
the three closure gates |
note_nonconforming never increments |
the flattened-pyramid gate |
Extent::intersects always true |
the misplaced-tile gate |
| the non-vacuity check removed | the zero-files gate |
a missing tags column fails CLOSED |
the fail-open gate |
Sabotage::{DropTagged, DropPoints, InflateTotal} drop or mis-attribute a population inside the real counting loop, so the red arm travels the same code the green one does. It is reachable from a shell: OSM_KATANA_CENSUS_SABOTAGE=drop-tagged, which on the live z6 prints UNACCOUNTED 343937, suppresses every % on the line, and exits 1.
What it costs — MEASURED on oden, 2026-08-31, 32 cores
The live Sweden pyramid, /home/korp/.korp/work/sweden/tiles, page cache warm (korp-server was serving the same objects):
| run | files | wall | what was read |
--pyramid |
1 635 971 | 11.4 s | footers only — no data page |
--pyramid --tags |
1 635 971 | 11.3 s | + the tags column (646 MB compressed) |
--pyramid --tags --geometry |
1 635 971 | 23.7 s | + geometry (34.53 GB compressed) |
tables, --tags |
4 | 3.5 s | 127 720 403 rows |
Seconds, not minutes, and the difference between the first two rows is the whole design: rows, bytes, row groups, column bytes and covering extents need no data page at all.
Fan-out is gatling (ROOT LAW #0), one unit per x directory, and each worker folds straight into its own GroupCensus. The pyramid is 1 635 971 files under 1 417 x directories; a Vec<PathBuf> over the leaves would be ~100 MB and the largest allocation in a process whose whole job is to hold nothing. 1 417 units over 32 cores is ~44 units each, so the longest single unit is a small fraction of a worker's total and gatling's shared cursor absorbs the z6-vs-z15 spread without a weight function.
What the first run found
MEASURED 2026-08-31, on the pyramid written at 06:15–06:17 UTC that day (mtimes from the census's own JSON), recipe token identical on all 1 635 971 files — one cut, not a mix:
ALL files 1 635 971 · bytes 44 449 184 179 (44.45 GB) · rows 33 357 988
geometry: LineString 9 052 208 (27.1 %) + MultiPolygon 16 961 280 (50.8 %)
+ Point 7 344 500 (22.0 %) · UNACCOUNTED 0
tags: tagged 28 959 168 (86.8 %) + untagged `{}` 4 398 820 (13.2 %) · UNACCOUNTED 0
geometry × tags: every one of the 4 398 820 untagged rows is a LineString —
0 untagged Points, 0 untagged MultiPolygons.
Two figures reproduced exactly, which is what a validation is for:
nodes.parquet: 101 081 991 of 103 488 583 untagged (97.7 %) — theScanLedgernumber, from the file rather than from the run,UNACCOUNTED 0. All 101 081 991 are the{}spelling; the NULL and empty-string spellings are 0 in this cut, which is why a census that tested only one of the three could report a fix as nearly no change.- z6: 31 981 of 375 918 (8.51 %) untagged, over 19 tiles —
korp f6aac7e's corrected figure, from all nineteen this time. That commit adds "and NONE of them is a point"; thegeometry × tagscross-tabulation confirms it, and generalises it to all four zooms.
⚠ A figure in
osm-katana-v2-design.mdthat this pyramid does not supportThat document quotes "sub-pixel areas shipped as
Point(11 101 365 of 12 628 347 area placements, 87.9 %)", beside a centroid-only figure explicitly updated for the 08-31 recut. The pyramid that recut produced holds 7 344 500Pointrows in total, across all four zooms — 3 756 865 fewer than the claim needs, before any of them is asked whether it is an area. The claim cannot be true of the tables now on disk.Which of the two is wrong is not determinable from the tables, and that is the point: the 87.9 % carries no date, no file list and no recipe token, so there is nothing to check it against. Every figure this verb prints ships with all three in its JSON.