nordisk/znippy-zoomies

osm-katana — guide

Class: guide · Function: osm-katana convert. Build, run, tune. The shape is in osm-katana-design.md; the why in osm-katana-reasoning.md.

Build

CARGO_TARGET_DIR=/mnt/work4t/cargo-target/zoomies cargo build --release -p osm-katana

Binary lands at target/release/osm-katana.

Run

osm-katana convert planet.osm.pbf                 # → out/{nodes,ways,relations}.parquet
osm-katana convert region.osm.bz2 -o region_geo   # bz2 input, custom output dir
osm-katana convert extract.osm.gz                 # gz input
osm-katana convert small.osm                       # plain XML input

Input format is chosen by extension (convert_inner, osm-katana/src/lib.rs:508): .osm.pbf, .osm.bz2, .osm.gz, .osm. Output is always three GeoParquet tables — nodes.parquet, ways.parquet, relations.parquet — in the -o directory (default out).

Every run is the same two-pass, all-core pipeline: pass 1 extracts node coordinates, a serial sort barrier builds the node store, pass 2 resolves ways/relations and encodes geometry, and a single in-order writer stitches the pre-compressed row groups. See the design doc's flow diagram.

Flags

Notes on the guardrails (all enforced in convert_inner):

• --geometry raw supports XML inputs only; PBF must use the default resolved (lib.rs:516).
• --clip / --poly require resolved — the clip is a node-membership test that needs decoded coordinates, so it cannot run on raw's node-ID lists (lib.rs:522). The clip is applied inside the pass-1 workers at coordinate decode (clip_keeps, lib.rs:401), not as a separate pass.
• With a clip, EVERY convert runs pass 2 as two sub-passes: ways first, then relations from where the relations start (PBF: the first relation blob; .osm: the first <relation; .osm.bz2: the last <way block; .osm.gz: byte 0 again, gzip cannot seek), and a relation is written only when at least one member survived the clip (a way in ways.parquet, a node in the node store, or a relation that was itself kept). Without this relations.parquet was the whole PLANET's relation set — 14.6 M rows, 1.05 GB, for a Sweden cut whose resolve stage keeps 274 k — and that cut is made from planet-latest.osm.bz2, so the bz2 arm is the one that pays. The pass-2 done line of convert.jsonl says which happened: rel_prefilter is clip or none, with rels_dropped_by_clip, rels_kept_via_relations, rel_oracle_ways, rel_pass_start_offset (PBF, .osm, .osm.bz2) and rel_blobs_planned (PBF) beside it. Measured on Liechtenstein with a bbox over Vaduz: 938 → 188 relation rows, 2 663 580 → 991 170 bytes, the same 188 ids via bz2 and via pbf.

Tune

• --vtd-workers N pins the worker count; leave it 0 to get cores − 1.
• Watch htop: both passes should sit near-flat across cores. Two short dips are expected and correct — the serial sort barrier between the passes (Stage 2, itself multithreaded) and the single-writer parquet stitch at the tail (Stage 4). Neither is a saturation bug; see the reasoning doc.
• For a bz2 input the progress bar length is the compressed size × 2 (two passes), so it looks slow during pass-1 decompression and fast afterward — the message field always shows the current activity (lib.rs:536-539).
• --compression none trades file size for the least tail work; zstd (default) is compressed on the pass-2 workers, not on the writer thread, so it does not stall the stitch.

Related subcommands

osm-katana ships sibling verbs for partial conversions (osm-katana/CLAUDE.md): xml2pbf (OSM XML/BZ2 → PBF), pbf2geo (PBF → GeoParquet), geo2arrow (GeoParquet → Arrow IPC), verify (inspect/verify output) and census (below). This guide covers convert, the end-to-end path.

What is actually in these tables? — census

Reach for this instead of writing a script. verify answers is this fit to serve?; census answers what is in it? — and it answers with books that close, so the number you quote cannot have quietly lost part of its population.

osm-katana census ~/.korp/work/sweden                    # the four extract tables
osm-katana census ~/.korp/work/sweden/tiles --pyramid    # 1.6 M tiles, footer only

nodes.parquet  files    1 · bytes 709045761 (0.71 GB) · rows 103488583 · row groups 217
                 rows cross-check: footer num_rows 103488583 vs row-group sum 103488583 — AGREE
                 column bytes: 707771791 compressed / 3618034439 uncompressed
                   geometry   514933550 B compressed (514933550 of 707771791 (72.8 %)) …
                 tags: 103488583 rows = tagged 2406592 of 103488583 (2.3 %)
                       + untagged: {} 101081991 of 103488583 (97.7 %) · UNACCOUNTED 0
VERDICT: books close, every population accounted for.

Read UNACCOUNTED first. It is on every population line, it is the file's own footer num_rows minus what the buckets claim, and if it is not 0 the verb prints no percentage on that line at all and exits non-zero. Every ratio carries its denominator in the same string — 101081991 of 103488583 (97.7 %), never a bare 97.7 %.

Timings on oden (32 cores, page cache warm), the live Sweden pyramid of 1 635 971 tiles / 44.45 GB: footer-only 11.4 s, --tags 11.3 s, --tags --geometry 23.7 s. The four extract tables with --tags: 3.5 s over 127 720 403 rows. If a census ever takes minutes, it is because you asked for --geometry.

When it refuses

It exits non-zero, and says which of these it is:

• read ZERO parquet files — you pointed it at the wrong directory. It never prints a clean empty table.
• NONCONFORMING n file(s) are not at a z<Z>/<X>/<Y>.parquet address — you pointed --pyramid at a flattened copy of a pyramid. Point it at the real root; a flat copy has already lost every tile that shared a <y> name with another.
• MISPLACED n of m addressed file(s) — a tile's own covering bbox does not intersect the tile its path names. Something moved files.
• partition <name> does not close — UNACCOUNTED n — a bug in the census itself. Do not quote anything from that run.

Quote the JSON, not the terminal

--json writes the whole census plus the identity of what was measured: every input's path relative to the root, its bytes, its mtime, and the osm-katana:tile-recipe token from its footer. A number without provenance is how the 57.0 % survived for a week in two design documents. 53 KB for the whole Sweden pyramid.

Proving the guard works

OSM_KATANA_CENSUS_SABOTAGE=drop-tagged osm-katana census <dir> --pyramid --tags; echo $?

It drops a whole population inside the real counting loop. Expect UNACCOUNTED non-zero, every % gone from that line, VERDICT: REFUSED and exit 1. Arms: drop-tagged, drop-points, inflate-total.

buildings — the height table

osm-katana buildings <extract-dir> -o <out-dir>

Cuts buildings.parquet + buildings.json out of ways.parquet + relations.resolved.parquet: one row per building=* / building:part=* feature, with height_m, min_height_m, levels and a height_source stamp. Hilbert-packed with covering.bbox through the same spatial::pack_batches the networks use, so korp reads it through the same seam. Write it BESIDE a served pyramid, never into it.

Read the histogram it prints — that is the point of the verb. Sweden, 2026-08-31, 3 841 483 rows / 297.5 MB / 16.0 s (154.3 MB / 11.3 s with --no-generalise, which drops DELTA 2's geometry_z6/z9/z12 — +92.8 % of the file is those three columns):

           absent                 3675499  95.6792 %
           tagged_metres             9551   0.2486 %
           tagged_feet                  1   0.0000 %
           derived_from_levels     156167   4.0653 %
           rejected                   265   0.0069 %

95.68 % of Swedish buildings have no height in OSM at all. derived_from_levels is building:levels × 3.2 m — an assumption with a ±20 % median error, which is why it is a separate bucket from tagged_metres rather than the same f32. rejected means a tag WAS there and could not be used (height=1;12, height=B): if that number jumps, the parser regressed — the mappers did not. Every number is in buildings.json too, so an operator never has to open the parquet. See .nornir/osm-katana-v2-design.md § DELTA 1b.

Did my change alter the output? — verify --digest

sha256 nodes.parquet is not the check: row groups are encoded on the workers and their boundaries move run-to-run, and the tail row groups land in worker-completion order, so two runs of the same binary give different bytes. Compare the content digest instead:

osm-katana convert in.osm.pbf -o out-before && osm-katana verify out-before --digest
# …change something…
osm-katana convert in.osm.pbf -o out-after  && osm-katana verify out-after  --digest

       nodes.parquet:    353167 rows  set 16a8386cf4b7ad76  ord 057676bc3141a944
        ways.parquet:     41702 rows  set 02ed64602fadfe31  ord 1a55d7e5a445c7a6
   relations.parquet:       938 rows  set 119bffec8cc479f0  ord 03cc98ab2913b696

Compare the set column (and the row count): it is the order-independent multiset digest, so it ignores the layout/order the pipeline legitimately varies while still catching any changed, dropped, added or duplicated row. ord additionally pins row order — use it when a change is expected to preserve order too. And cmp a/node_coords.arrow b/node_coords.arrow: that file is byte-identical.

Bench

Driven by nornir through the shared harness. Honest all-core numbers land in the README.md zoomies-katana mashup after the quiet-box planet run reaches the warehouse. Run nornir bench run znippy-zoomies, then nornir docs book znippy-zoomies to fill the mashup.

PREVIEW — the seconds-turnaround lens for the optimise loop

The heavy bake-off builds a 1 GB corpus in every rival format before it gets anywhere near osm-katana. NORNIR_ZOOMIES_CORPUS_MB=0 means "no decompressor corpus", so the decoder roster is empty and only the osm-katana arm runs:

NORNIR_ZOOMIES_CORPUS_MB=0 NORNIR_ZOOMIES_WORK=/tmp/zb \
  cargo run --release --example nornir-bench           # uses ~/work/liechtenstein-latest.osm.pbf

NORNIR_ZOOMIES_CORPUS_MB=0 NORNIR_ZOOMIES_WORK=/tmp/zb \
NORNIR_OSM_KATANA_INPUT=/path/big.osm.pbf NORNIR_ZOOMIES_ITERS=3 \
  cargo run --release --example nornir-bench

Same code path, same rows, same metric keys as the heavy run — including the per-phase occupancy rows and the digest gate — just small. It prints, per phase:

osm-katana per-phase core occupancy (fastest iteration):
  pass1                1.145s   16.35 / 32 cores  busy  51.1%  collector  55.3%
  sort_and_stree       0.450s    4.74 / 32 cores  busy  14.8%  collector   0.0%
  pass2                1.346s   20.70 / 32 cores  busy  64.7%  collector  13.5%
  convert              2.963s   16.45 / 32 cores  busy  51.4%  collector   0.0%
osm-katana output digest (3 runs): nodes.parquet=…/35316700 …  → STABLE

A run counts as PREVIEW when the corpus is under the heavy 1024 MB default or the osm-katana input is under 1024 MB, and a preview never emits the BenchRun contract line — stdout says PREVIEW RUN, the numbers go to stderr. So nornir bench run cannot warehouse a light number and it can never replace a real host row in a README table. There is no override; run it heavy to publish.