Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/znippy-zoomies

osm-katana/CLAUDE.md

osm-katana

Fast OSM format converter. Part of znippy workspace. See znippy/WORKSPACE.md for dependency graph and release gate protocol.

CLI subcommands

osm-katana --version                     # `osm-katana v<semver> (<git-hash>)` — run it
                                        # before quoting a number from a binary
osm-katana convert  <input>              # OSM XML/BZ2/GZ/PBF → GeoParquet,
                                        # Hilbert-packed with `covering.bbox`
                                        # (--pack false for convert order)
osm-katana xml2pbf  <input> [output]     # OSM XML/BZ2 → PBF
osm-katana pbf2geo  <input>              # PBF → GeoParquet
osm-katana geo2arrow <input> <output>    # GeoParquet → Arrow IPC
osm-katana verify   [dir]               # inspect/verify GeoParquet output
osm-katana verify   [dir] --geometry-quality
                                        # + a SHAPE census of every geometry column
                                        # (geometry + geometry_z6/z9/z12): decode
                                        # failures, point/line/poly histogram,
                                        # unclosed rings, <4-vertex rings, zero-area
                                        # outers, self-intersecting rings (exact up
                                        # to 512 segments, larger counted `big?`,
                                        # never silently passed). Books close per
                                        # column or the verb exits 1.
                                        # Added for map v2.1 (the z12.3 water-tangle
                                        # investigation, 2026-09-02): it answers "is
                                        # the geometry bad at the SOURCE?" in one run.
osm-katana verify   [dir] --geometry-quality --geometry-baseline zero|recorded
                                        # WHICH recorded defect budget the census is
                                        # judged against (row 39). decode_fail /
                                        # unclosed / selfx ABOVE the record /
                                        # unbalanced books = FAULT; <4v and area0 are
                                        # reported and gate nothing. `recorded`
                                        # (default) is the live Sweden census of
                                        # 2026-09-02 — it tolerates the 99 unclosed +
                                        # 599 selfx `ways::geometry` carries out of
                                        # CONVERT, PRINTS each as a DEBT on every run,
                                        # and fails on one more. `zero` tolerates
                                        # nothing: what a fresh cut off the fixed
                                        # encoder must pass. The verdict comes from
                                        # `verify::check_cut_geometry`, the SAME call
                                        # the cut chain's post-condition step makes.
osm-katana census   [dir] [--pyramid]   # WHAT IS IN these tables — rows, bytes, column
                                        # bytes, geometry/tag/class histograms — with
                                        # books that CLOSE. Every population sums to the
                                        # footer's own num_rows, every ratio carries its
                                        # denominator, and UNACCOUNTED != 0 suppresses
                                        # every percentage and exits 1. Footer-only by
                                        # default (11.4 s over 1 635 971 tiles);
                                        # --tags/--geometry/--classes read real columns.
osm-katana admin-poly <input> --name <region> --admin-level 2 -o <out.poly>
                                        # region boundary → Osmosis .poly, from
                                        # a .osm.bz2 (block-parallel) or .osm.pbf
osm-katana networks <extract> -o <dir>  # cut roads/rails/trails/rail_nodes as
                                        # their own GeoParquet tables, PLUS the
                                        # adjacency (graph_edges/graph_nodes)
                                        # out of the same pass. --no-graph opts
                                        # the graph out.
osm-katana buildings <extract> -o <dir> # cut `buildings.parquet` + `buildings.json`:
                                        # one row per building/building:part with
                                        # height_m, min_height_m, levels and a
                                        # height_source stamp (absent / tagged_metres /
                                        # tagged_feet / derived_from_levels / rejected).
                                        # Sweden: 3 841 483 rows, 297.5 MB, 16.0 s —
                                        # and 95.68 % `absent`. Read the histogram.
                                        # --no-generalise drops geometry_z6/z9/z12
                                        # (154.3 MB / 11.3 s; +92.8 % of file is them)
osm-katana graph <dir> --question unguarded-crossings
osm-katana graph <dir> --question nearest-crossing --lon 18.07 --lat 59.33
                                        # ask the road/rail graph something a
                                        # table of geometries cannot answer
osm-katana spatial-pack <input> -o <out> # Hilbert row order + GeoParquet 1.1
                                        # `covering.bbox`, so a reader can prune
                                        # whole row groups by min/max stats,
                                        # PLUS geometry_z6/z9/z12 generalisation
                                        # columns (--generalise false to omit)
osm-katana tile <extract-dir> -o <out> --zooms 6,9,12,15
                                        # cut a slippy-tile pyramid + manifest

convert --pack (default true)

Convert writes in PBF order, so every row group spans the whole extract and a viewport query has nothing to prune on. MEASURED on the Sweden cut, central Stockholm ~25 km: the unpacked 902.13 MB ways.parquet is a full scan — 902.13 MB, 3 538.2 ms; packed it reads 71 of 1 139 row groups for 27.08 MB, 144.0 ms. osm-katana relations packs relations.resolved.parquet the same way, through the same spatial::pack_batches the networks already use.

The price, also measured: 78.3 s for the three Sweden tables (+2.3 % on a 56-minute cut) and up to 15.6 GB resident while the largest is sorted — plus disk, which is the real cost. See .nornir/osm-katana-v2-design.md for the per-column census: the growth is id/node_refs delta locality, not the 8 192-row groups, and --node-refs false takes ways.parquet from +29.9 % to +10.6 %. The covering.bbox column is quantised onto a 2^-12° grid (spatial::COVERING_GRID_SHIFT), which halves it — the three tables pack to 2 711.33 MB (+50.5 %), down from 3 091.29 MB (+71.6 %), with the row groups a viewport scans measured IDENTICAL through korp's shipped reader.

Content-preserving: verify --digest's set digest is unchanged.

spatial-pack --generalise (default true)

Writes geometry_z6 / geometry_z9 / geometry_z12 beside geometry — the same per-ring Douglas–Peucker tiles.rs runs at each rung, at the same epsilon, in a column instead of a pyramid. geometry IS rung 15; there is no fourth derived column, because korp's level type is a four-variant enum.

MEASURED on Sweden (2026-08-31), central Stockholm, warm, ways.parquet: geometry_z12 reads 96.4 ms / 18.67 MB against 134.6 ms / 25.49 MB for full precision, and against a pyramid z12 read that actually decodes its tile geometry (121.4 ms, 42 files) rather than only pruning it (17.1 ms — the number phase 0 quoted). At z15 the column path is 3.1× faster than the pyramid with one file open instead of 1 764.

Cost: +14.9 % pack wall and +55.8 % disk on ways.parquet (53.89 s → 61.94 s; 1 161 → 1 810 MB), +90.4 % on relations.resolved.parquet. Both display tables together add 822.7 MB — 1.9 % of the 44.45 GB pyramid they replace. Point tables (nodes.parquet, rail_nodes) and the graph tables get no rung columns.

TileRecipe is unmoved (no pyramid re-cut). NetworkRecipe MOVES — every cut network table reads Stale and is re-cut, 21.3 s. See .nornir/osm-katana-v2-design.md DELTA 2 for the full census and the red arms.

spatial-pack and min_zoom (DELTA 3, 2026-09-01)

The packer writes rows in (min_zoom, hilbert) order — band first, Hilbert inside — so a reader asking min_zoom <= camera prunes whole row groups off the footer. A table with tags and no min_zoom (every cut before 2026-08-31 14:00) gets the column DERIVED by the pack through the one ladder (shared/class.rs); a present column is kept. In the rung columns a row whose min_zoom is deeper than the rung is NULL, not a collapsed point.

MEASURED on the re-packed Sweden ways.parquet (9 325 142 rows, 1 139 groups): a min_zoom <= 6 reader opens 201 of 1 139 groups (17.65 %) with no viewport at all; geometry_z6 is NULL on 82.41 % of rows, and the file is 1 572.70 MB against 1 810.22 MB for the DELTA 2 pack of the same table (−13.1 %). is_packed now also answers false for a Hilbert-only table, so pack_in_place upgrades it.

spatial-pack and the POINTS table (lane nd-node-rungs, 2026-09-03)

DELTA 3 shipped the band on the two LINE tables and left nodes.parquet behind: the live Sweden cut was packed on 2026-09-01 from a convert that predated DELTA 1, so it carried covering.bbox and tags and no min_zoom — 12 633 row groups a country camera had to cull with the covering box alone. Three things closed it, and none of them is a new rule:

On the real Sweden points table, packed off unpacked-backup/nodes.parquet:

  103 488 583 rows, 12 633 row groups, 47.8 s wall, 17.4 GB peak RSS
  file            1 376.55 MB  ->  1 045.35 MB   (-24.1 %)
    geometry        655.18 MB  ->    488.12 MB   (-25.5 %)
    id              324.12 MB  ->    158.08 MB   (-51.2 %)
    bbox.*          344.55 MB  ->    344.57 MB   (unchanged - already 2^-12 quantised)
    min_zoom              ---  ->      0.75 MB

bbox is UNTOUCHED on purpose. For a Point xmin == xmax bit for bit, so half of it is a duplicate — and both ways out were already measured and refused in spatial::COVERING_GRID_SHIFT: emitting two leaves takes korp's prune from 277 to 12 633 of 12 633 row groups (its reader names all four leaves), and there is no encoding left to try. The 2^-12° grid already halved the column (693.94 -> 344.55 MB) at an identical prune. Dropping it entirely would be worse than either.

verify now FAULTS this shape. Fault::UnbandedPacked — covering.bbox present, tags present, min_zoom absent or the row groups not band-first. Footer-only, through spatial::is_band_first (the packer's own predicate, so the guard cannot drift from the writer). Run against the live cut on 2026-09-03 it names two tables: nodes.parquet and the raw relations.parquet. An UNPACKED convert output is not a fault — it is a pipeline stage on its way to the pack — and the guard tells the two apart.

networks: per-edge speed on the graph (lane nvdb, 2026-09-02)

graph_edges.parquet carries speed_kmh: Float64 + speed_source: Utf8 (maxspeed | class) beside oneway. A way's own maxspeed wins where it parses (bare km/h, N mph, walk; zone spellings like SE:urban and multi-values like 70;50 REFUSE and fall back — guessing would poison the route through that street); otherwise the class default from graph::SPEED_TABLE — the ONE speed table, moved here from korp's roadgraph.rs so the cut and every router price off one source (LAW 5). Graph::load puts the speed on every NetArc (u8 km/h — the arc stays 24 bytes) and prices class defaults on an old cut without the column; Graph::speed_from_column says which happened, so a consumer can NAME its speed source. MEASURED on the live Sweden cut 2026-09-02 (km-weighted): 90.2 % of arterial km (motorway/trunk/primary/secondary/tertiary + links, 104 026 of 115 271 km) carry a tagged maxspeed; whole road network 23.7 % (176 346 of 745 201 km — tracks and paths are untagged and that is fine, they price at class defaults). Both columns ride the recipe token, so every existing graph cut reads Stale and re-cuts.

tile

The same tiles::tile_region korp's --tile-maps calls — one writer of a pyramid — but it writes files and nothing else: no S3, no manifest upload, no service restart. That is what makes it usable for measuring a pyramid, or for cutting one BESIDE a live one that a server is serving.

--zooms 6,9,12          # levels; part of the TileRecipe, so adding one makes
                        # every existing pyramid read STALE and be re-cut
--region-name sweden    # name written into tiles/manifest.json
--region <bbox>         # min_lon,min_lat,max_lon,max_lat; tiles outside refused
--budget-gib 4.0        # resident payload budget; over it the biggest tiles
                        # spill to per-shard scratch. 0 = unbounded
--workers 0             # 0 = all cores
--raster-px <px>        # simplification basis; also part of the TileRecipe.
                        # OMIT it to take the library's DEFAULT_TILE_RASTER_PX —
                        # a number spelled here would be a second writer of a
                        # recipe axis, exactly as `--zooms 6,9,12` once was
--exclude-networks <b>  # leave road/rail/route ways to the network datasets.
                        # Omit ⇒ TileOptions::default() (true), same reason

Sweden at [6,9,12,15] from ~/.korp/work/sweden: 1 623 474 tiles, 20.16 GB, 168.9 s on oden (2026-08-03, 32 cores at load 55 — contended).

⚠ That line is dated, and the date matters. Censused 2026-08-31 with osm-katana census --pyramid: 1 635 971 tiles, 44.45 GB — the tile count barely moved but the bytes are 2.2× larger, which the 08-03 figure hides completely. Neither number is wrong; quoting either without its date is. Use census --json, which emits input paths, sizes, mtimes and the recipe token beside every ratio.

The output is not byte-reproducible. Two runs of one binary on one extract give the same tile key set and the same row counts, and ~0.14 % different rows: the sub-pixel dedup keeps whichever of 32 workers reached the shard lock first. Compare pyramids on key set + row counts + tag multiset, never on bytes.

vtd_bench remains a separate binary (dev/benchmark tool, not shipped).

Row 39 hand-off — the geometry gate the CUT CHAIN still has to call

verify::check_cut_geometry (this crate, src/verify.rs) is the chain-facing half and it is done: verify --geometry-quality calls it, tests/geometry_gate_in_the_chain.rs holds it red-first. The other half lives in korp-collectors, which cannot be edited from this repo. korp-collectors takes osm-katana as a PATH dependency (osm-katana = { path = "../znippy-zoomies/osm-katana" }), so the patch below applies as soon as this branch is in /home/rickard/git/znippy-zoomies.

Two dead gates go with it. refused_placements and tiles_per_zoom are rows in that module's doc table and nothing else: tiles_per_zoom went with the tile pyramid (check_pyramid was deleted; its one non-tile question moved into check_extract as geography_asserted), and refused_placements never had a producer at all — the only other occurrence of the name is a string literal in a unit test. A table that promises a gate which cannot go red is what hides the gates that can, so both are deleted rather than re-pointed.

--- a/src/osm_cut_verify.rs
+++ b/src/osm_cut_verify.rs
@@ -18,8 +18,7 @@ (the doc table, lines 18-19)
-//! | `refused_placements` | the tiler refused no out-of-region placement | 17 425 features / 51 535 placements in the leaky cut, 0 in a clean one |
-//! | `tiles_per_zoom` | the pyramid has the declared shape, and is never EMPTY | a clean Sweden cut is z6 19 / z9 517 / z12 24 007 |
+//! | `geometry_quality` | every geometry column holds SHAPES — no undecodable blob, no unclosed ring, no NEW self-crossing ring, books that close | 11 510 bowties in `relations::geometry_z9` + 99 unclosed and 599 self-crossing rings in `ways::geometry`, every one found by a human running `osm-katana verify --geometry-quality` AFTER a chain run that reported success (2026-09-02) |
 //! | `present` | **named geometry actually EXISTS where it was declared to** | … |
+//!
+//! ## Two rows were DELETED from that table, and the one above replaces them
+//!
+//! `refused_placements` and `tiles_per_zoom` were in the table and in nothing else.
+//! `tiles_per_zoom` went with the tile pyramid; `refused_placements` never had a
+//! producer in this module — the table promised a gate that could not fail. The shape
+//! of a check that cannot go red is a check that hides the ones that can.
@@ (CutExpectations, after `free_bytes_convert`)
+    /// **The recorded geometry-defect budget this cut is judged against.** Default
+    /// [`osm_katana::verify::GeometryBaseline::RECORDED`] — the live Sweden census of
+    /// 2026-09-02, which tolerates the 99 unclosed + 599 self-crossing rings
+    /// `ways::geometry` carries OUT OF CONVERT and fails on one more. Declare
+    /// `"geometry_baseline": "zero"` once a re-cut has cleared them (the encoder fix
+    /// is zoomies `10e63b8`).
+    pub geometry_baseline: &'static osm_katana::verify::GeometryBaseline,
@@ (impl Default for CutExpectations)
+            geometry_baseline: &osm_katana::verify::GeometryBaseline::RECORDED,
@@ (CutExpectations::from_config, beside the `free_gb_convert` arm)
+        // Only "zero" moves it; an unrecognised spelling keeps the default rather
+        // than silently loosening the gate.
+        if o.get("geometry_baseline").and_then(|x| x.as_str()).map(str::trim)
+            == Some("zero")
+        {
+            e.geometry_baseline = &osm_katana::verify::GeometryBaseline::ZERO;
+        }
@@ (check_extract, after the `present` loop, before the `for c in &v.checks` log)
+    // ── The SHAPES, not just their places. ──────────────────────────────────
+    //
+    // Everything above asks WHERE a geometry is. This asks whether it IS one:
+    // osm-katana's own census — the one that found the 11 510 bowties — judged
+    // against a recorded budget, so a known-historic defect cannot fail a cut and a
+    // NEW one always does.
+    let t = std::time::Instant::now();
+    match osm_katana::verify::check_cut_geometry(
+        dir,
+        &osm_katana::verify::EXTRACT_TABLES,
+        exp.geometry_baseline,
+    ) {
+        Ok(g) => {
+            for n in &g.notes {
+                log.line(format!("post-condition geometry_quality {n}"));
+            }
+            let cost = t.elapsed().as_millis();
+            v.push(match g.reason() {
+                Some(r) => CheckOutcome::fail("geometry_quality", r, cost),
+                None => CheckOutcome::pass(
+                    "geometry_quality",
+                    format!(
+                        "{} geometry column(s) censused against baseline `{}`, no defect \
+                         above the record ({} UNASSERTED)",
+                        g.census.len(),
+                        g.baseline,
+                        g.notes.len(),
+                    ),
+                    cost,
+                ),
+            });
+        }
+        // A census that cannot READ the tables has not proven them clean — the same
+        // rule `scan_outcome` applies to a probe that fails to open a file.
+        Err(e) => v.push(CheckOutcome::fail(
+            "geometry_quality",
+            format!("the geometry census could not read this cut: {e:#}"),
+            t.elapsed().as_millis(),
+        )),
+    }
@@ (mod tests, `a_verdict_names_which_check_failed_and_with_what_numbers`)
-        v.push(CheckOutcome::pass("refused_placements", "0 refused", 1));
+        v.push(CheckOutcome::pass("wholly_foreign_ways", "0 of 8612923", 1));
@@ (mod tests — the new guard, SEEN RED before the deletions above)
+    /// **No post-condition names a gate that cannot fail** (row 39).
+    ///
+    /// The names are split across `concat!` on purpose: this test reads its own file,
+    /// and a whole literal here would match itself.
+    #[test]
+    fn the_module_names_no_check_that_does_not_exist() {
+        let src = include_str!("osm_cut_verify.rs");
+        for dead in [concat!("refused_", "placements"), concat!("tiles_per_", "zoom")] {
+            assert!(
+                !src.contains(dead),
+                "`{dead}` is named here and produced nowhere — a gate that cannot go \
+                 red hides the gates that can. Delete it or give it a producer",
+            );
+        }
+    }

The chain-side red to see before applying it: plant an unclosed ring and a bowtie in a fixture ways.parquet, run check_extract over that directory, and assert the verdict refuses. It passes today — check_extract asks four questions about WHERE a geometry is and none about whether it is a shape.

Benchmarks

cargo bench -p osm-katana

History: znippy-zoomies/osm-katana/bench_history.json

Required columns per bench entry

{
  "date": "2026-05-30",
  "version": "0.2.0",
  "machine": "T14s",
  "cores": 16,
  "results": [
    {
      "name":              "liechtenstein_bz2_to_geo",
      "converter":         "convert",
      "input_format":      "bz2",
      "input_mb":          5.2,
      "reference_tool":    "bzip2 -d | osmconvert",
      "reference_mbs":     8.5,
      "single_core_mbs":   14.2,
      "multi_core_mbs":    95.0,
      "cores_used":        16
    }
  ]
}

Release

Follow release gate in WORKSPACE.md §Release gate protocol. Publish: cargo publish -p osm-katana