nordisk/znippy-zoomies

osm-katana v2.0 — planet.osm.pbf → GeoParquet 2+, without a pyramid

Written 2026-08-31 on oden. The consumer side of this is korp/.nornir/map-v2-design.md; this file is the WRITER's half.

What already exists (do not rebuild it)

The pipeline that produces the v2.0 layout is running today. Sweden, 2026-08-30, end to end 56 minutes (21:45 cut start → 22:41 served):

Step e is already the v2.0 layout. It is the existence proof: roads.parquet is Hilbert-sorted GeoParquet with a covering.bbox struct column and network/class as columns read through a parquet RowFilter before WKB decode. korp measures that path at 124 ms for central Stockholm z12 and 749 ms for whole-Sweden z6 over 2 251 123 roads — 711 MB for all four tables, no pyramid, no tile cap.

So v2.0 is three edits to a working pipeline, not a new one. Steps f and g are deleted.

Things a design brief will tell you to build that are already here

• Hilbert ordering — spatial.rs, hilbert_d(), 31 bits per axis (a 16-bit grid would collapse city-scale detail into one cell). Guarded: network.rs's the_covering_bbox_prunes_and_input_order_would_not fails unless the Hilbert layout shrinks the mean row-group bbox area by more than 4× against input order.
• covering.bbox — four Float32Array leaves, written by spatial::pack_batches, named in the GeoParquet 1.1 footer under covering.
• Node coordinate storage at planet scale — NodeStore::from_arrow_mmap / MmapTree. No rocksdb. No osmium. LAW: pure Rust, no shelling.
• Parallelism — gatling (rotaryengine). rayon is FORBIDDEN (LAW 3, enforced by rayon_free_law.rs). A brief that says "read PBF blobs in parallel with rayon" is out by name, not by preference.
• LogicalType::Geometry + geo_statistics — landed 57aafb3, one writer, shared::schema::geometry_field(). Completed by geo-strip-1 below, which is also the place to read about why the guards that landed it were green while the shipped file was wrong.

The three deltas

DELTA 1 — min_zoom: u8 — BUILT 2026-08-31 (osm-katana 0.2.10)

One UInt8 non-null column: the first rung at which a feature is allowed to draw, out of korp's ladder [6, 9, 12, 15].

Where the single source lives, and why

The survey came first, because "reuse the ladder that already exists" needs the ladder to be findable. Surveyed 2026-08-31 across korp, facett, znippy-zoomies and edda:

Three homes were weighed:

• facett-map, and osm-katana depends on it. Rejected. It is an egui component crate; a headless PBF → GeoParquet converter that publishes to crates.io cannot take a GUI framework as a hard dependency to learn that a lake is a lake.
• edda — the constellation's declared home for shared specs, and correct in principle. Rejected on facts, and recorded as the migration target the moment a THIRD consumer appears: it holds no geo crate, so this is a new crate plus a crates.io release cascade that must land BEFORE osm-katana can publish again, and a path = dep on a sibling repo breaks in every worktree that is not a sibling of edda/ on disk.
• HERE, osm-katana/src/shared/class.rs. Chosen. It adds no dependency edge and opens no cycle — korp already depends on osm-katana; osm-katana depends on neither korp nor facett. And it is the only site in the pipeline that can bake the decision into the data and make a change to it re-cut that data: class::rules_spelling() is a term of NetworkRecipe::token(), so editing a rule marks every table the old rule wrote Stale. facett's ladder cannot do that, and its own doc says why it should not try — LOD_TIER_ZOOMS is "a LOOK decision, not a data one".

The residual, said plainly. classify_with's match arms and class.rs's RULES are two texts describing one taxonomy. RULES is a faithful, arm-for-arm transcription with every arm citing its source — but a transcription is not a construction, and LAW 5 prefers construction. The by-construction close is to lift classify_with's kind+tier half out of facett-map into an edda crate and have both sides call it; that is a facett + edda + znippy-zoomies change and was out of this lane's repo. Until it lands, Tier::ladder_spelling() is the machine-readable statement of this side's ladder that the facett side can be asserted against.

Two ladders, one rule — and four rungs, not five

facett steps its cartographic tier on [0, 6, 9, 12, 15] (five tiers). The DATA ladder is [6, 9, 12, 15] (four rungs). Tier::min_zoom is the projection: the coarsest rung at or above the tier's camera. Country (camera 0) and Region (camera 6) both land on rung 6, because 6 is the coarsest rung that exists — which is exactly why five tiers do not force a fifth rung. Two lists, one rule, one classification, which is the shape facett's own lod_step already documents as "one rule, any level list".

The place=* hole the union closes

classify_label reads place=*; classify_with has no place arm at all. So today a place=city NODE tiers to Lod::Country and a place=city closed WAY falls through to Lod::City — the same shape as korp-map-pt-1 (2026-08-31), where a sub-pixel lake became an off-by-default grey dot because points and ways went through different maps. RULES unions the two functions into ONE ordered table read by ONE function taking only tags, so the question cannot be asked twice. The settlement rungs carry the measured national counts that set them: country 1 · region 5 · city 6 · town 67 · municipality 132 · village 996 · hamlet 14 348 · isolated_dwelling 25 221.

The unknown case is a decision

UNCLASSIFIED_MIN_ZOOM = 12 — City's rung, and the same rung classify_with's own fallthrough paints at. Not 6: an unclassified feature admitted to the coarsest rung is the osm-other defect (57 % of a z6 tile as grey remainder over the road network). Not 15 either: that would make the data withhold, between camera 12 and 15, something the painter is ready to draw — the dangerous direction. Loudness is separate from the value: classify_tier returns Option<Tier> and class::unclassified_share counts the Nones, so a hole in the table is a NUMBER a run reports.

What it buys today — MEASURED, and it is not a prune

tests/min_zoom_one_ladder.rs::band_first_order_prunes_and_hilbert_only_does_not, the same 65 536 rows written twice at 8 192 rows per group, counting from footer statistics alone how many row groups a min_zoom <= 6 reader must open:

8.0×, and the Hilbert arm is the control: on the shipped layout the column prunes nothing at row-group level, because every row group holds every rung. The standard advice — add the column, push the predicate down — is right in spirit and wrong in mechanism on a spatially sorted table. The prune is DELTA 3's, and DELTA 1 may not claim it. (Synthetic rows with a road-network-shaped rung mix, not the live Sweden cut.)

What DELTA 1 buys on its own is a per-ROW cull costing one u8 compare instead of a serde_json parse of tags — the read path's worst hotspot, 55.9 % of all allocations on a tile read (2026-08-03) — and the rung decided once at write time for the whole planet rather than once per feature per frame.

Where the column is — COMPLETE 2026-08-31 (lane WIRE)

Live on all seven tables: the four network tables (roads · rails · trails · rail_nodes) and now the three base tables (nodes · ways · relations). min_zoom sits in network_fields() for the first four, so it is in NetworkRecipe's column list automatically and a table cut before it existed reads Stale; for the base tables it is appended by schema::{nodes,ways,relations}_schema — last, so every positional read of id/geometry/tags/node_refs/version that existed before keeps its index.

writer.rs holds nine RecordBatch::try_new calls. Seven are production, and each gained one UInt8Builder and one column:

The remaining two are in writer.rs's own test modules (make_batch, and the PendingOutput guard's hand-built batch); the second needed the column added. NodeWriter is wired because the schema is shared and an unwired builder set does not compile — it is listed as dead rather than quietly counted as coverage, and it is tested anyway (writer::min_zoom_law), because a retained fallback that is wrong is worse than one that is absent. Three fixture builders in spatial.rs's test modules also needed the column; that file belongs to the packing lane and only its #[cfg(test)] text was touched.

The distribution — the cheapest possible detector, MEASURED

End to end off disk, from a real convert of liechtenstein-latest.osm.pbf through the new column (2026-08-31, osm-katana 0.2.10, 0 nulls, uint8, non-null):

And the same ladder applied to the tags column of the live Sweden cut (/home/korp/.korp/work/sweden, cut 2026-08-31 — a projection of the rules over real tags, NOT a column read back, because that cut predates the column and its PBF is gone):

Two things the distribution says out loud, and DELTA 3 must hear both.

1. nodes.parquet is one rung. 97.67 % of Sweden's 103.5 M nodes carry no tags at all — they are the geometry vertices ways refer to — so they take UNCLASSIFIED_MIN_ZOOM and the whole table sits on 12. A (min_zoom, hilbert) sort of nodes.parquet is therefore very close to a no-op: the leading key is constant for 99.99 % of rows and the order degenerates to Hilbert. DELTA 3's 8.0× is a ways and roads claim, not a nodes claim, and measuring it on nodes would be measuring nothing.
2. relations.parquet has a real classification hole — 37.65 % unmatched with only 568 untagged rows. The unmatched keys, counted: type 5 505 729 · restriction 2 260 349 · boundary 717 019 · admin_level 599 853 · public_transport 552 642. Turn restrictions have no geometry to draw and belong nowhere on the ladder; boundary / admin_level genuinely do, and neither RULES nor facett's classify_with had an arm for them. The boundary arm landed 2026-09-01 (lane R) — see The boundary block below for the measured effect; the residual is type/restriction, which is correct. On ways the hole is 12.24 % and its top tagged keys are barrier 120 111 · leisure 50 211 (values outside park/garden/nature_reserve/recreation_ground) · man_made 49 390 · natural 47 058 (values outside the listed set) · power 34 058.

The boundary block — 2026-09-01 (lane R)

Six rules in the ONE table, above place for the coastline reason (a boundary relation that also carries a settlement label is still the boundary being drawn): admin_level 1–2 → Country, 3–4 → Region, 5–7 → District, any other level → City; boundary ∈ {national_park, protected_area} → Region; any other boundary → City. The level is asked before the key so a way segment tagged with the level alone lands where its relation does. osm-katana census --classes over the live cut, read only, both binaries built from this tree on 2026-09-01:

The relation hole closes by 4.9 pp — the 717 019 boundary rows almost exactly — and what is left is type/restriction, which has nothing to draw. On ways the arm is worth 0.03 pp because boundary WAYS in Sweden mostly carry a road or coastline tag that matched first; the 12.2 % way hole is barrier / man_made / power / the unlisted leisure and natural values, and 7.48 pp of it is untagged geometry. Two side effects seen in the census, both by the chosen order: relation Country drops 53 171 → 49 253 (city-boundary relations with place=city now take their admin_level rung, City) and District rises 1 323 595 → 1 372 216 (the kommun relations).

Seen RED (class.rs::a_boundary_is_drawn_at_the_rung_its_admin_level_earns, nine literal cases spread over three rungs plus a turn-restriction control): "classify_tier = None for a national border". tests/facett_ladder_fixture.rs went red on the six new lines, as built, and its literal moved in the same commit. facett's half is stale. facett-map/osm-katana-ladder.txt exists now (it did not on 2026-08-31) and its copy still had natural=coastline below place — i.e. it predates islet-1 too. The regenerated file was written into /home/rickard/git/facett/facett-map/ uncommitted; facett's osm_katana_ladder guard will report the boundary signatures as divergences (facett's classify_with paints boundary=administrative as an untagged generic line at City), which is a facett KNOWN_DIVERGENCES entry or an arm — facett's lane, not this one.

The gates, and the red arm each was proved on

The transcription, and the guard that now watches it

RULES is an arm-for-arm transcription of facett_map::style::classify_with + classify_label, in another repo, with nothing making the two stay in step. That is a LAW 5 twin, openly declared, and the by-construction fix (lift the kind+tier half into edda, both sides call one function) is still the migration target.

Landed 2026-08-31 (lane WIRE), the detector until then:

• examples/ladder_spelling.rs — the named command. Prints the whole ladder (rungs, unclassified rung, tiers, rules) as parseable text: cargo run -p osm-katana --example ladder_spelling > ../facett/facett-map/osm-katana-ladder.txt.
• facett checks itself against that checked-in fixture in facett_map::style::osm_katana_ladder, over the 142 most frequent tag signatures of the live Sweden cut. It parses the rule TESTS and evaluates them, so it is red for a facett change and for an osm-katana one.
• tests/facett_ladder_fixture.rs — the staleness half, and the reason the fixture is not just a copy that rots. It pins the generator's output against a literal, so editing RULES turns a test red HERE, with facett's path and the regeneration command in the message. Seen RED: moving place=town from Region to District ⇒ "rule 2 is now place∈[town]>9, the copy facett holds says place∈[town]>6".

What the guard found on its first run — one of them is ours

Seven divergences, six of them facett's known place hole (classify_with has no place arm, so a place=city WAY lands on City while the NODE lands on Country).

The seventh is a defect in RULES and it has 31 112 rows. The place=* block is FIRST and ends in the catch-all place? at City. An islet's shoreline carries both natural=coastline and place=islet, so the catch-all fires before the coastline arm is reached and the row is written min_zoom = 12 — while facett paints coastline at Lod::Country, rung 6. MEASURED on ~/.korp/work/sweden/ways.parquet (2026-08-31): 31 112 ways carry exactly {natural: coastline, place: islet}, plus 260 more with place=island. 31 372 rows of Swedish shoreline the server would withhold until camera 12 that the client is ready to paint at 6 — the direction the UNCLASSIFIED_MIN_ZOOM note above names as the dangerous one, reached by rule ORDER rather than by a rule.

The fix is one line — ask natural=coastline before the place? catch-all, or stop the catch-all claiming rows that carry a geometry class — and it moves rules_spelling(), which re-cuts every network table. Left for the lane that owns class.rs, recorded here and in facett's KNOWN_DIVERGENCES so it cannot be lost.

DELTA 1b — the buildings side table — BUILT 2026-08-31

buildings.parquet + buildings.json, cut by osm-katana buildings <extract> -o <dir>, written through the same spatial::pack_batches the networks use. It is DELTA 1's shape applied to a second concern: a table per concern, Hilbert-packed, covering.bbox, read through korp's existing query_ways_from seam.

Measured on korp's live Sweden cut (2026-08-31): 3 841 483 rows, 469 row groups, 297.5 MB, cut in 16.0 s on oden. 3 836 448 rows from ways.parquet, 5 035 from relations.resolved.parquet, 26 with no geometry, 11 439 building:part, 949 with a min_height_m.

The table carries DELTA 2's geometry_z6 / z9 / z12 (a footprint is an area, so spatial::generalised_rungs_for says yes), and they are most of the file: the SAME cut with --no-generalise is 154.3 MB in 11.3 s, i.e. +92.8 % of bytes and +4.7 s for the three coarse columns. Both runs measured on oden 2026-08-31; coverage is identical either way.

The headline, and it is the disappointing one

95.68 % of Swedish buildings carry no height information in OpenStreetMap at all. Only 4.31 % get a height, and only 0.2487 % of them are heights a human stated; the rest is building:levels × 3.2, an assumption with a ±20 % median error. A height column does not fill Sweden with skyscrapers — it fills central Stockholm, Gothenburg and Malmö and leaves rural Sweden as the default box it is today. That changes what the 3D lane should plan for: the interesting product decision is now what to DRAW for the 95.68 %, and the height_source column is what lets a renderer draw it honestly.

Not a defect of this table: of the 3 836 471 building rows in ways.parquet, 3 825 565 are WKB Polygon (99.72 %) and 10 905 are unclosed LineString. The z6 census's "all 57 160 buildings were Point" is a TILE defect, downstream of here.

The schema

id:Int64, source:Utf8 (way|relation — the two id spaces overlap, so (source,id) is the key), kind:Utf8 (the building= value), is_part:Boolean, min_zoom:UInt8 (free from DELTA 1), height_m:Float32?, min_height_m:Float32?, levels:UInt8?, height_source:UInt8 (non-null), geometry (via shared::schema::geometry_field()), tags:Utf8, plus covering.bbox from the packer. Row groups 8 192 — not tuned here.

height_source non-null is the whole point: Option<f32> cannot distinguish "surveyed", "assumed" and "somebody tagged nonsense", and BuildingStats::height_sources puts the histogram — every variant, including the zero buckets — into buildings.json so an operator reads the coverage without opening the parquet.

The parser's decisions, and why each one

shared/height.rs is the ONE writer of the height decision. Every rule was written against the measured value zoo of the Sweden cut, not against the wiki:

Gates, with the RED arm actually run

Two defects this lane FOUND and did not fix (not its files)

1. network::short_type has no UInt8 arm, so min_zoom, levels and height_source all spell as Other in every NetworkRecipe token — and that function's own doc says "Other is the one spelling that cannot distinguish two types". Retype DELTA 1's min_zoom from UInt8 to UInt16 today and not one recipe on disk moves. FIXED 2026-08-31 (lane TYPE) — and the one-line repair this entry proposed was the wrong one. See The recipe token could not spell UInt8 below: the arm was the symptom, the Other catch-all was the defect, and the re-cut this entry feared turned out to cost nothing at all on the live cut. The report was right; the diagnosis was one level too shallow, which is exactly why reporting it rather than patching it was the right call.
2. The brief's premise was wrong. "A grep across facett's 3D map sources for a height/levels consumer returns nothing" — it does not. facett_map::parse_building_height_m (facett-map/src/lib.rs:398) reads height and building:levels today, per Way, off the JSON, in korp::map_paint::classify_ways. See the hand-off below for what that changes.

HAND-OFF — making the extruder read the table

Everything below is surveyed, not guessed (2026-08-31). The next lane should not have to re-derive any of it.

What the extruder does today.

• facett-map3d/src/extrude.rs:2670 — let h = prism_height_for(&ring, w.kind, w.height_m, cx.du_per_m, w.osm_id, cx.height);
• facett-map3d/src/extrude.rs:2366 — fn building_design_height(height_m: Option<f32>, du_per_m: f32, osm_id: i64, ring: &[(f32,f32)]) -> f32, and the decision is lines 2367–2370:

let (tagged, base_m) = match height_m {
    Some(m) => (true, m),
    None    => (false, DEFAULT_BUILDING_LEVELS * METRES_PER_LEVEL),  // 2.0 * 3.0 = 6 m
};

then ±25 % deterministic hash jitter on the untagged case only (:2381), a cap at MAX_DEFAULT_WALL_OVER_FOOTPRINT (0.9) × footprint short side, a floor of MIN_DEFAULT_BUILDING_HEIGHT_M = 2.5 m, and × du_per_m into design units (:2403).

• Constants: facett-map/src/lib.rs:387 METRES_PER_LEVEL = 3.0, :391 DEFAULT_BUILDING_LEVELS = 2.0.

Where Way::height_m comes from today.

• facett_map::Way::height_m: Option<f32> — facett-map/src/lib.rs:336
• set at facett-map/src/lib.rs:633: let height_m = if style.kind == WayKind::Building { parse_building_height_m(tags) } else { None }; inside from_lonlat_styled, reached via from_lonlat_with (:545).
• parse_building_height_m — facett-map/src/lib.rs:398. It takes the leading run of digits and dots, which is why height=1;12 is a 1 m building, height=25' is a 25 m building, height=102,5 is 102 m, building:levels=3-4 is 3 levels, and height=99999 is a 99 999 m spike — all silently. min_height, roof:levels and est_height are read nowhere in facett, korp or osm-katana.
• the tags reach it as a per-Way serde_json parse: korp::map_paint::classify_ways (korp/src/map_paint.rs:394) → classify_rows (:428) → facett_osm::parse_tags_into (facett-osm/src/lib.rs:342) → Way::from_lonlat_with (:442).

The change, precisely.

1. korp reads the table. korp::mapquery::query_ways_from (korp/src/mapquery.rs:696) → query_ways_selected_from_exact (:741) already does the row-group covering.bbox prune and the WKB decode, so buildings.parquet needs no new reader — it is the same shape as roads.parquet. The discovery/serve side is korp/src/map_networks.rs: Network enum (:90), table() (:118), networks_dir (:489), manifest_path (:494, → networks.json), expected() (:523), availability (:570), read_overlay (:1339) + read_one (:1472), merge_networks (:1228), and korp::pipeline::networks_state (korp/src/pipeline.rs:1095). A buildings entry goes in each; buildings.json is the manifest equivalent, and it already carries the height_sources histogram a scan_scale_1024-style reader would want. Producer side is already done: osm-katana buildings <extract> -o <dir>.
2. MapWay carries the columns. korp::mapquery::MapWay (korp/src/mapquery.rs:208) is lonlat + raw tags: String + osm_id + network: u8 + tile. Add height_m: Option<f32>, min_height_m: Option<f32>, height_source: u8 — read from the parquet columns, never re-derived from tags. That is the whole point: it deletes a serde_json-fed height parse from the per-frame path, and parse_tags is 55.9 % of a tile read's allocations.
3. classify_ways stops parsing. korp/src/map_paint.rs:442 calls Way::from_lonlat_with(...); give facett_map::Way a constructor (or a post-set) that takes height_m from the row rather than from parse_building_height_m. When the row has no height_m, the existing fallback in building_design_height runs unchanged.
4. building_design_height gains the stamp. Option<f32> cannot say WHY a height is missing. Passing height_source in lets the 95.68 % absent case keep today's jittered 6 m fallback, the 4.07 % derived_from_levels case be drawn as a real height (optionally flagged in a legend as assumed), and the 0.25 % tagged_metres/tagged_feet case keep the existing "a tagged height is never capped" rule. rejected should be drawn like absent and COUNTED — a spike in it is a parser regression, not a mapping change.
5. min_height_m unlocks parts. facett-map3d-flight/src/extrude_gpu.rs:43's Footprint and extrude.wgsl:31's BuildingMeta already have height and min_height fields and nothing populates them from OSM. The CPU extruder (extrude_prism_based, extrude.rs:3727) takes a base_y already, so lifting a part is a value, not a new code path. 949 Swedish rows need it — small, so it is optional in v1, but is_part must be honoured either way or the outline and its parts z-fight.
6. The two METRES_PER_LEVEL become one decision. After step 3, facett's 3.0 is only the DEFAULT_BUILDING_LEVELS × METRES_PER_LEVEL fallback for an unmapped building and shared::height::METRES_PER_LEVEL = 3.2 is the only converter of a mapped building:levels. Until then they overlap, and a way built from a TILE gets 3.0 while a row of buildings.parquet gets 3.2 for the same building.

The renderer, for whoever asks. facett-map3d/src/map3d.wgsl:45 VsIn is pos: vec3 / nrm: vec3 / col: vec4 and facett-map3d/src/gpu.rs:43 Vertex matches: there is no per-instance or per-vertex height attribute, because the height is baked into pos.y on the CPU. So none of the above touches a shader. (LAW 4 holds: everything is wgpu; facett-core/tests/webgl_free_law.rs and korp/tests/law4_no_opengl_backend.rs ban the alternative.)

A number the next lane will want. facett-map3d/tests/wall_roof_ratio.rs:17 measured 69 of the embedded Sigtuna extract's 10 373 buildings — 0.7 % — carrying height or building:levels. This lane measured 4.31 % over all of Sweden. Both are right: Sigtuna is a small town and the national figure is carried by the three big cities. Whatever the extruder does for the untagged case will still be what the map mostly looks like.

The recipe token could not spell UInt8 — FIXED 2026-08-31 (lane TYPE)

Reported by lane HEIGHT (above) as "one line in network.rs". It is not one line, the line it proposed fixes today and not the next change, and the re-cut everyone was afraid of does not exist. All three of those are measured below.

What was actually broken

Two private copies of the same 15-line match — tiles::short_type and network::short_type — both ending in _ => "Other". Every recipe token in the crate is built as name:short_type(data_type) per column, so Other is a spelling that eight different Arrow types share. Seen red, by restoring the catch-all and running the new guard:

         Other <- [Int16, UInt8, UInt16, UInt32, UInt64, Date32,
                   Duration(Second), Time32(Second)]
min_zoom under five types: ["min_zoom:Other", "min_zoom:Other", "min_zoom:Other",
                            "min_zoom:Int8",  "min_zoom:Other"]

Three live columns sat inside that collision: DELTA 1's min_zoom and DELTA 1b's levels and height_source. The applied-output arm is the one that matters — a real parquet written with today's schema and today's stamp, judged against the token for the same table with min_zoom retyped to UInt16:

a UInt8 → UInt16 retype of `min_zoom` did not move the token, so this table reads
CURRENT with a column that is no longer the width the recipe implies
  left : …,min_zoom:Other,geometry:Binary,…
  right: …,min_zoom:Other,geometry:Binary,…

Byte-identical. That is collauto-6 with a different column, and ladder-1's shape (korp ebf46d1) with a different judge.

The twin had already drifted, too. The doc on network::short_type said it "mirrors tiles::short_type deliberately: two recipes an operator diffs side by side must spell Int64 the same way" — and by 2026-08-31 network's copy had an Int8 arm for the graph's oneway that tiles' copy did not. A doc comment is not a mechanism.

The decision, and the two options that were rejected

• Rejected — retype the columns (min_zoom to UInt16) so they land on arms that already exist. Cheapest today and wrong in principle: it makes the type of a column a function of what the digest happens to spell, and it leaves the hole open for the next UInt8 column. It also would not have been free — see the cost section; the free window is free for a different reason than anyone thought.
• Rejected — refuse loudly. A recipe that cannot spell a column is a recipe that should not be written, so make an unspellable type a hard failure at write time. Genuinely attractive, and still wrong: it buys nothing over a total spelling and pays for it with an outage. The only way to reach it is a developer adding a column of a new type, and the failure would land in the middle of a 13-minute planet cut instead of at the keyboard.
• Taken — ONE spelling function, with no catch-all at all. shared::schema::short_type is now the single writer (LAW 5); tiles.rs and network.rs both call it and neither owns a copy. Named arms cover every type the crate puts in a schema; anything else spells as its own Debug form, so it is injective on the discriminant and no future UInt8-shaped hole can open. Adding an arm is from now on an optimisation of the spelling, never a correctness fix.

The deciding argument for the fallback over the refusal is the direction of failure. Debug is not a stability contract of arrow, so an upstream rename would move every token — and a moved token means Stale, which re-cuts. The Other catch-all failed the other way: a false Current, serving data that no longer matches its own recipe. A spurious re-cut of the network family costs 30.5 s (measured below). A false Current costs correctness, and you do not find out.

Struct and List were still spelled without their children — the children of bbox are fixed by GeoParquet 1.1 and spelling them out makes the token longer — and that was the one accepted collision left: Struct(a) and Struct(b) read alike, so a retype of a bbox leaf from Float32 to Float64 was invisible.

CLOSED 2026-09-01 (lane R). GeoParquet fixes the leaf NAMES, not their width (1.1 allows either), and a covering half the width the recipe implies is exactly the false-Current the token exists to prevent. short_type now recurses: bbox:Struct{xmin:Float32,ymin:Float32,xmax:Float32,ymax:Float32}, node_refs:List<Int64>, all the way down. NetworkRecipe::for_table spells the packer's own spatial::bbox_field() instead of a literal Struct, and tiles::tile_fields reads its leaf fields off the same function rather than declaring a four-line twin (LAW 5). Seen RED with the childless arms: Struct == Struct for the two coverings, and the network token spelled bbox:Struct.

The price, said plainly: every token that carries a bbox moves once. The tile pin the_tile_columns_spell_exactly_what_the_cut_pyramid_declares moved from bbox:Struct to the recursive spelling and every tile of the live pyramid (1 635 971, 44.45 GB) reads Stale one time — a 222 s write + 5 min 35 s upload on korp's next run, on a pyramid v2.0 is deleting anyway — and the seven packed network-family tables re-cut once, 30.5 s. That is a one-time, fail-safe move (Stale → re-cut), against a permanent false-Current on the next covering retype.

The cost — MEASURED on the live Sweden cut, 2026-08-31, read-only

/home/korp/.korp/work/sweden, while korp-server was serving from it. Every token below was read out of the object's own footer and compared against the token this build produces, and against the token the Other spelling would have produced.

And those four were already Stale before this lane touched anything. The tables on disk were cut on 2026-08-31 06:15 by pre-DELTA-1 code: no min_zoom column, no geometry_z6/z9/z12, and no minzoom= ladder term in the token. The verdict against this build is STALE — and it is STALE under the Other spelling too. So:

The incremental cost of this fix on the live Sweden cut is ZERO additional re-cuts. The window lane HEIGHT described is real, and it is wider than it looked: the four tables that would have had to be re-cut are already being re-cut for three other reasons.

The cost if the window is ever closed is now a measured number rather than an inherited one. A full osm-katana networks re-cut of the Sweden extract, oden, 2026-08-31:

wall 30.5 s · 10 831 048 rows · 831 MB · 4.71 GB peak RSS
  roads 2 277 368 · rails 29 683 · trails 75 694 · rail_nodes 64 545
  graph_edges 4 689 310 · graph_nodes 3 694 448

The 21.3 s for 10.8 M rows quoted throughout this document (DELTA 2, and shared/schema.rs) is the same six tables and the same row count — 10 831 048 exactly — but it predates the generalisation and min_zoom columns those tables now carry. 30.5 s is the current figure; the row count was right all along.

The gates, and the red arm each was seen on

All four red arms were produced by restoring _ => "Other" in shared::schema::short_type and running the suite, not by argument.

The fifth, the_tile_columns_spell_exactly_what_the_cut_pyramid_declares, is the opposite kind of guard: it must pass under BOTH spellings, because its job is to prove the refactor moves no tile token. It is pinned against the literal string a real tile on the live cut declares (tiles/z6/36/19.parquet), not against a recomputation — a token test that builds both sides from the same function is the korp_tile_options().zooms == DEFAULT_TILE_ZOOMS trap, and this file has paid for that shape before.

One thing this lane found on the way past — relations.parquet is the PLANET's relations

Open question handed over by lane HEIGHT: relations.parquet holds 14 632 472 rows for a Sweden cut against 274 206 in relations.resolved.parquet, a 53× gap. Settled from the footers plus a one-row-group census (2026-08-31), and it is neither a row-count bug nor a naming confusion:

• 14 632 472 distinct ids, min 1, max 21 209 926 — that is the whole planet's relation set, not Sweden's.
• Its geometry column is 100 % null (0.0 MB compressed over the whole file). The bulk is members 750.2 MB + tags 255.3 MB + id 23.1 MB.
• This is by construction of the region clip: clip_keeps filters NODES at decode and a way is dropped when none of its nodes survived (reader.rs). A relation carries no coordinates, so there is nothing for the clip to test and every relation passes through. relations.resolved.parquet is the geometry-bearing product — the 1.87 % whose members resolved inside the clip.

So the names are honest. The cost is real and is worth someone's lane: ~1.005 GB of members+tags written for relations that can never resolve, and 14.36 M rows the resolve stage scans to find 274 k. A prefilter — keep a relation if ANY member id survived the clip — is a strict superset of what resolves, so it is safe, and it would remove ~98 % of that. Not fixed here: it is reader.rs/relations.rs, not this lane's files.

The relation clip prefilter — BUILT 2026-09-01 (lane R)

The rule is the admission rule relgeom::resolve_relations already applies, moved to the writer and made the ONE keep-decision (reader::RelationClipOracle, LAW 5): a relation is written when ANY member survived the clip — a way present in the cut's ways.parquet, a node present in the node store, or a relation that was itself kept. The third is a fixpoint, because route_master binds routes and has no way members of its own; it runs on the collector (RelationClipOracle::settle), 8 laps like relgeom's.

The mechanism, and why it is two sub-passes rather than a filter in the loop. Ways are decided in the same pass 2 that decodes relations, and a relation's way members are in other blobs on other workers — so at the moment a worker holds a relation, nobody knows the way set yet. A planet file orders every way before every relation, and pass 1 already probes each blob for its group kinds (reader::BlobKinds, replacing the one bool), so with a clip active read_resolved_pbf runs pass 2 twice over the file: the way sub-pass opens only the blobs the plan saw a way group in (BlobPlan::needs_kind) and reports the ids it kept; the relation sub-pass is opened at the FRAME of the first relation blob (BlobPlan::first_rel_blob → pbf_gatling_run_typed_from + SlotCursor::at) and judges every relation against the way ids and the node store as it decodes, before the members are serialised — a dropped relation costs one verdict and no JSON. Without a clip there is one pass and every relation is written, byte-identical. The XML twins (.osm / .osm.bz2 / .osm.gz with a clip) were NOT prefiltered by this lane — and "the planet is PBF" was wrong: node_coords.provenance of the live Sweden cut names /home/rickard/planet/planet-latest.osm.bz2. The bz2 reader is the production path. See the next section (lane W).

One trap paid for on the way: BlobPos::offset — the key every plan is written in — is the compressed PAYLOAD, not the frame, and align_and_collect treats offset 0 of a slice as a boundary without looking. Opening the stream at a payload offset gives a slot no frame parses in and a sub-pass that silently decodes nothing (seen: rels: 0, no blob opened). BlobPos now carries frame too, and only that is ever handed to a seek.

MEASURED on the live Sweden cut, 2026-09-01, read only — the oracle replayed over the real relations.parquet (the pre-prefilter cut of 2026-08-31, zstd) against the real ways.parquet and nodes.parquet id columns, the kept rows written as a real zstd parquet through writer::write_relations, 120.6 s wall in release (reader::relation_clip_oracle_law::the_prefilter_over_the_live_sweden_cut, #[ignore]):

Where the 14.36 M rows went: 14 145 157 dropped on the worker (no node or way member survived), 215 184 held pending for the fixpoint, of which 2 075 were promoted through a kept relation member and 213 109 dropped. The superset proof, on live data: every one of the 274 206 ids in the live relations.resolved.parquet is in the kept set — and the kept set is exactly that size, so on Sweden the prefilter is not merely a superset of what resolves, it IS what resolves. The oracle (9 325 142 way ids + a 103 488 583-node store) builds in 7.1 s.

What was NOT measured: an end-to-end planet cut with this binary (the planet PBF is not on oden today), so the wall-time delta of the second file open is a claim, not a number. The relation sub-pass reads from the first relation blob to EOF — the planet's relation tail — and leaves every way blob closed; the way sub-pass leaves every relation blob closed. On the 2026-08-31 cut those were 1 787 relation row groups the resolve stage will no longer scan.

Seen RED (tests/relation_clip_prefilter.rs, nine relations whose fate is a literal, cut as a real .osm.pbf with a bbox clip and read back off disk): "relations.parquet holds {1, 2, 3, 4, 5, 6, 7, 8, 9}, expected {1, 3, 4, 7, 9}". Its controls: an unclipped cut still writes all nine, and resolve_relations over the prefiltered table resolves relations 1 and 9 through way 10. The unit half (relation_clip_oracle_law) pins every verdict arm and the two-lap fixpoint with literals.

The prefilter reaches the XML readers — BUILT 2026-09-01 (lane W)

The Sweden cut oden serves was cut from planet-latest.osm.bz2, so the 14.6 M-row relations.parquet lane R measured was written by read_resolved_bz2, which the PBF prefilter never touched. Same rule, same oracle, same fixpoint — now the same orchestration: xml_reader::run_pass2 is the ONE place pass 2 is split into the way sub-pass and the relation sub-pass (LAW 5), and all four readers hand it a closure that only says how to reach the bytes:

Three things moved to make that by construction rather than by a copy:

• RelationClipOracle::admit is where a decoded relation meets its verdict, for BOTH decoders (reader::parse_ways_rels for a PBF block, xml_reader::parse_chunk for an XML element). make runs only for a row that leaves it, so a dropped relation still costs no JSON on either path; parse_relation is split into parse_relation_parts for that reason.
• the plain .osm reader's serial pass-2 sink (Pass2ResolvedCodec / Pass2ResolvedSink, WayWriter::push per row on the collector) is deleted. It now runs RawXmlPass2Codec → pass2_parse_decoded → Pass2TypedSink, the path bz2/gz/pbf already ran. A second sink would have needed a second copy of the sieve, the fixpoint and the ledger.
• the collector's stub path (Pass2TypedSink::flush_stub) carries the sieve too: PBF has no boundary stubs, XML does, and a relation straddling a decode-slot boundary resolves on the collector and must meet the same oracle.

pass2_wants indexes only the element kinds a sub-pass writes, so the way sub-pass no longer parses every relation it was about to discard (it did: parse_chunk parsed them and tables.rels dropped the vector).

MEASURED, Liechtenstein (lbzip2/test_data/liechtenstein.osm.bz2, 5.49 MB), a bbox clip over Vaduz, tests/relation_clip_prefilter.rs, 2026-09-01:

20.0 % of the rows and 37.2 % of the bytes kept; the bz2 and pbf cuts hold the SAME 188 ids (the_bz2_reader_agrees_with_the_pbf_reader_on_a_real_extract — the workspace has no bzip2 encoder, so the bz2 arm asks the stronger question on a real extract instead of the nine-relation fixture). The Sweden number is lane R's replay, 14 632 472 → 274 206 rows, and now applies to the cut that is actually made.

Seen RED, in order, before any implementation: the_xml_readers_prefilter_too — "[osm] relations.parquet holds {1, 2, 3, 4, 5, 6, 7, 8, 9}, expected {1, 3, 4, 7, 9}" and the same for [gz]; the_bz2_reader_agrees_… — "bz2 clipped cut wrote 938 relations, the unclipped cut 938 — nothing was filtered". The unclipped control now runs on every format. The pass-2 done line carries rel_pass_start_offset for bz2 and .osm too, so where a sub-pass started is a logged fact.

--version — BUILT 2026-09-01 (lane W)

osm-katana --version prints osm-katana v<semver> (<short-hash>), the build-version law's one shape, from osm_katana::VERSION_LINE. build.rs reads the hash PURE-FS — .git/HEAD, the loose ref (worktree or common dir), or packed-refs — which is korp's build_provenance::head_sha pattern carried over verbatim in shape; unknown for a checkout-less tarball, never a failed build.

islet-1 rode along — because the re-cut was already happening

Reported by lane WIRE against shared/class.rs, deliberately not fixed there for the same reason lane HEIGHT did not fix short_type: it moves rules_spelling(), which is a term of NetworkRecipe::token(), so every network table is judged Stale. Since this lane was already moving those tokens and the incremental cost is measured at zero, the two fixes are one re-cut and it landed here.

RULES is order-sensitive — tier_of_tags takes the first arm that matches — and the place block ends in a place? catch-all at City. natural=coastline sat below it, so an islet's shoreline matched place first and was written min_zoom = 12, while facett paints coastline at rung 6. The geometry a z6–z9 view wanted had never been written, and a client cannot ask for what the server never wrote — the direction class.rs's own doc calls the dangerous one.

MEASURED independently on the live Sweden cut, 2026-08-31 (ways.parquet, 9 325 142 rows):

natural=coastline ways                    79 084
  … of which also carry a `place` tag     32 049   (40.5 %)
      place=islet                         31 761
      place=island                            286
      other place values                        2

Two fifths of Sweden's shoreline. (Lane WIRE's 31 112 / 260 is the same defect counted on a narrower population — ways whose tags are exactly {natural, place}. Both are right; 40.5 % of all coastline ways is the number that says what was lost.)

The fix is the ORDER, and the order is the rule: a tag that says what the geometry IS beats a tag that says what to call it. natural=coastline is now the first rule in the table. Guarded by an_islets_shoreline_is_drawn_at_the_coastline_rung, which asserts the consequence rather than a rule index — an order test that re-derives the order from RULES is RULES compared with a copy of itself. Red arm seen 2026-08-31: with the rule moved back below place, the combined case reads left: 12, right: 6. The test carries controls (a bare place=islet must stay at 12) so it cannot pass by everything being 6.

osm-katana/tests/facett_ladder_fixture.rs went red on this, exactly as lane WIRE built it to — rule 0 is now natural∈[coastline]>6 — and its literal is updated in the same commit. Its other half does not exist yet: there is no facett-map/osm-katana-ladder.txt and no facett_map::style::osm_katana_ladder in /home/rickard/git/facett at 52e449e6. Nothing is stranded today, but whoever lands the facett half must generate it after this commit.

Observed 2026-08-31, FIXED 2026-09-01 (lane R)

NetworkRecipe::for_table put the minzoom= ladder term in every token, including graph_edges and graph_nodes, which carry no min_zoom column at all. A ladder edit therefore re-cut the two graph tables — 8 383 758 rows on the Sweden cut — for a change that cannot appear in them. An over-invalidation fails safe, so it was recorded rather than repaired; it is repaired now because the boundary arm below IS a ladder edit, and the two graph tables would have paid for it a second time.

min_zoom_ladder is Option<String>, Some exactly when the field list carries min_zoom — decided in for_table, the same place the column axis is decided, so a future table cannot get the term by default. The token has no minzoom= term at all for such a table rather than an empty one. The network tokens are byte-identical; the two graph tokens lose the term once. Guarded by network::tests::a_table_without_min_zoom_carries_no_ladder_in_its_recipe on the token as built AND as read back off a real cut's footer; seen RED with the unconditional term: "graph_edges token carries a minzoom= term for a column it does not have".

DELTA 2 — generalisation columns — LANDED, MEASURED 2026-08-31

geometry_z6, geometry_z9, geometry_z12, geometry (full precision).

The simplification is the per-ring Douglas–Peucker tiles.rs ALREADY runs at each rung — the same code, same tolerances, written into a column instead of smeared over 1 635 971 files. Per-ring, not over the flat vertex list: concatenated rings put a chord between the outer ring's last vertex and the first hole's first vertex, and DP then measures deviation from an edge the geometry does not have. tiles.rs learned that already; the column writer inherits it rather than re-deriving it.

All four columns go through shared::schema::geometry_field() (LAW 5) or they are four twins that will drift apart the first time the extension type changes.

Do not add a fifth rung. korp's level type is a four-variant enum; a fifth column forces a fifth variant, which is the trap the z9 lane hit on 2026-08-24. Only THREE are derived — geometry IS rung 15, under the name every existing reader already uses; at z15 the epsilon is ≈ 0.6 m and a derived column there would be a byte-for-byte duplicate of the largest column in the file.

Where it lives: shared/generalise.rs (the one simplifier — it calls tiles::simplify_polys, tiles::douglas_peucker_into and tiles::tile_pixel_tol), written by spatial::pack_batches per row group, alongside covering.bbox and through the same packer. --generalise false is the RED arm on the CLI.

What it fixed in the instrument first

The pyramid's z12 number that this whole delta was aimed at never decoded any tile geometry. phase0/phase0b project only the bbox.* leaves out of a tile, so the "15.5 ms z12" the plan set a 2× target against is a PRUNE, not a read. Measured with geometry actually projected, same 42 tiles, same run:

The pyramid hands back MORE WKB at z12 (60.08 MB) than the base table does at full precision (52.38 MB), because of tile fan-out duplication.

The gate, ways.parquet, central Stockholm (17.85, 59.22)–(18.30, 59.45), warm, one run

Prune and decode reported separately, as PACK did: the prune arm projects only bbox, the decode figure is the difference.

Verdict, said plainly. The literal target — ≤ ~31 ms, i.e. 2× the pyramid's 15.5 ms — is NOT met: 96.4 ms. It is not reachable by any column: the prune alone is 15.6 ms, so a decode of zero bytes would still land at 1.0× the target's basis. Against the comparable pyramid read (121.4 ms) the column path is 0.79×, i.e. faster, and at z15 it is 3.1× faster (134.6 ms against 418.2 ms) with one file open instead of 1 764. Arm by arm:

• prune vs prune — parquet 15.6 ms, pyramid 17.1 ms → parquet wins
• decode vs decode — parquet z12 80.8 ms, pyramid 104.3 ms → parquet wins

geometry_z12 is worth 1.40× on wall and 1.49× on WKB bytes against full precision, not the 5× the 15.5 ms basis implied. Why it is only 1.4×, measured: tile_pixel_tol(12, 2048) is 4.29e-5° ≈ 4.8 m, and Swedish OSM ways are short and sparse — mean 13.35 vertices at full precision, 8.27 at z12. The epsilon is derived from ONE tile drawn across a 2048-px viewport, but a z12 Stockholm viewport shows ~7 tiles across, so the tolerance is about 7× finer than the screen can resolve. Raising DEFAULT_TILE_RASTER_PX's basis is the open knob; it rides in TileRecipe and would re-cut the pyramid, so it was NOT touched here.

The z6 byte gate, whole-Sweden viewport

Compressed bytes the projection touches, geometry_z6 against geometry:

The ≥10× gate passes on the relation table and fails on the way table, and the reason is structural, not tunable: every row keeps a cell, and a WKB cell has a floor — 21 bytes for a degraded point, 41 for a two-vertex line. At z6 ways.parquet is already down to a mean of 1.52 vertices per row, with 5 609 616 of 9 323 619 rows reduced to a bare Point. The column is within ~1.3× of its own floor; no epsilon gets it to 10×. The way to go further is to NULL the rows a z6 camera would not draw — that is min_zoom (DELTA 1) plus the band sort (DELTA 3), not a coarser epsilon.

The simplification is visible and correct — on real Swedish geometry

Full-file census of the packed Sweden tables (not a fixture):

(The 13.35 cross-checks the 13.3 vertices/way tiles.rs measured in the source extract on 2026-08-03 — two instruments, two methods, same number.)

A lake keeps its island. ways.parquet has zero holed areas — a way is one closed ring, holes come from assembled relations — so the gate was run where holes live. Of relations.resolved.parquet's 78 850 areas with at least one hole / 462 608 inner rings:

At z6 the epsilon is ≈ 300 m, so most Swedish islands genuinely ARE sub-pixel; the drop is the tolerance, not a lost hole. Seen red (LAW 2): flattening the rings in generalise_wkb_into makes both island guards fail with the seam chord visible in the failure output.

Cost — measured, not buried

ways.parquet, same binary, same input, only --generalise differing:

Per column, compressed: geometry 577.15 MB · geometry_z12 390.87 MB · geometry_z9 165.87 MB · geometry_z6 91.39 MB. relations.resolved.parquet: +6.67 s vs 3.94 s, 192.5 MB → 366.6 MB (+90.4 %); geometry 176.56 → z12 122.08 / z9 44.88 / z6 7.06 MB. Peak RSS is unchanged (4.97 GB vs 5.00 GB for ways) — the columns are built per row group at write time, never retained for the table.

Against what they replace, this is cheap. The pyramid is 44.45 GB across 1 635 971 files and a 222 s write step. The generalisation columns on both display tables cost +822.7 MB, i.e. 1.9 % of the pyramid's disk, and +8.7 s against its 222 s.

nodes.parquet gets no rung columns: a point has no vertex DP can drop, so three columns there would be three byte-for-byte copies of 655.18 MB. The graph tables opt out too — a router reads an edge's geometry for its LENGTH, and a coarser copy is a wrong length.

Recipes

• TileRecipe does NOT move. The columns are appended to the base tables by the packer, never to a tile; tile_fields() is untouched. Pinned by the_generalisation_columns_do_not_move_the_tile_recipe. No re-cut of 1 635 971 tiles.
• NetworkRecipe DOES move, deliberately: it spells the packer's contributed columns into its token, so a table cut before DELTA 2 and one cut after must not share a token. Every cut network table reads Stale and is re-cut — 21.3 s for 10.8 M rows at the time this was written; re-measured 30.5 s for the same 10 831 048 rows on 2026-08-31 once the rung and min_zoom columns were in the cut (lane TYPE).
• is_packed is now layout-aware: a table with covering.bbox but no rung columns reports false so the upgrade actually runs. That is only safe because pack_batches now DROPS the derived columns on the way in (carry_columns) and rebuilds them, so a re-pack is an upgrade and can never produce two columns called bbox.
• verify --digest is unmoved: the rung columns are excluded from the content digest for the same reason covering.bbox is — a pure function of geometry carries no independent content. Both arms pinned, both seen red.

One thing this lane found on the way past

digest.rs had no UInt8 arm, so the moment DELTA 1's min_zoom reached the base schemas (8b50546) digest_file bailed on every freshly cut nodes/ways/relations table — i.e. verify --digest, the only equivalence gate a pack has, was dead on the pipeline's own output. Found by this lane's digest test, fixed here, and guarded by a_uint8_rung_column_is_hashed_and_changing_it_moves_the_digest, whose second arm exists because the obvious repair (add UInt8 to the skip list) would make the error go away while hashing nothing.

DELTA 3's PREREQUISITE — the base tables are packed at all (LANDED 2026-08-31)

DELTA 3 says "band first, Hilbert within the band". It could not be written, because on 2026-08-31 the base tables were not Hilbert-sorted at all. MEASURED on korp's live Sweden cut that morning:

• ways.parquet schema was id, geometry, tags, node_refs, version — no bbox column existed, so there was nothing for a reader to prune on;
• 9 325 142 rows in 32 row groups (~291 000 rows each);
• the central-Stockholm viewport (17.85, 59.22)–(18.30, 59.45) was therefore a full scan: 902.13 MB, 3 538.2 ms.

Meanwhile networks/roads.parquet, packed by spatial::pack_batches since step e existed, answered the same viewport from 30 of 278 row groups. The layout was not missing — its application to the base tables was.

What was built

No new packer (LAW 5). The existing spatial::pack_batches — 31-bit Hilbert, covering.bbox, BYTE_STREAM_SPLIT on the bbox leaves — is now reached from three more places:

spatial::PACK_ROW_GROUP_ROWS = 8_192 is now the ONE source of the row-group size; PackOptions::default, NetworkOptions::default and both CLI defaults read it. A literal 8192 in a caller was a second writer of the one axis that decides pruning.

Two supporting pieces:

• spatial::is_packed — footer-only probe for the bbox column, so packing is idempotent. Without it a re-run or resume would hand pack_batches a schema that already has bbox and get a file with two columns of that name: valid parquet, which every reader resolves positionally, i.e. a layout fork no row count or digest would show.
• spatial::pack_in_place — packs a table over itself through PendingOutput, so the original survives a pack that dies half-way.

A pass, not a fold — and why

The brief preferred folding. It does not fit, and the reason is not effort:

A Hilbert order is global — the first output row can depend on the last input row — so a streaming writer cannot emit it without buffering the whole table. crate::writer's writers flush every ROW_GROUP_SIZE = 524 288 rows; sorting only within a flush would order rows inside a window that already spans the whole extract, which is exactly the layout that prunes nothing.

So the buffer is unavoidable either way. What a separate pass avoids is paying for it on top of convert's own working set — pass 2 holds the 1.66 GB mmap'd node store and every PBF decode buffer — and it lets the tables be packed one at a time, so the peak is the LARGEST table rather than their sum. MEASURED peak RSS: 15.6 GB (nodes), 4.8 GB (ways), 0.9 GB (relations.resolved). Folded into convert those would have stacked.

The price of the pass is one extra read of a file written seconds earlier, hence in page cache. It is in the cost table below and it is small.

THE GATE — measured end to end on the real Sweden cut

oden, 2026-08-31, same viewport, all rows in one warm-cache run so the arms are comparable to each other. (The morning's cold-cache figures for the pyramid were z12 30.5 ms / z15 328.9 ms; both are given because the difference is cache state, not code.)

Against the before: 33.3× fewer bytes and 24.6× less wall on ways.parquet.

Against the target — "within 2× of z12, beating z15" — the honest verdict is split:

• the prune itself passes: 18.5 ms is 1.19× z12's wall, and 4.21 MB is 7.5× FEWER bytes than z12 touches;
• with full-precision geometry decoded it does not: 144.0 ms is 9.3× z12 and 1.20× SLOWER than z15 on warm cache (it beats the morning's cold z15 by 2.28×). It still touches 2.84× fewer bytes than z15 and opens 1 file instead of 1 764.

The residual is attributable and it is not the layout: 18.5 ms of the 144.0 ms is the prune, and the other 125.5 ms is zstd-decompressing and Arrow-decoding 52.38 MB of full-precision WKB for rows the pyramid had already simplified at cut time. That is what DELTA 2 removes — a z12 read should touch geometry_z12, not geometry. The gate as stated cannot be met by packing alone, and no row-group tuning closes a 125 ms decode.

The cost, in full

Superseded on disk by The covering column now costs half, below: the same tables now pack to 2 711.33 MB (+50.5 %). The wall times and peak RSS above are unchanged — the fix is in what the covering leaves store, not in how they are produced.

78.3 s against a 3 360 s cut is +2.3 %, and v2.0 deletes step f (222 s of tiling) outright. Time is not the problem. Disk is, and the per-column census says why — compressed bytes, input order → Hilbert order:

  ways.parquet     902.13 MB → 1171.55 MB   (+29.9 %)
    node_refs      239.12 MB →  438.52 MB   +199.40 MB   <- 74 % of the growth
    bbox (x4)            —   →   72.18 MB    +72.18 MB
    id              19.05 MB →   30.74 MB    +11.68 MB
    tags            47.17 MB →   33.94 MB    -13.24 MB   <- locality HELPS here
    geometry       593.89 MB →  591.30 MB     -2.59 MB   <- unchanged

  nodes.parquet    709.05 MB → 1726.95 MB  (+143.6 %)
    bbox (x4)            —   →  693.94 MB   +693.94 MB   <- 68 % of the growth
    id             164.39 MB →  324.12 MB   +159.73 MB
    geometry       514.93 MB →  655.18 MB   +140.24 MB
    tags            19.72 MB →   26.08 MB     +6.36 MB

Two things fall out, and neither is the row-group size — which is the answer the tuning guides would have given:

1. The growth is id-delta locality, not small row groups. geometry on ways.parquet is unchanged at 8 192 rows per group (−0.4 % across 1 139 groups), so the fine groups cost essentially nothing. What costs is that id and node_refs are near-sequential in OSM order and near-random after a spatial sort. The mitigation is already in the schema and already wanted by v2.0: ways_schema(include_node_refs: false) drops that column, and by the census above that takes ways.parquet from +29.9 % to +10.6 % (663.01 → 733.03 MB). A map-delivery cut needs node_refs for nothing; only relation ring assembly and verify read it.
2. On a POINT table, half the covering column is a byte-exact duplicate. nodes.parquet's bbox.xmin and bbox.xmax chunks are the same size to the byte (198 873 981 each), as are ymin/ymax (148 093 832) — for a point they hold the same values. That is 346.97 MB of pure redundancy on Sweden alone, 20 % of the packed file. CLOSED — see The covering column now costs half, below.

The covering column now costs half (2026-08-31)

The duplication is real at the VALUE level, not only at the chunk-size level: sampled on the real file, xmin == xmax and ymin == ymax bit for bit in 24 576 of 24 576 rows, because the node coordinates the WKB carries are already exactly f32 (down(x) and up(x) therefore land on the same value). Both obvious fixes were measured and both fail:

Two leaves instead of four — REJECTED, it kills the prune. Measured against korp's real mapquery::row_group_intersects (not a copy of its logic) on the real Sweden files: with bbox.xmax absent, gxmax stays -inf, !gxmax.is_finite() fires and the function returns true — scan. The prune goes from 277 of 12 633 row groups to 12 633 of 12 633 on a 25 km Stockholm viewport (51× the rows), and to 12 633 on a 1 km z15 cell (21 064×) — including for a viewport over open ocean with zero hits. Safe, in that it never wrongly prunes. Useless, in that it never prunes.

A better encoding — there is none left. The writer already sets BYTE_STREAM_SPLIT with the dictionary off, and that is measurably the right choice. Re-encoding the real 103 488 583-row column through the same writer:

  BYTE_STREAM_SPLIT, dict off   693.94 MB   <- shipped
  PLAIN, dict off             1 002.38 MB   (+44.4 %)
  dictionary on               1 389.83 MB   (+100.3 %)
  two leaves only               346.97 MB   (= the redundancy figure, exactly)

What worked: quantise the covering onto an absolute 2^-12° grid (spatial::COVERING_GRID_SHIFT), rounding outward — [down] floors onto it, [up] ceilings onto it, so the box still CONTAINS the geometry and the covering stays conservative. BSS already flattens the sign/exponent/high-mantissa planes into runs; the low mantissa is the only expensive plane, and it carries sub-metre precision a row-group bounding box has no use for. All four leaves stay, so the shipped reader is untouched.

The alternative was truncating mantissa bits — the same bytes, but relative precision: 10 m of slop at longitude 24, 109 m at latitude 69, 217 m near longitude 180 (a planet cut). The absolute grid buys the same bytes at half the worst-case slop and behaves identically everywhere on Earth. Covering-column bytes on the real nodes.parquet:

  shipped (no quantisation)   693.94 MB      slop 0
  2^-15  (3.4 m)              540.17 MB   -22.2 %
  2^-14  (6.8 m)              481.91 MB   -30.6 %
  2^-13  (13.6 m)             415.55 MB   -40.1 %
  2^-12  (27.2 m)             344.55 MB   -50.3 %   <- shipped
  mantissa-16 (10...109 m)    343.40 MB   -50.5 %   (same bytes, twice the slop)

Per-column census, before and after, on the real Sweden tables (row order held constant, so the delta is purely the covering):

What it costs at read time: nothing the prune can see. Driven through korp's whole shipped read path (mapquery::query_ways_from) on both real tables, before and after, the row groups scanned are identical at every viewport — 10 of 12 633 Sweden-wide, 4 of 12 633 at a 1 km z15 cell, 3 of 12 633 at a 150 m z18 cell, 0 of 12 633 over open ocean — and the node reply is identical to the point (4 913 / 167 / 0), because the reader re-checks a point's exact coordinate (if !q.contains(coords[0].0, coords[0].1)). Only the per-row cull on ways.parquet admits more, and it admits, never drops: 1 364 ways rather than 1 255 at a z15 cell and 97 rather than 64 at z18, all within one grid step of the viewport edge, against a 160 000-way budget.

Guards: spatial::tests::covering_leaves_land_on_the_grid_and_still_contain_the_point asserts, on the leaves a packed file actually holds, that every value is an exact grid multiple, that every box still contains its own point, that no box is wider than one grid step, and that the fixture was actually MOVED by the grid (a fixture already on it could not tell a working quantiser from a dead one). Proven red three ways: reverting down/up to the plain as f32 cast, putting the fixture on the grid, and — for the prune itself — removing the Hilbert sort, which still takes the_base_tables_prune_and_convert_order_would_not red with the quantised covering in place (packed 181.31 deg² vs convert-order 181.30 deg²).

That leaves node_refs as the one unfixed line of the pack cost, and --node-refs false already answers it.

Should nodes.parquet drop untagged nodes? No.

97.67 % of the table is untagged — 101 081 991 of 103 488 583 rows (measured on the 2026-08-31 Sweden cut, tags null / "" / "{}"), which is where the reflex comes from: the tiler already stopped shipping them (89cd49c, ScanLedger::untagged_nodes, "an untagged node is a vertex, not a feature") and the recut cut z9 mean tile size by 43 %. But 89cd49c dropped them at tile placement, not from the table, and that distinction is load-bearing.

nodes.parquet is a SOURCE table, and one consumer resolves OSM ids against it: relgeom.rs's node oracle (resolve_relations reads the id column into a sorted Vec<i64> at relgeom.rs:305, and node_ids.binary_search(&m.member_ref) at relgeom.rs:384 decides whether a relation is admitted via_node). Its own doc already prices it: 9 882 relations, 3.7 % of the kept set, almost all type=public_transport stop areas whose members are platform NODES. Drop the rows and those relations vanish from relations.resolved.parquet with no error and no ledger column.

And it is not theoretical. Of the 32 392 resolved relations that name at least one in-cut node member, 5 954 (18.4 %) have no TAGGED node member at all — every id they are visible through is a row the drop would delete. (5 954 is an upper bound on relations lost: some are also held by way members.) 11 000 of the 190 643 in-cut node members (5.8 %) are untagged nodes.

Every other consumer is safe — verify (footer statistics), digest (per-row hashes, order-independent), tiles (already skips them at tiles.rs:3768), network (every rule needs a tag), graph (fed in memory from the network cut; its coordinates come from the WAY geometry, never from nodes.parquet), optimize, side_outputs/search (discards untagged rows itself), geo2arrow. Ring assembly matches member-way endpoints by node id, but both the ids and their coordinates come from ways.parquet's node_refs + geometry, positionally zipped — nodes.parquet is never consulted for a coordinate. Pass-2 node resolution uses node_coords.arrow via NodeStore, not this table either.

So the answer is no, not silently. verify --digest's set digest would change and nothing would compare it — there is no automated digest gate, only prose baselines. If a map-delivery cut wants the 694 MB, the honest shapes are the ones this repo already prefers: an explicit flag or a second output, exactly as shared/schema.rs argues for columns — "an absent column is honest; a present all-null column claims the fact is missing from the DATA rather than from the writer." The same holds for rows. --no-node- oracle (RelationResolveOptions::members_via_nodes) is today's sanctioned way to give up those relations on purpose; a row drop would give them up by accident. Note also that the covering fix above already recovers 349.39 MB of the 693.94 without touching a row.

8 192 re-measured on the base tables — and one claim narrowed

The shipped size was measured on roads.parquet. Once the base tables were packed the same sweep ran on them (same viewport, ground truth 387 521 rows for ways, 2 304 820 for nodes):

              ways.parquet (9 325 142 rows)       nodes.parquet (103 488 583 rows)
 rows/group   groups hit    scanned  vs ideal     groups hit     scanned  vs ideal
      1 024   512 of 9107   523 862     1.35x   2500 of 101064  2 560 000    1.11x
      4 096   135 of 2277   551 510     1.42x    628 of  25266  2 572 288    1.12x
      8 192    72 of 1139   584 278     1.51x    314 of  12633  2 572 288    1.12x
     16 384    47 of  570   756 310     1.95x    162 of   6317  2 654 208    1.15x
     65 536    19 of  143  1 198 678    3.09x     48 of   1580  3 145 728    1.36x
    100 000    16 of   94  1 525 142    3.94x     37 of   1035  3 700 000    1.61x

nodes.parquet behaves exactly like roads.parquet: 8 192 sits on the floor and 4 096 buys nothing. ways.parquet does not. There 8 192 scans 5.9 % more rows than 4 096 and 11.5 % more than 1 024 — because ways carries long features (coastlines, administrative boundaries) whose extent widens every row group they land in, so the curve keeps improving past the point the other tables flatten.

8 192 stays. 5.9 % of scan is not worth 4× the row groups and their footer, and one size across every table is what stops the layout forking. But the sentence "8 192 is the largest size that still reaches the floor" is a statement about a table, not a law, and it is corrected here rather than left to be quoted at the next lane. What IS uniform across all three tables is the direction of the guides' advice: 100 000 rows per group is 2.03× worse on roads, 3.94× on ways, 1.61× on nodes.

What is guarded, and what was seen RED

spatial::base_table_pack_law, four guards, each proven to fail on an injected defect before being trusted (LAW 2):

The first guard is written around the trap this codebase keeps hitting — both its numbers are "mean row-group bbox area", one line away from comparing a value to a copy of itself. So the control is pinned three ways before the 4× is trusted: the convert-order mean must be most of the whole fixture extent; the same helper run over the PACKED order must reproduce the number read from the parquet statistics (two independent routes to the same quantity, so the gap is order and not method); and the unpacked file must expose no bbox statistics at all, so "before" is a distinguishable state rather than a small number.

Compatibility, checked rather than assumed

• Content is preserved. verify --digest on liechtenstein, packed vs --pack false: the set (order-independent) digests are identical on all three tables (0314ac6d497f44e7 / 0e8633e7d5fd800a / 119bffec8cc479f0) while ord changes — which is exactly what a reorder should do.
• The TileRecipe token does not move. Byte-identical for a packed and an unpacked input, so adding bbox to ways.parquet does NOT mark 1 635 971 tiles stale. Checked because TileRecipe derives its columns term from the input schema's field names and types, which is precisely the shape of thing that would have.
• networks, graph and tile all run clean on a packed extract.

DELTA 3 — write order (min_zoom, hilbert)

Band first, Hilbert within the band.

Today the sort is Hilbert-global. That is correct for a spatial prune and useless for an attribute prune: every row group holds every zoom class, so min_zoom row-group statistics read [0..15] everywhere and prune nothing. The usual advice — "add min_zoom and push the predicate down" — is right in spirit and wrong in mechanism. On a spatially sorted table an attribute predicate cannot prune a row group; it has to be part of the sort.

Sorted by band, a z6 read touches only the low-min_zoom row groups, and inside each of them the rows are still Hilbert-ordered, so the spatial prune is unchanged.

UNBLOCKED 2026-08-31. The prerequisite was never the sort — it was that nodes/ways/relations had no min_zoom value to sort ON. They have one now (DELTA 1 above), non-null on every row, so spatial::pack_base_tables can take (min_zoom, hilbert) as its key without deriving anything: the leading key is already a column in the table it is re-ordering.

What DELTA 3 may now claim, and what it may not:

• May: run the (min_zoom, hilbert) order on ways.parquet and roads.parquet and measure the row-group prune against the Hilbert-only order. MINZOOM's synthetic 8.0× (8 of 8 row groups opened vs 1 of 8) is the shape to look for; it has never been measured on real data.
• May not: quote that number for nodes.parquet. 99.99 % of its rows are on rung 12 (measured above, Sweden and Liechtenstein alike), so the leading key is a constant and the order collapses back to Hilbert. A band-first nodes sort is not wrong, it is merely empty, and a gate that measured it would report a real 1.0× as a failure of the idea rather than a property of the table.
• Watch: the recipe. min_zoom is in NetworkRecipe's column list, so a rule edit already re-cuts the network tables. The base-table pack has no equivalent token, so a (min_zoom, hilbert) re-order of ways.parquet after a rules change is NOT detected as stale by anything today. That is DELTA 3's to close.

LANDED 2026-09-01 (lane Q) — the re-pack, the order, and the NULLs

Three changes, one packer (spatial::pack_batches), no second rule table:

1. A re-pack writes min_zoom. pack_batches runs ensure_min_zoom first: a table with tags and no min_zoom gets the column derived per row through class::min_zoom_of_tags — the ONE tag → rung writer — and appended LAST, the position ways_schema gives it, so a re-packed table and a convert-then-packed table have one layout. A present column is kept verbatim (a re-pack does not second-guess convert). A table with neither tags nor min_zoom is packed as before (one band, Hilbert only).
2. Order (min_zoom, hilbert), for every table through the packer — ways, roads, the resolved relations, and nodes (where it is empty, as predicted: 99.99 % on one rung). is_packed is now order-aware: a table carrying the column in Hilbert-only order reads false (spatial::is_band_first, footer-only), so pack_in_place upgrades it.
3. geometry_z<z> is NULL where min_zoom > z. The sub-pixel rule (degrade to a POINT, never delete) still holds for every row the rung ADMITS; a row the ladder withholds has no cell. A reader projecting geometry_z6 therefore never decodes a service road for a country camera.

The real table — /home/korp/.korp/work/sweden/unpacked-backup/ways.parquet (pre-DELTA-1 cut of 2026-08-31, no min_zoom) re-packed to /home/rickard/scratch/q-lane-pack/ways.parquet

Off the footer of the packed file (2026-09-01, osm-katana @ lane Q build), 9 325 142 rows, 1 139 row groups of 8 192:

The rung-6 share is 17.59 % against the 17.25 % projected on 2026-08-31 — the difference is the islet-1 coastline fix (31 372 shoreline ways moved from rung 12 to 6), now on disk.

korp's phase-1 gate — "a z6 read touches ≤ 20 % of row groups" — passes on the whole national table without a viewport: 17.65 %. With a viewport the covering prune stacks on top; the viewport numbers are in korp's map-v2-design.md, DELTA 3.

What the NULLs cost and save

Wall: 12 min 00 s for the 9.3 M-row table at load average 93–113 on 32 cores (four other lanes building) — NOT comparable with the 61.94 s measured on 2026-08-31 on a quiet box; not re-measured quiet. Peak RSS 4.90 GB, unchanged (4.97 GB before): the band key is one byte per row on the sort tuple.

The gates, and the red arm each was SEEN on

tests/min_zoom_repack.rs, 65 536 rows in a pre-DELTA-1 schema, packed by the real spatial_pack at 1 024 rows per group, everything read back off disk:

spatial::generalisation_law's fixture rows were untagged and so sat on rung 12; they now carry natural=water / natural=coastline (rung 6), because a fixture that measures a simplifier must have cells at every rung. Nothing else in the suite moved.

Still open

• The recipe watch above is NOT closed. A rules edit re-cuts the network tables (min_zoom_ladder in NetworkRecipe); a re-pack keeps a present min_zoom and stamps nothing, so a base table valued by an old ladder reads packed. One line of design, not done here: stamp class::rules_spelling() into the pack footer and let is_packed compare it.
• nodes.parquet was not re-packed in this lane (the live one is the 2026-09-01 10:06 Hilbert-only pack, no min_zoom); korp's reader admits every group of it, as before. CLOSED 2026-09-03 by lane nd-node-rungs — see DELTA 3b below.

DELTA 3b — the POINTS table (lane nd-node-rungs, 2026-09-03)

The bullet above was the whole defect, and it cost more than it looked. korp's parquet mirror measured it from the other end on 2026-09-02: a country viewport admitted 44–74 % of nodes.parquet's 12 633 row groups and cost 587–990 MB, against 7.8–15.2 MB for the two banded tables — so MIRRORED_TABLES shipped as ways + relations and points stayed on the wire. The fix is a re-pack, not a new rule.

Source: the convert output, not the packed file

Re-packed from ~/.korp/work/sweden/unpacked-backup/nodes.parquet — the pre-DELTA-1 convert output (id, geometry, tags, version, 217 groups, no covering) — and not from the live packed file. pack_batches drops the derived columns on the way in, so both roads lead to the same layout; the convert output is the honest one because packing it is exactly what the pipeline does today, so the artefact is a pipeline OUTPUT rather than a repair. 47.8 s wall, 17.4 GB peak RSS, verify reports 103 488 583 rows and 103 488 583 geometries set — identical to the source.

The band, off the footer

  min_zoom <= 6      1 of 12 633 row groups   (0.01 %)
  min_zoom <= 9      2 of 12 633              (0.02 %)
  min_zoom <= 12  12 633 of 12 633            (100 %)
  band-first (max(g) <= min(g+1))  true
  groups by (lo,hi):  (6,9) 1 · (9,12) 1 · (12,12) 12 630 · (12,15) 1

The 100 % at rung 12 is the number that has to be said out loud rather than buried. Rows per rung, read off the packed column:

  rung  6         6 078   0.0059 %
  rung  9         2 894   0.0028 %
  rung 12   103 479 582  99.9913 %
  rung 15            29   0.0000 %

101 081 991 rows (97.67 %) carry no tags at all — way-member vertices, which the ladder gives UNCLASSIFIED_MIN_ZOOM = 12; most of the 2 406 592 tagged nodes are POIs (amenity, shop, building), which the ladder also puts on 12. So the band is worth everything at a country/regional camera and nothing below it. Contrast ways.parquet, where rung 6 is 17.65 % of groups. Same one ladder, different populations — which is why the ladder's own doc calls the distribution a property of a table and not a law.

No rung columns for points — the decision, and its measurement

generalised_rungs_for already answered [] for a point-only geo document. This lane turned that from a comment into APPLIED OUTPUT: tests/nodes_carry_the_band.rs reads the packed file back and fails if geometry_z6/z9/z12 are in the SCHEMA at all — not "present and null", not "present and equal to geometry", not there. A rung column on a Point is Douglas–Peucker applied to one vertex, i.e. three byte-for-byte copies of geometry: +1.97 GB on this table (655.18 MB × 3) buying a reader nothing the base column does not already hand it. min_zoom alone gates a point; the READ projects the one geometry column it needs.

The two fat columns

id and the WKB were carrying an RLE dictionary of near-unique values — a second copy of the column with an index array bolted on, and dictionary pages are RLE-encoded per page, which pre-empts the plain byte runs zstd would otherwise flatten. The covering leaves have had the dictionary off since they landed and nobody had asked the same question of the other two. spatial::dictionary_is_waste is the one list; tags/version/min_zoom keep theirs, because those values repeat and that is the whole reason they are small. Measured across all three tables (60/24/12 row-group samples re-encoded through the same writer), with an as shipped arm that reproduces the footer to within 0.3 % on nodes so the harness is measuring the writer and not itself.

bbox is left alone and the reason is already in spatial::COVERING_GRID_SHIFT: for a Point xmin == xmax bit for bit, so 172 MB of it is a byte-exact duplicate — and both ways out were measured and refused. Two leaves takes korp's real prune from 277 to 12 633 of 12 633 groups (its reader names all four leaf paths, and a missing one reads as "scan"). Re-encoding is exhausted: BSS 693.94 MB, PLAIN +44.4 %, dictionary-on +100.3 %. The 2^-12° grid already halved the column at an identical prune. This lane re-derived the 344.55 MB off the live file and confirmed the quantisation is what shipped.

MEASURED AND DECLINED: sorting each row group by id

Re-sorting a row group's rows by ascending id leaves the group's contents and its statistics untouched, so no prune moves, and it makes id delta-encodable: 182 247 bytes on the nodes sample against 748 870, another −8.6 % of the whole file. Declined, and the reason is that it wins on ONE table: the same change costs ways.parquet +1.3 % and relations.resolved +0.3 %, because breaking Hilbert locality inside a group costs the geometry and covering columns more than id gains wherever the rows are lines rather than points. A pack order that is right for points and wrong for ways is a per-table tuning knob, i.e. a second write order. The number is recorded in dictionary_is_waste's doc so a future lane knows what it is buying.

What it buys korp's OWN read (korp/benches/nodes_band.rs)

Both files through korp's Footer::admits + projection_leaves + nornir_parquet_delta::plan — the served prune, the served mask, the mirror's own pricer:

At lod 13 the group count is unchanged, as it must be — every group is rung 12 and the covering box is doing all the work, exactly as before. The 6.2/12.9 against 7.9/16.6 is the dictionary change alone.

It is a MAP CHANGE, and the gate had to be built three times

Serving fewer points is the point, so a gate on served COUNTS goes red on a healthy re-cut, and a gate on admitted ROWS goes red on where the row-group boundaries happened to fall (0.08 % at Sthlm lod 13). Both were driven and discarded. The gate that survives is a SET CONTAINMENT over the reply's own OSM ids with min_zoom_of_tags as the oracle: every point the LADDER admits at a camera, that the unbanded read served, must still be served. 0 lost, on both camera pairs where neither arm truncates. At the country cameras the unbanded arm cannot be read whole at any budget the bench can afford (it truncates at 4 000 000 points while the banded arm reads the same ground completely in 50 765), and that is reported as the finding it is rather than silently skipped.

What actually changes on screen: on the unbanded cut a Stockholm lod-10 points read served 4 000 000 rows of which 3 804 629 (95.1 %) were untagged — by rung, {6: 13 001, 9: 108, 12: 3 986 887, 15: 4}. Banded, the same camera serves 13 694. Nobody has looked at the painted result yet; that is a watch item, not a claim.

The guard

verify::Fault::UnbandedPacked — PACKED (covering.bbox) and taggable (tags) and not band-first. Footer-only, and banded is read through spatial::is_band_first, the PACKER's own predicate, so the guard cannot drift from the writer. All three conditions are load-bearing: an unpacked convert output is a pipeline stage, not a defect, and a table with no tags has nothing for the ladder to read. Run against the live cut it names two tables — nodes.parquet and the raw relations.parquet.

Red arms driven on this build, both seen failing:

Two of my own fixture assumptions were red first and were the fixture's fault, not the code's: highway=crossing is rung 12 and not 15 (an unlisted highway value falls to City), and a z12 camera on a points table legitimately opens every group — asserting a prune there would have been asserting that data goes missing.

Still open after this

• The artefact is in /home/rickard/scratch/nd-lane-pack/nodes.parquet and the live cut is not swapped. ~/.korp/work/sweden is not this lane's to write.
• The raw relations.parquet faults the same way and was not re-packed here — it carries no assembled geometry, so nothing reads it for a viewport, but verify now says so.
• The recipe watch from DELTA 3 is still open: a rules edit re-cuts the network tables and a re-pack keeps a present min_zoom and stamps nothing.

Row groups stay at 8 192

MEASURED on the live roads.parquet (2 277 368 rows), central Stockholm ~25 km, against a ground truth of 138 229 genuinely-intersecting rows:

The 100 000-row group that every GeoParquet tuning guide recommends is 2.03× worse than what is shipped. Written down here because the advice is otherwise reasonable-sounding and will be proposed again.

The parquet page index buys nothing on this layout. It is written (arrow-rs default; has_column_index is true on every leaf of both roads.parquet and a z12 tile), but the 30 hit row groups contain 30 data pages — 8 192 float32 rows is 32 KB against a 1 MB page limit, so the page IS the row group. Page-index pruning is where DuckDB and Sedona win on wide tables; on a Hilbert-sorted geometry table the win is already taken by the row-group prune.

What deleting the pyramid removes

tiles.rs is 8 817 lines. The WayLane budget ladder, the dedup grid, the fan-out cap, the spill machinery, the dust/small/dominant tiers — all of it is budget administration that a columnar read does not need. The tile-specific defects go with it: sub-pixel areas shipped as Point (11 101 365 of 12 628 347 area placements, 87.9 % — ⚠ SEE BELOW, this figure is not reproducible from the pyramid on disk), long features present only in their centroid tile (417 after the 08-31 recut), and untagged nodes consuming tile budget (fixed separately, 89cd49c).

⚠ The 87.9 % does not survive a census of the tables it describes (2026-08-31)

osm-katana census ~/.korp/work/sweden/tiles --pyramid --geometry over the 08-31 recut — 1 635 971 files, one recipe token on every one of them, tile mtimes 06:15–06:17 UTC, UNACCOUNTED 0 — reports the whole pyramid as:

geometry: 33 357 988 rows = LineString 9 052 208 (27.1 %) + MultiPolygon 16 961 280
          (50.8 %) + Point 7 344 500 (22.0 %) · UNACCOUNTED 0

7 344 500 Point rows exist across all four zooms, before any of them is asked whether it is an area. The claim needs 11 101 365 — 3 756 865 more Points than the pyramid contains — so it cannot describe these tables.

Which of the two is wrong is not determinable from the tables: the 87.9 % carries no date, no file list and no recipe token, so there is nothing to check it against. That is the same gap the 57.0 % fell through. Re-derive it from the run that is supposed to produce it, or withdraw it; do not quote it meanwhile.

It IS determinable from the history — and the answer is that both are right

Not from the tables, no. But the commit timeline settles it, and the resolution is worth writing down because it is the reason the figure was so persuasive:

• the pyramid the figure describes was cut at 2026-08-30 22:36 — 1 589 935 tiles
• korp b937078 quoted the 87.9 % at 05:32, from that run's own summary line
• korp f6aac7e at 06:22 explicitly carried it forward — "the 87.9 % area-placement figure is unaffected; that one came from the run's own summary" — while correcting the separate flattened-census 57.0 %
• but the recut ran 05:37 → 06:24, and it replaced that pyramid: 89cd49c dropped 97.7 % of nodes and the byte-based fan-out budget placed more ways

So the figure was measured correctly, on a pyramid that was already being replaced as it was being quoted. Both numbers are right about different tables. Withdraw it as a present-tense claim; it is true of the 08-30 cut and of nothing on disk.

The DEFECT it motivated is untouched: korp-map-pt-1 was real and is fixed — base points went into one untagged bucket without their tags being read and were inked building-grey at Lod::City in a row that is off by default. What is gone is the magnitude, and with it the sentence that made the pyramid look indefensible.

A replacement number needs a tag predicate, not a geometry histogram: census reports the geometry × tags cross-tab but cannot yet ask whether a given Point is an area. Open, and recorded as open rather than guessed at.

The gate on the deletion is korp's phase 0: the parquet path must come within 2× of the pyramid at z12 and z15. Until that number exists, nothing here is deleted.

geo-strip-1 — the type reached the schema and not the file, and both guards were green

2026-08-31. 57aafb3 landed LogicalType::Geometry through one writer, shared::schema::geometry_field(), with two guards seen red. Hours later a real packed table off the shipped pipeline — /home/rickard/scratch/reader-gen/ways.parquet, 1.81 GB, 9 325 142 rows — read back:

  optional binary geometry;                             <- no logical type at all
  optional binary geometry_z6  (Geometry(crs=srid:0))
  optional binary geometry_z9  (Geometry(crs=srid:0))
  optional binary geometry_z12 (Geometry(crs=srid:0))
  is_geo_stats_set(geometry) = False

The three DERIVED columns carried the type. The one column every reader actually reads did not.

What the mechanism was

spatial::pack_schema built the rung columns fresh, through generalised_geometry_field(), and carried the base geometry field over from the input file's schema. The declaration lives on the arrow field as the parquet_geospatial::WkbType extension type, i.e. as field METADATA, and a parquet reader reconstructs the arrow schema from what the FILE says — so a table cut before 57aafb3 hands back a bare Binary field, and the pack faithfully wrote a bare column.

That is worse than a stale file, because spatial::is_packed answers true for the result (it has covering.bbox and all three rungs), so no later pack would ever touch it. Only a full re-convert would have fixed the declaration.

FIXED by re-declaring geometry through shared::schema::geometry_field() in pack_schema — the same one writer every other declaration of that column goes through (LAW 5), and no re-attachment sprinkled anywhere else. carry_columns's own doc already claimed a re-pack is an UPGRADE rather than a duplication; this is that claim honoured for the base column too.

The reusable lesson: the original guards were round-trip-shaped

schema::geometry_columns_carry_the_logical_type and the_rung_fields_are_the_geometry_field_renamed both called a helper called round_trip: take a Field, hand it straight to an ArrowWriter, read the footer back.

A guard that STARTS from the field can only ever prove that the field works. It is blind by construction to everything between the declaration and the file — a Schema rebuilt from names and DataTypes, a field cloned without its metadata, a field carried out of an input the pipeline read. Every one of those is a step the real write path takes and the guard never did.

spatial::the_rung_columns_carry_the_geometry_logical_type was closer — it packs a real file and reads the real footer — and still could not see this, because its INPUT schema is built in memory by the fixture. Hand pack_schema a good field and it carries a good field.

The generalisation of the trap, and it is the one this repo keeps hitting: a value compared with a copy of itself can never go red. Asserting that a Schema you just built equals geometry_field() is exactly that shape, and so is asserting that a field you just handed to a writer comes back the way you declared it.

The guard that replaced them

osm-katana/tests/geometry_type_on_disk.rs. It runs the real stages — convert (with its pack), resolve_relations, extract_networks, extract_buildings, tiles::tile_region — over one XML fixture, then opens every .parquet under the output and reads logical_type + geo_statistics off the column chunk metadata in the footer. Nothing in it inspects an in-memory Schema, and nothing in it is compared with geometry_field().

Measured coverage of one run: 17 parquet files, 38 geometry column chunks, 12 tables — nodes · ways · relations · relations.resolved · roads · rails · trails · rail_nodes · graph_edges · graph_nodes · buildings · the tile pyramid — with the count printed and MUST_REACH pinning the table list as a LITERAL, so a stage that silently stops writing turns the guard red instead of quietly shrinking what it covers.

Both halves or neither: a chunk that declares Geometry over non-null rows and carries no geo_statistics is a fault, because the chunk bbox is the half that gives a reader a box without a covering column.

Its three arms, all seen red:

What is on disk, and it is not fixed by this commit

Measured 2026-08-31 on /home/korp/.korp/work/sweden (cut 05:56–06:15, i.e. before 57aafb3 at 14:05):

• 10 of 10 geometry columns across the four base tables and the six network tables read logical_type: None
• 30 of 30 sampled tiles, out of 1 635 971, read logical_type: None

Those tables also predate DELTA 2 — they carry no geometry_z* columns at all. They are stale, not corrupt; the fix makes the next pack of them an upgrade rather than a laundering. Nothing here re-cuts them.

DELTA 4 — speed_kmh + speed_source on graph_edges (lane nvdb, 2026-09-02)

korp's road-graph priced minutes off a class table (SPEED_TABLE in korp's roadgraph.rs) — Stockholm C → Uppsala C answered 68.5 min for 68 km, an E4 priced at its class default. The honest fix, decided by MEASUREMENT before any API was touched: OSM already carries the answer. On the live Sweden cut (km-weighted, 2026-09-02): maxspeed on 90.2 % of arterial km (motorway 100 %, trunk 99.6 %, primary 99.7 %, secondary 98.7 %, tertiary 84.6 %), oneway resolved on 100 % of motorway km. NVDB via Trafikverket needs an API key this machine does not hold (TRAFIKVERKET_API_KEY — the collector exists in korp-collectors, key-in-body), so NVDB stays the queued enrichment and OSM is wired first — deterministic, no network, no key.

What changed, one writer each:

• graph::SPEED_TABLE + class_speed_kmh + maxspeed_kmh + edge_speed_kmh MOVED INTO osm-katana — korp re-exports them; the cut and the router share one source by construction (LAW 5).
• edge_fields() gains speed_kmh: Float64 (non-null) and speed_source: Utf8 (maxspeed | class). The recipe derives from the field list, so the token moved and every cut graph reads Stale — the re-cut is the point, not a cost.
• maxspeed parse: bare number = km/h; N mph × 1.609; walk = 7; anything else (zones, 70;50, 0) refuses to the class default and speed_source says so. Range-gated to (0, 250].
• Graph::load: every NetArc carries speed_kmh: u8 (rounded — the arc stays 24 bytes; an f32 pads to 32 and costs ~150 MB over Sweden's two CSRs). An old cut without the column loads fine and prices class defaults; Graph::speed_from_column is the typed fallback flag.
• GraphStats reports speed_tagged_edges and the km-weighted road coverage (speed_tagged_road_m / road_m) — the CLI prints it per cut.

Red first (LAW 2), all seen failing 2026-09-02 against planted identities: maxspeed ignored at the cut (the column full of class defaults), maxspeed ignored at load (the arc priced 80 where the tag says 70), and korp's metre router picking a 30 km/h street over a 110 km/h motorway (5 727 m seen) before route cost moved to travel seconds.