osm-katana/src/shared/schema.rs
use std::sync::Arc;
use arrow::array::{
ArrayRef, BooleanBuilder, Int8Builder, UInt8Builder, UInt16Builder,
};
use arrow::datatypes::{DataType, Field, Schema};
/// **The `geometry` column — the ONE place it is declared (LAW 5).**
///
/// Every parquet this crate writes that carries WKB gets its field from here, and the
/// field carries the [`WkbType`] arrow extension type. That is not decoration: it is the
/// only door through which `ArrowWriter` emits `LogicalType::Geometry` into the parquet
/// schema, and with it the writer-side bounding-box accumulator that fills
/// `ColumnChunkMetaData::geo_statistics`.
///
/// # What this buys, measured — and what it does NOT
///
/// MEASURED here 2026-08-31 (`geometry_columns_carry_the_logical_type`, seen red with the
/// plain field restored): two WKB points written through `ArrowWriter` come back as
/// `Geometry(GeometryType { crs: Some("srid:0") })` with `geo_statistics` present and the
/// bbox `x (10.594, 24.178) · y (55.137, 69.06)`. Written as a bare `Binary` field the
/// same bytes come back `logical type: None`, `geo_statistics: NONE`.
///
/// It buys **interop and a second source for the prune**: a reader learns the column is
/// geometry from the SCHEMA instead of parsing the `geo` footer JSON, and the chunk bbox
/// exists whether or not a covering column was appended.
///
/// It buys **no speed**. `geo_statistics` sits on `ColumnChunkMetaData` — the same
/// granularity korp's `mapquery::row_group_intersects` already prunes at through the
/// `bbox.*` leaf statistics — and `parquet`'s page index carries no geospatial variant,
/// so there is no page-level spatial prune to reach for. The GeoParquet 1.1
/// `covering.bbox` column STAYS: it is what korp culls per ROW with, before any WKB is
/// decoded, and deleting it would push that cull onto the 487.9 MB (uncompressed) the
/// geometry column of `roads.parquet` weighs.
///
/// # It cannot move a recipe token
///
/// [`crate::tiles::TileRecipe`] derives its `columns` term from the field's NAME and
/// `DataType` only. An extension type is field METADATA and changes neither, so adding
/// this cannot mark a cut pyramid stale. `the_logical_type_does_not_move_the_recipe`
/// pins that.
pub fn geometry_field() -> Field {
let mut f = Field::new("geometry", DataType::Binary, true);
f.try_with_extension_type(parquet_geospatial::WkbType::new(None))
.expect("WkbType is a valid extension type for a Binary field");
f
}
/// **The generalisation rungs that get their OWN geometry column** — korp's ladder
/// `[6, 9, 12, 15]` minus its last rung, because rung 15 IS `geometry`.
///
/// # Why three names and not four
///
/// The four columns a reader sees are `geometry_z6`, `geometry_z9`, `geometry_z12` and
/// `geometry`. The finest rung is not a fourth *derived* column: at z15 the epsilon is
/// `360 / (2048 · 2^15)` ≈ 5.4e-6° ≈ **0.6 m**, and running Douglas–Peucker at that
/// tolerance over Swedish OSM geometry moves almost nothing while costing a full decode,
/// a full re-encode and a byte-for-byte duplicate of the largest column in the file. So
/// `geometry` is the z15 column, under the name every existing reader already uses.
///
/// **Do not add a fourth entry here.** korp's level type is a four-variant enum; a fifth
/// COLUMN forces a fifth variant, which is the trap the z9 lane hit on 2026-08-24.
/// [`generalisation_rungs_match_korps_ladder`] fails if this list grows, shrinks, or
/// stops being a prefix of [`crate::shared::class::RUNGS`].
pub const GENERALISED_RUNGS: [u8; 3] = [6, 9, 12];
/// The column name for generalisation rung `z` — `geometry_z6`, `geometry_z9`, …
///
/// One formatter, so the writer, the projection mask and the tests cannot spell the
/// column three ways. Rung 15 is not spelled here at all: see [`GENERALISED_RUNGS`].
pub fn generalised_geometry_name(z: u8) -> String {
format!("geometry_z{z}")
}
/// **A generalisation column — [`geometry_field`] under a rung name.**
///
/// It is literally [`geometry_field`] with `with_name`: the `LogicalType::Geometry`
/// declaration, the `WkbType` extension type and the writer-side bbox accumulator all
/// come from the ONE writer (LAW 5). Building a second `Field::new(name, Binary, true)`
/// here would produce a column that carries WKB, is named for geometry, and is not
/// DECLARED geometry — four twins that drift the first time the extension type changes,
/// which is exactly what the v2.0 design forbids by name.
///
/// `geometry_columns_carry_the_logical_type` runs every rung column through the same
/// round trip as the base column, so a regression in either shows up in both.
pub fn generalised_geometry_field(z: u8) -> Field {
geometry_field().with_name(generalised_geometry_name(z))
}
/// **The `min_zoom` column — the ONE place it is declared (LAW 5).**
///
/// The first slippy rung at which the feature is allowed to draw, out of korp's ladder
/// `[6, 9, 12, 15]`. The VALUE comes from [`crate::shared::class::min_zoom_of_tags`],
/// which is the one tag → rung decision in this crate; this function only says what the
/// column is called and what type it holds, so the SEVEN tables that carry it — the four
/// network tables plus `nodes` / `ways` / `relations` — cannot declare it seven ways.
/// `writer::min_zoom_law::every_base_table_declares_the_same_min_zoom_column` pins the
/// base-table half of that.
///
/// `UInt8`, non-null: every row has a rung, including the ones no rule matched (see
/// [`crate::shared::class::UNCLASSIFIED_MIN_ZOOM`] — a named rung, not a null). A
/// nullable column would let "we did not classify this" and "this draws everywhere" wear
/// the same cell, which is the ambiguity [`nodes_schema`]'s own note about all-null
/// `changeset` is about, from the other side.
///
/// # What this column buys, and where the prune comes from
///
/// On its own it does **not** prune row groups. A Hilbert-sorted table has every rung in
/// every row group, so the `min_zoom` chunk statistics read `[6..15]` everywhere. (On
/// `nodes.parquet` they read `[12..12]` for a different reason: 97.67 % of Sweden's
/// nodes carry no tags at all, so 99.99 % of that table sits on the unclassified rung —
/// measured off disk 2026-08-31. A band-first sort of `nodes` is empty, not wrong.) The
/// standard advice — add the column, push the predicate down — is right in spirit and
/// wrong in mechanism on a spatially sorted table. It pays when `min_zoom` is the FIRST
/// sort key with Hilbert inside it — which `spatial::pack_batches` writes since DELTA 3
/// (2026-09-01): order `(min_zoom, hilbert)`, and a re-pack of a table that lacks the
/// column derives it from `tags` through the one writer. The measured prune is in
/// `.nornir/osm-katana-v2-design.md` under DELTA 3.
///
/// What it buys today is a per-ROW cull that costs one `u8` compare instead of a
/// `serde_json` parse of the `tags` column — the read path's worst hotspot (korp's
/// `parse_tags` is 55.9 % of all allocations on a tile read, measured 2026-08-03) — and,
/// more than that, it moves the rung decision to WRITE time, where it is made once for
/// the whole planet instead of once per feature per frame.
pub fn min_zoom_field() -> Field {
Field::new("min_zoom", DataType::UInt8, false)
}
/// **The class columns — the ONE place they are declared (row 18, 2026-09-05).**
///
/// Six columns that ride beside [`min_zoom_field`] on every table that carries tags, in
/// this exact order, valued in one pass by
/// [`crate::shared::class::class_of_tags`]:
///
/// | column | type | what it holds |
/// |---|---|---|
/// | `class_id` | `UInt16` | the INDEX of the rule `classify_rule` stopped on, or `u16::MAX` |
/// | `kind` | `UInt8` | [`crate::shared::class::Kind::code`] — `facett_map::WayKind`'s OSM half |
/// | `layer` | `Int8` | the `layer` tag, −5..=5, 0 when absent or unusable |
/// | `bridge` | `Boolean` | any non-`no` `bridge`, or `man_made=bridge` |
/// | `tunnel` | `Boolean` | any non-`no` `tunnel` |
/// | `oneway` | `Int8` | [`crate::network::oneway_of`] — 1 forward, −1 backward, 0 both |
///
/// # Why these six and why now
///
/// `classify_tier` already walked the rule table and knew which rule it stopped on, and
/// then kept only the rung. `min_zoom` says WHEN a feature opens; nothing on the wire
/// said WHAT it is, so korp re-derives that from the `tags` JSON per feature per frame —
/// `map_paint::classify_ways` → `facett_osm::parse_tags_into` → `classify_with` — and
/// korp's `parse_tags` is **55.9 % of all allocations on a tile read** (measured
/// 2026-08-03). The four tag columns are the same argument for PAINT ORDER: korp
/// `6e22d78` (2026-09-05) had one `landuse=grass layer=1` area outrank the whole road
/// network, a term the painter computed from a tag it parsed itself.
///
/// **All six are non-null**, the decision [`min_zoom_field`] made and for its reason: a
/// nullable column would let "no rule matched this" and "the classifier never ran" wear
/// one cell. The unmatched case has NAMED values
/// ([`crate::shared::class::UNCLASSIFIED_CLASS_ID`] and friends), not an absence.
///
/// They are appended **after** `min_zoom`, so every positional read of
/// `id`/`geometry`/`tags`/`node_refs`/`version`/`min_zoom` that existed before keeps its
/// index — the same rule DELTA 1 followed when it appended `min_zoom` last.
pub fn class_fields() -> Vec<Field> {
vec![
Field::new("class_id", DataType::UInt16, false),
Field::new("kind", DataType::UInt8, false),
Field::new("layer", DataType::Int8, false),
Field::new("bridge", DataType::Boolean, false),
Field::new("tunnel", DataType::Boolean, false),
Field::new("oneway", DataType::Int8, false),
]
}
/// [`min_zoom_field`] and [`class_fields`] together, in wire order — the whole block a
/// tags-carrying table appends, so a caller cannot add one half and forget the other.
pub fn class_block_fields() -> Vec<Field> {
let mut f = vec![min_zoom_field()];
f.extend(class_fields());
f
}
/// **The builder for that block — one `push` per row, seven arrays out (LAW 5).**
///
/// Every production writer in [`crate::writer`] used to own a bare `UInt8Builder` for
/// `min_zoom` and its own `append_value(class::min_zoom_of_tags(..))` call. Seven
/// columns × seven writers is forty-nine places to forget one, so the block is a TYPE:
/// [`ClassColumns::push`] takes the row's `tags` and nothing else, and
/// [`ClassColumns::finish`] hands back the arrays in exactly [`class_block_fields`]'
/// order. A writer that adds the fields to its schema and forgets a column does not
/// compile — the arity of `RecordBatch::try_new` refuses it.
pub struct ClassColumns {
min_zoom: UInt8Builder,
class_id: UInt16Builder,
kind: UInt8Builder,
layer: Int8Builder,
bridge: BooleanBuilder,
tunnel: BooleanBuilder,
oneway: Int8Builder,
}
impl Default for ClassColumns {
fn default() -> Self {
Self::new()
}
}
impl ClassColumns {
#[must_use]
pub fn new() -> Self {
Self {
min_zoom: UInt8Builder::new(),
class_id: UInt16Builder::new(),
kind: UInt8Builder::new(),
layer: Int8Builder::new(),
bridge: BooleanBuilder::new(),
tunnel: BooleanBuilder::new(),
oneway: Int8Builder::new(),
}
}
/// One row, classified once. `tags` is the flat JSON `tags::serialize_tags` writes;
/// an untagged feature passes `"{}"` and takes the named unclassified values.
pub fn push(&mut self, tags: &str) {
let c = crate::shared::class::class_of_tags(tags);
self.min_zoom.append_value(c.min_zoom);
self.class_id.append_value(c.class_id);
self.kind.append_value(c.kind);
self.layer.append_value(c.layer);
self.bridge.append_value(c.bridge);
self.tunnel.append_value(c.tunnel);
self.oneway.append_value(c.oneway);
}
/// **The six arrays of [`class_fields`] alone** — the block WITHOUT `min_zoom`.
///
/// For [`crate::network::network_fields`], which has carried `min_zoom` in its own
/// position (above `geometry`) since DELTA 1 and appends only the six new columns.
/// Moving that column to join its block would renumber `geometry`, `tags` and
/// `node_refs` for every positional reader of a network table, which buys nothing —
/// the block is a construction of the WRITER, not of the column order.
///
/// Defined as [`finish`](Self::finish) minus its head rather than as a second
/// builder set, so the two cannot drift: a column added to the block appears in both
/// or in neither.
pub fn finish_without_min_zoom(&mut self) -> Vec<ArrayRef> {
let mut all = self.finish();
all.remove(0);
all
}
/// The seven arrays, in [`class_block_fields`] order.
pub fn finish(&mut self) -> Vec<ArrayRef> {
vec![
Arc::new(self.min_zoom.finish()) as ArrayRef,
Arc::new(self.class_id.finish()) as ArrayRef,
Arc::new(self.kind.finish()) as ArrayRef,
Arc::new(self.layer.finish()) as ArrayRef,
Arc::new(self.bridge.finish()) as ArrayRef,
Arc::new(self.tunnel.finish()) as ArrayRef,
Arc::new(self.oneway.finish()) as ArrayRef,
]
}
}
/// The node table.
///
/// # `changeset` and `timestamp` are GONE (`katana-13`)
///
/// They were declared here and `append_null`ed by every node writer,
/// unconditionally, on every row. MEASURED on korp's 2026-08-03 `sweden` cut:
/// 102 518 979 rows, 102 518 979 nulls in each — **205 037 958 cells that no code
/// path in this crate could ever fill**, because neither writer ever read the PBF
/// `Info` block they come from. A consumer joining on `changeset` got zero rows and
/// nothing anywhere went wrong.
///
/// A schema is a promise. Declaring a column no producer can fill is the same defect
/// as `ways_schema`'s note below states in the other direction: an absent column is
/// honest, a present all-null column claims the fact is missing from the DATA rather
/// than from the writer. The byte cost was never the point (both compressed to ~0.2 MB);
/// the lie was.
///
/// Nothing read them: `grep changeset` over korp, facett and this crate finds only the
/// XML changeset-prologue seek, which is unrelated. If OSM metadata is wanted later it
/// comes back as columns with VALUES, read out of `Info` — not as a declaration.
pub fn nodes_schema() -> Schema {
let mut fields = vec![
Field::new("id", DataType::Int64, false),
geometry_field(),
Field::new("tags", DataType::Utf8, true),
Field::new("version", DataType::Int32, true),
];
fields.extend(class_block_fields());
Schema::new(fields)
}
/// The way table.
///
/// `node_refs` is the resolved node-id list behind each way's geometry. MEASURED on
/// korp's 2026-08-03 `sweden` cut: 237.1 MB of a 893.3 MB `ways.parquet` — **26.5 %
/// of the file** — and 1 009.3 MB of 3 648.3 MB uncompressed. Only two consumers
/// read it: [`crate::relations`]'s ring assembly, which matches member-way endpoints
/// by node id, and `osm-katana verify`. A pure map-delivery cut needs neither, so
/// `include_node_refs = false` drops the column and a quarter of the file with it.
///
/// **Schema-optional, not null-filled.** An absent column is honest; a present column
/// of empty lists would claim every way has no nodes.
pub fn ways_schema(include_node_refs: bool) -> Schema {
let mut fields = vec![
Field::new("id", DataType::Int64, false),
geometry_field(),
Field::new("tags", DataType::Utf8, true),
];
if include_node_refs {
fields.push(Field::new(
"node_refs",
DataType::List(Arc::new(Field::new("item", DataType::Int64, true))),
true,
));
}
fields.push(Field::new("version", DataType::Int32, true));
fields.extend(class_block_fields());
Schema::new(fields)
}
pub fn relations_schema() -> Schema {
let mut fields = vec![
Field::new("id", DataType::Int64, false),
geometry_field(),
Field::new("tags", DataType::Utf8, true),
Field::new("members", DataType::Utf8, true),
Field::new("version", DataType::Int32, true),
];
fields.extend(class_block_fields());
Schema::new(fields)
}
/// **The ONE spelling of an Arrow type in a recipe token (LAW 5).**
///
/// [`crate::tiles::TileRecipe`] and [`crate::network::NetworkRecipe`] both derive their
/// `cols=` term as `name:short_type(data_type)`, and both used to own a private copy of
/// this match. The copies were kept in step "deliberately" by a doc comment — and by
/// 2026-08-31 they had **already drifted**: `network`'s copy had grown a `DataType::Int8`
/// arm for the graph's `oneway` column and `tiles`' copy had not, so the two recipes an
/// operator diffs side by side spelled the same type two different ways. One function is
/// the fix by construction; a guard watching two copies agree is the shape LAW 5 names.
///
/// # There is no `Other`, and that is the whole point
///
/// Both copies ended in `_ => "Other"`, and `Other` is the one spelling that cannot
/// distinguish two types. That was not theoretical: `min_zoom` (DELTA 1), `levels` and
/// `height_source` (the buildings side output) are all `UInt8`, all spelled `Other`, so
/// **retyping any one of them moved no token at all** — every table on disk would have
/// been judged *Current* while its contents no longer matched the recipe. That is
/// precisely the defect the token exists to prevent (`collauto-6`), and its cousin
/// already cost this fleet once (`ladder-1`, korp `ebf46d1`).
///
/// # Why a total spelling and not a hard refusal
///
/// The obvious alternative is to make an unspellable type a **write-time failure**: a
/// recipe that cannot spell a column is a recipe that should not be written. It was
/// weighed and rejected, because it buys nothing the fallback below does not already buy
/// and it pays for it with an outage: the only way to reach it is a developer adding a
/// column of a new type, and the failure would land in the middle of a 13-minute planet
/// cut rather than at the keyboard.
///
/// The fallback spells an unknown type with its own `Debug` form instead. Two things
/// follow, and both are the direction you want:
///
/// * **It is injective on the discriminant**, so a retype ALWAYS moves the token. No
/// future `UInt8`-shaped hole can open: adding an arm below is an optimisation of the
/// spelling, never a correctness fix.
/// * **It fails safe.** `Debug` is not a stability contract of `arrow`, so an upstream
/// rename would move every token — and a moved token means *Stale*, which re-cuts. The
/// `Other` catch-all failed the other way: a *false Current*, serving stale data. A
/// spurious re-cut costs 21.3 s on the network family; a false Current costs correctness.
///
/// A "short arm" is therefore only about readability.
///
/// # `Struct` and `List` spell their children
///
/// Until 2026-09-01 they did not — "the children of `bbox` are fixed by GeoParquet 1.1
/// and spelling them out would make the token unreadable" — and that was the one
/// accepted collision left: `Struct(a)` and `Struct(b)` read alike, so a retype of
/// `bbox.xmin` from `Float32` to `Float64` was invisible to every recipe. GeoParquet
/// fixes the leaf NAMES, not their width (1.1 allows either), and a covering half the
/// width the recipe implies is exactly the false-Current this function exists to
/// prevent. So the spelling recurses: `Struct{xmin:Float32,ymin:Float32,…}`,
/// `List<Int64>`, all the way down. The price is a longer token and ONE move of every
/// token that carries a `bbox` — measured and written down in
/// `.nornir/osm-katana-v2-design.md`.
pub fn short_type(t: &DataType) -> std::borrow::Cow<'static, str> {
// Every arm here is an exact match for the arrow variant's own name, so a token an
// operator greps reads the same as the schema a `parquet` dump prints.
let s: &'static str = match t {
DataType::Boolean => "Boolean",
DataType::Int8 => "Int8",
DataType::Int16 => "Int16",
DataType::Int32 => "Int32",
DataType::Int64 => "Int64",
// The three columns the `Other` catch-all could not tell apart: `min_zoom`,
// `levels`, `height_source`.
DataType::UInt8 => "UInt8",
DataType::UInt16 => "UInt16",
DataType::UInt32 => "UInt32",
DataType::UInt64 => "UInt64",
DataType::Float16 => "Float16",
DataType::Float32 => "Float32",
DataType::Float64 => "Float64",
DataType::Binary => "Binary",
DataType::LargeBinary => "LargeBinary",
DataType::Utf8 => "Utf8",
DataType::LargeUtf8 => "LargeUtf8",
DataType::Date32 => "Date32",
DataType::Date64 => "Date64",
DataType::Null => "Null",
DataType::Struct(children) => {
let inner: Vec<String> = children
.iter()
.map(|f| format!("{}:{}", f.name(), short_type(f.data_type())))
.collect();
return std::borrow::Cow::Owned(format!("Struct{{{}}}", inner.join(",")));
}
DataType::List(item) => {
return std::borrow::Cow::Owned(format!("List<{}>", short_type(item.data_type())));
}
DataType::LargeList(item) => {
return std::borrow::Cow::Owned(format!(
"LargeList<{}>",
short_type(item.data_type())
));
}
// NOT a catch-all spelling: a type this crate has never put in a schema still
// gets a spelling of its OWN, so it can never collide with another one.
other => return std::borrow::Cow::Owned(format!("{other:?}")),
};
std::borrow::Cow::Borrowed(s)
}
#[cfg(test)]
mod recipe_type_spelling_tests {
use super::*;
use arrow::datatypes::{Field, Fields};
use std::collections::BTreeMap;
/// Every arrow type this crate actually puts in a schema, plus the near neighbours a
/// retype would realistically reach for. Built by hand, NOT from `short_type`, so the
/// test cannot be the `korp_tile_options().zooms == DEFAULT_TILE_ZOOMS` shape where
/// both sides come out of the same function and the guard can never go red.
fn types_in_play() -> Vec<DataType> {
vec![
DataType::Boolean,
DataType::Int8,
DataType::Int16,
DataType::Int32,
DataType::Int64,
DataType::UInt8,
DataType::UInt16,
DataType::UInt32,
DataType::UInt64,
DataType::Float32,
DataType::Float64,
DataType::Binary,
DataType::LargeBinary,
DataType::Utf8,
DataType::LargeUtf8,
DataType::Date32,
DataType::Struct(Fields::from(vec![Field::new(
"xmin",
DataType::Float32,
true,
)])),
DataType::List(Arc::new(Field::new("item", DataType::Int64, true))),
// Two types NO arm below names — the fallback must still separate them.
DataType::Duration(arrow::datatypes::TimeUnit::Second),
DataType::Time32(arrow::datatypes::TimeUnit::Second),
]
}
/// ★ **RED-when-broken: no two types may share a spelling.**
///
/// This is the whole property. **Seen red 2026-08-31** by restoring the
/// `_ => "Other"` catch-all, which collapsed eight distinct types onto one string:
///
/// ```text
/// Other <- [Int16, UInt8, UInt16, UInt32, UInt64, Date32,
/// Duration(Second), Time32(Second)]
/// a recipe token cannot tell these types apart, so a retype would leave every
/// table on disk reading Current with contents that no longer match
/// ```
///
/// The `UInt8` entry in that list is `min_zoom`, `levels` and `height_source`.
#[test]
fn every_type_gets_its_own_spelling() {
let mut by_spelling: BTreeMap<String, Vec<DataType>> = BTreeMap::new();
for t in types_in_play() {
by_spelling
.entry(short_type(&t).into_owned())
.or_default()
.push(t);
}
for (spelling, types) in &by_spelling {
println!("{spelling:>16} <- {types:?}");
}
let collisions: Vec<_> = by_spelling.iter().filter(|(_, v)| v.len() > 1).collect();
assert!(
collisions.is_empty(),
"a recipe token cannot tell these types apart, so a retype would leave every \
table on disk reading Current with contents that no longer match: \
{collisions:?}"
);
}
/// ★ **RED-when-broken: the `UInt8` columns this crate really writes are separable.**
///
/// Named rather than generic, because these three are the live instance of the
/// defect: `shared::schema::min_zoom_field` and the buildings side output's `levels`
/// and `height_source`. Retyping any one of them must be visible in the token.
#[test]
fn a_uint8_column_is_distinguishable_from_its_retypes() {
let mz = min_zoom_field();
assert_eq!(
mz.data_type(),
&DataType::UInt8,
"min_zoom stopped being UInt8 — this test is now pointed at nothing"
);
let spellings: Vec<String> = [
DataType::UInt8,
DataType::UInt16,
DataType::UInt32,
DataType::Int8,
DataType::Int16,
]
.iter()
.map(|t| format!("{}:{}", mz.name(), short_type(t)))
.collect();
println!("min_zoom under five types: {spellings:?}");
let mut uniq = spellings.clone();
uniq.sort();
uniq.dedup();
assert_eq!(
uniq.len(),
spellings.len(),
"two of these retypes of `min_zoom` spell the same in a recipe token: \
{spellings:?}"
);
assert_eq!(spellings[0], "min_zoom:UInt8");
}
/// ★ **RED-when-broken: a retype of a `bbox` LEAF must move the spelling.**
///
/// Until 2026-09-01 `Struct` and `List` were spelled without their children — "the
/// children of `bbox` are fixed by GeoParquet 1.1" — which is the accepted collision
/// the doc above used to name: `Struct(a)` and `Struct(b)` read alike, so retyping
/// `bbox.xmin` from `Float32` to `Float64` (the one retype the covering column will
/// ever realistically see) left every packed table reading Current with a covering
/// half the width the recipe implied. GeoParquet fixes the NAMES of the leaves, not
/// their width; 1.1 allows both. The spelling now recurses.
///
/// Seen RED 2026-09-01 with the childless arms: `Struct == Struct` for the two
/// covering structs, and the network token spelled `bbox:Struct`.
#[test]
fn a_retyped_bbox_leaf_moves_the_spelling() {
fn covering(leaf: DataType) -> DataType {
DataType::Struct(Fields::from(vec![
Field::new("xmin", leaf.clone(), true),
Field::new("ymin", leaf.clone(), true),
Field::new("xmax", leaf.clone(), true),
Field::new("ymax", leaf, true),
]))
}
let f32s = short_type(&covering(DataType::Float32)).into_owned();
let f64s = short_type(&covering(DataType::Float64)).into_owned();
println!("Float32 leaves: {f32s}\nFloat64 leaves: {f64s}");
assert_ne!(
f32s, f64s,
"a Float32 → Float64 retype of every bbox leaf did not move the spelling, so \
every packed table reads Current with a covering of the wrong width"
);
// ONE leaf, not all four — the partial retype is the one a hand edit produces.
let one = DataType::Struct(Fields::from(vec![
Field::new("xmin", DataType::Float64, true),
Field::new("ymin", DataType::Float32, true),
Field::new("xmax", DataType::Float32, true),
Field::new("ymax", DataType::Float32, true),
]));
assert_ne!(short_type(&one), short_type(&covering(DataType::Float32)));
// A renamed leaf is a different covering too — the reader looks the leaves up by
// name — and the pinned spelling is the literal a reader of the token expects.
assert_eq!(
f32s,
"Struct{xmin:Float32,ymin:Float32,xmax:Float32,ymax:Float32}",
"the struct spelling must name every child with its type"
);
// Lists: `node_refs` is `List<Int64>` and an `Int32` retype would truncate ids.
let i64s = short_type(&DataType::List(Arc::new(Field::new(
"item",
DataType::Int64,
true,
))))
.into_owned();
let i32s = short_type(&DataType::List(Arc::new(Field::new(
"item",
DataType::Int32,
true,
))))
.into_owned();
assert_ne!(i64s, i32s, "a List<Int64> → List<Int32> retype did not move the spelling");
assert_eq!(i64s, "List<Int64>");
// Recursion goes all the way down: a struct inside a list inside a struct.
let nested = DataType::Struct(Fields::from(vec![Field::new(
"rings",
DataType::List(Arc::new(Field::new("item", covering(DataType::Float32), true))),
true,
)]));
assert_eq!(
short_type(&nested),
"Struct{rings:List<Struct{xmin:Float32,ymin:Float32,xmax:Float32,ymax:Float32}>}"
);
// Applied: the network token spells the covering the packer actually writes, so
// the retype is visible where the staleness verdict is made.
let tok = crate::network::NetworkRecipe::current(&crate::network::ROADS, 8_192).token();
assert!(
tok.contains(&format!("bbox:{f32s}")),
"the network recipe does not spell the covering's leaves: {tok}"
);
}
/// ★ **RED-when-broken: a refactor of the spelling must not move a TILE token.**
///
/// `tiles.rs` had its own copy of this match and `network.rs` had another. Collapsing
/// them (2026-08-31) was only free because every type a TILE column uses spelled the
/// same as before, and this pin is what proved it: the literal a tile on the live
/// Sweden cut declared (`.../tiles/z6/36/19.parquet`, read 2026-08-31) was
/// `cols=id:Int64,geometry:Binary,tags:Utf8,bbox:Struct`.
///
/// **The literal moved ONCE, deliberately, on 2026-09-01**, when `Struct` learned to
/// spell its children so a `bbox` leaf retype is visible — that re-cuts the pyramid
/// one time, and the cost is written in the design doc. The pin now holds the
/// spelling a tile cut AFTER that change declares, so the next refactor that would
/// re-cut 1 635 971 tiles for nothing is still caught here.
#[test]
fn the_tile_columns_spell_exactly_what_the_cut_pyramid_declares() {
let spelled: Vec<String> = crate::tiles::tile_fields()
.iter()
.map(|f| format!("{}:{}", f.name(), short_type(f.data_type())))
.collect();
assert_eq!(
spelled.join(","),
"id:Int64,geometry:Binary,tags:Utf8,\
bbox:Struct{xmin:Float32,ymin:Float32,xmax:Float32,ymax:Float32}",
"the shared spelling moved the TILE recipe — 1 635 971 tiles on the Sweden \
cut alone now read Stale for a refactor that changed no tile"
);
}
}
#[cfg(test)]
mod geometry_logical_type_tests {
use super::*;
use arrow::array::{ArrayRef, BinaryArray};
use arrow::record_batch::RecordBatch;
use parquet::arrow::ArrowWriter;
use parquet::basic::LogicalType;
use parquet::file::reader::{FileReader, SerializedFileReader};
use std::sync::Arc;
/// Little-endian WKB `Point` — the smallest thing the accumulator can measure.
fn wkb_point(x: f64, y: f64) -> Vec<u8> {
let mut v = Vec::with_capacity(21);
v.push(1u8);
v.extend_from_slice(&1u32.to_le_bytes());
v.extend_from_slice(&x.to_le_bytes());
v.extend_from_slice(&y.to_le_bytes());
v
}
/// Write ONE `geometry` column through `ArrowWriter` and read the footer back.
/// Returns `(logical type, geo_statistics present)`.
fn round_trip(field: Field) -> (Option<LogicalType>, bool) {
let schema = Arc::new(Schema::new(vec![field]));
let col: ArrayRef = Arc::new(BinaryArray::from_vec(vec![
&wkb_point(10.594, 55.137)[..],
&wkb_point(24.178, 69.060)[..],
]));
let batch = RecordBatch::try_new(schema.clone(), vec![col]).unwrap();
let mut buf: Vec<u8> = Vec::new();
let mut w = ArrowWriter::try_new(&mut buf, schema, None).unwrap();
w.write(&batch).unwrap();
w.close().unwrap();
let reader = SerializedFileReader::new(bytes::Bytes::from(buf)).unwrap();
let col = reader.metadata().row_group(0).column(0);
(
col.column_descr().logical_type_ref().cloned(),
col.geo_statistics().is_some(),
)
}
/// ★ **RED-when-broken: [`geometry_field`] stops declaring the extension type.**
///
/// ⚠ **THIS GUARD IS ROUND-TRIP-SHAPED, AND THAT IS NOT COVERAGE OF THE PIPELINE.**
///
/// It takes a `Field` and hands it straight to an `ArrowWriter`. A guard that STARTS
/// from the field can only ever prove the field works. It is blind, by construction,
/// to every step in between — a `Schema` rebuilt from names and `DataType`s, a field
/// cloned without its metadata, or (the one that actually shipped, `geo-strip-1`) a
/// geometry field carried out of an INPUT FILE that never had the extension type.
///
/// This test and [`the_rung_fields_are_the_geometry_field_renamed`] were both green
/// on 2026-08-31 while `~/scratch/reader-gen/ways.parquet` — a real
/// packed table off the shipped pipeline — carried
/// `optional binary geometry;` with `is_geo_stats_set = False`.
///
/// So keep this as what it is: a UNIT check on the one writer, with a control arm
/// that proves the mechanism is real. The coverage of what lands on disk is
/// `tests/geometry_type_on_disk.rs`, which runs the real stages and reads the type
/// out of the FOOTER of every parquet they wrote.
///
/// Both arms here run on the SAME bytes, so a green cannot be the WKB encoder being
/// agreeable — it is the field declaration and nothing else. Restore
/// `Field::new("geometry", DataType::Binary, true)` in [`geometry_field`] and the
/// first arm reports `logical type: None / geo_statistics: NONE`, which is exactly
/// what the control arm asserts a BARE field does.
#[test]
fn geometry_columns_carry_the_logical_type() {
// The control: identical bytes, no extension type. If this ever comes back
// Geometry the test below proves nothing, because everything would pass.
let (plain_type, plain_stats) =
round_trip(Field::new("geometry", DataType::Binary, true));
assert!(
plain_type.is_none() && !plain_stats,
"control arm is not a control: a BARE Binary field came back as \
{plain_type:?} / geo_statistics {plain_stats}. Without a field that fails \
this way the assertion below is vacuous."
);
// Every production schema this crate writes WKB through.
let mut checked = 0usize;
for (what, schema) in [
("nodes", nodes_schema()),
("ways", ways_schema(true)),
("ways (no node_refs)", ways_schema(false)),
("relations", relations_schema()),
] {
let field = schema
.field_with_name("geometry")
.unwrap_or_else(|_| panic!("{what} declares no geometry column"))
.clone();
let (ty, stats) = round_trip(field);
assert!(
matches!(ty, Some(LogicalType::Geometry(_))),
"{what}: geometry came back as {ty:?}, not LogicalType::Geometry — a \
reader cannot learn from the schema that this column is geometry"
);
assert!(
stats,
"{what}: LogicalType::Geometry is declared but geo_statistics is absent \
— the writer-side accumulator did not run"
);
checked += 1;
}
assert!(
checked >= 4,
"only {checked} schemas were examined; a scan that reaches nothing reports \
clean for the wrong reason"
);
}
/// ★ **RED-when-broken: FOUR rungs, and they are korp's ladder.**
///
/// korp's level type is a four-variant enum. A fifth generalisation COLUMN forces a
/// fifth variant — the trap the z9 lane hit on 2026-08-24 — and a rung this crate
/// writes that korp's ladder does not have is a column nothing can ever select.
///
/// So both directions are asserted: [`GENERALISED_RUNGS`] must be exactly the
/// ladder's first three rungs, and the fourth (finest) rung must be absent, because
/// `geometry` IS that rung. The `RUNGS` import is the control — if the ladder itself
/// changed, this fails rather than quietly agreeing with a moved target.
#[test]
fn generalisation_rungs_match_korps_ladder() {
use crate::shared::class::RUNGS;
assert_eq!(
RUNGS.len(),
4,
"korp's ladder is no longer four rungs ({RUNGS:?}) — the four-variant level \
enum this column set is bound to has moved and this whole design needs \
re-deciding, not a silent extra column"
);
assert_eq!(
GENERALISED_RUNGS.to_vec(),
RUNGS[..3].to_vec(),
"the generalisation columns are {GENERALISED_RUNGS:?} against a ladder of \
{RUNGS:?} — they must be its first three rungs exactly"
);
assert!(
!GENERALISED_RUNGS.contains(&RUNGS[3]),
"rung {} has its own column AND is `geometry` — two columns for one rung",
RUNGS[3]
);
// The names are the ones korp will project by, spelled once.
assert_eq!(
GENERALISED_RUNGS
.iter()
.map(|&z| generalised_geometry_name(z))
.collect::<Vec<_>>(),
vec!["geometry_z6", "geometry_z9", "geometry_z12"],
);
}
/// ★ **RED-when-broken: a rung column is [`geometry_field`], renamed — not a twin.**
///
/// ⚠ Round-trip-shaped, exactly like the test above — see its header for why that is
/// a unit check on `with_name` and not coverage of the pipeline. What it DOES settle,
/// and it is worth settling: `Field::with_name` preserves field metadata, so a rung
/// column really is the one writer renamed. That is confirmed off disk too — the
/// rung columns of the shipped `reader-gen/ways.parquet` carried
/// `Geometry(crs=srid:0)` while its base `geometry` column did not.
///
/// The control is the same bare-`Binary` field the test above uses: if that ever
/// comes back Geometry, nothing here means anything.
#[test]
fn the_rung_fields_are_the_geometry_field_renamed() {
let (plain_type, plain_stats) = round_trip(Field::new("geometry_z12", DataType::Binary, true));
assert!(
plain_type.is_none() && !plain_stats,
"control arm is not a control: a BARE Binary field came back as \
{plain_type:?} / geo_statistics {plain_stats}"
);
let base = geometry_field();
for &z in &GENERALISED_RUNGS {
let f = generalised_geometry_field(z);
assert_eq!(f.name(), &generalised_geometry_name(z));
assert_eq!(
f.data_type(),
base.data_type(),
"geometry_z{z} is not the same arrow type as `geometry`"
);
assert_eq!(
f.metadata(),
base.metadata(),
"geometry_z{z} carries different field metadata from `geometry` — it was \
built with `Field::new` instead of through `geometry_field`, so it is a \
twin that will drift the first time the extension type changes"
);
let (ty, stats) = round_trip(f);
assert!(
matches!(ty, Some(LogicalType::Geometry(_))),
"geometry_z{z} came back as {ty:?}, not LogicalType::Geometry"
);
assert!(stats, "geometry_z{z} has no geo_statistics");
}
}
/// ★ **RED-when-broken: the extension type must not move a recipe token.**
///
/// `crate::tiles::TileRecipe` derives its `columns` term from `f.name()` and
/// `short_type(f.data_type())` — nothing else. If [`geometry_field`] ever changed
/// either, every cut pyramid on every box would be judged *Stale* and re-cut:
/// 1 635 971 tiles for a metadata edit. Pinned here rather than discovered there.
/// ★ **RED-when-broken: the generalisation columns must not move the TILE recipe.**
///
/// [`crate::tiles::TileRecipe`] derives its `columns` term from
/// [`crate::tiles::tile_fields`]. The generalisation columns are appended by
/// `spatial::pack_batches` onto the BASE tables, not by the tile writer onto a tile,
/// so the tile schema must be untouched — otherwise this delta re-cuts **1 635 971
/// tiles** for a change that does not appear in one of them.
///
/// (The NETWORK recipe is the opposite case and moves deliberately — see
/// `NetworkRecipe::for_table`. That re-cut is 21.3 s, not 222 s + 45 GB.)
///
/// **Red arm:** add a rung field to `tile_fields()` and the column count changes.
#[test]
fn the_generalisation_columns_do_not_move_the_tile_recipe() {
let names: Vec<String> = crate::tiles::tile_fields()
.iter()
.map(|f| f.name().clone())
.collect();
assert_eq!(
names,
vec!["id", "geometry", "tags", "bbox"],
"the TILE schema changed — every cut pyramid on every box now reads Stale \
and is re-cut: 1 635 971 tiles for a change that lives on the base tables"
);
for &z in &GENERALISED_RUNGS {
assert!(
!names.contains(&generalised_geometry_name(z)),
"a tile carries geometry_z{z} — a tile IS a rung, so a rung column inside \
one is the pyramid and the columns doing the same job twice"
);
}
}
#[test]
fn the_logical_type_does_not_move_the_recipe() {
let bare = Field::new("geometry", DataType::Binary, true);
let ours = geometry_field();
assert_eq!(ours.name(), bare.name(), "the recipe reads the column NAME");
assert_eq!(
ours.data_type(),
bare.data_type(),
"the recipe reads the column DataType — an extension type is field metadata \
and must leave the arrow type alone"
);
assert!(
!ours.metadata().is_empty(),
"geometry_field carries no metadata at all, so it cannot be declaring an \
extension type — this test would then pass for the wrong reason"
);
}
}