osm-katana/src/side_outputs/buildings.rs
//! **`buildings` — a table per concern, so a 3D view can extrude something other than a
//! default box.**
//!
//! # Why a table and not three columns on `ways.parquet`
//!
//! The height of a building is a fact about ~3.8 M of the 9.3 M rows in a Swedish
//! `ways.parquet` and about none of the rest. Three nullable columns on the way table
//! would be 5.5 M all-null cells that no producer can ever fill — the exact defect
//! [`crate::shared::schema::nodes_schema`] documents about `changeset` and `timestamp`
//! (205 037 958 such cells on the 2026-08-03 Sweden cut, and nothing anywhere went
//! wrong, because nothing could).
//!
//! `networks/` already proved the alternative: one table per concern, Hilbert-packed
//! with `covering.bbox` by [`crate::spatial::pack_batches`], read through the same
//! `query_ways_from` seam korp uses for everything else. This is that shape, for
//! buildings.
//!
//! # What is in it
//!
//! One row per feature carrying `building=*` or `building:part=*`, from `ways.parquet`
//! and from `relations.resolved.parquet`. MEASURED on korp's live Sweden cut
//! (2026-08-31): **3 836 448** such rows from `ways.parquet` and **5 035** from
//! `relations.resolved.parquet`, of which 26 have no geometry at all.
//!
//! The geometry is already extrudable at this layer, which is worth stating because the
//! z6 census said otherwise about the TILES: of the 3 836 471 building rows in
//! `ways.parquet`, **3 825 565 are WKB `Polygon`** and 10 905 are `LineString`
//! (unclosed outlines) — 99.72 % polygons, 1 with no geometry. The "all 57 160 z6
//! buildings were `Point`" problem is downstream of here, in the tile cut, and is lane
//! LOD's.
//!
//! The height columns come from [`crate::shared::height`], which is the ONE writer of
//! the height decision and carries the whole justification for it. The one number an
//! operator has to see before trusting this table is in [`BuildingStats::height_sources`]
//! and in the manifest, and for Sweden it is not flattering:
//!
//! | `height_source` | rows | share of buildings |
//! |---|---|---|
//! | `absent` | 3 675 499 | **95.6792 %** |
//! | `derived_from_levels` | 156 167 | 4.0653 % |
//! | `tagged_metres` | 9 551 | 0.2486 % |
//! | `rejected` | 265 | 0.0069 % |
//! | `tagged_feet` | **1** | 0.00003 % |
//!
//! 3 841 483 rows, 469 row groups, **297.5 MB** on disk, cut in **16.0 s** on oden by
//! `osm-katana buildings /home/korp/.korp/work/sweden -o …` (2026-08-31). That is this
//! table's own output, not an estimate: the numbers above are the `height_sources`
//! block of the `buildings.json` it wrote.
//!
//! DELTA 2's generalisation columns are most of those bytes: the SAME cut with
//! `--no-generalise` is **154.3 MB in 11.3 s** (measured, not inferred), so
//! `geometry_z6`/`z9`/`z12` cost **+92.8 % of file** and +4.7 s on this table. That is
//! the trade DELTA 2 already argued for; it is not re-argued here, only measured here.
//!
//! **That is the headline and it is not a bug in this table.** 95.68 % of Swedish
//! buildings have no height information in OpenStreetMap at all, and no parser can
//! conjure one. Only **4.31 %** get a height at all, and only **0.2487 %** of them are
//! heights a human actually stated — the other 94 % of that 4.31 % are
//! `building:levels x 3.2`, an assumption with a ±20 % median error.
//!
//! What this table changes is that the 4.31 % now reach the renderer as an `f32`
//! instead of as text inside a JSON blob nobody may parse per frame, that the
//! assumption is separable from the measurement, and that the 95.68 % are a countable,
//! named `absent` rather than a silent default nobody can see.
//!
//! **What it means for the 3D lane**, said plainly: a height column does not fill
//! Sweden with skyscrapers. It fills central Stockholm, Gothenburg and Malmö — where
//! `building:levels` is mapped — and leaves rural Sweden as the default box it is
//! today. Whether that is worth drawing differently (a per-`kind` fallback prism for
//! `absent`, honestly flagged as such) is a LOOK decision for the renderer, and this
//! table gives it the `height_source` column to make that decision WITH.
//!
//! # `is_part`, and the double-draw an extruder will otherwise commit
//!
//! Simple 3D Buildings says a `building:part` REPLACES the part of the outline it
//! covers. An extruder that draws both puts a 4-storey box inside a 20-storey tower and
//! z-fights it. So the distinction is a column and not something a consumer has to
//! re-derive from `tags`. **11 439** Swedish rows are parts, and **949** rows carry a
//! `min_height_m` — measured, and small enough that an extruder can ignore parts
//! entirely in v1 and lose almost nothing, as long as it knows it is doing so.
//!
//! `type=building` relations — the explicit outline↔parts grouping — are **out of scope
//! for this table**, and the reason is measured: `relations.resolved.parquet` (the only
//! relation table in the cut carrying resolved geometry) holds 274 206 rows, of which
//! 5 040 are buildings and 314 are parts. The grouping would be a join against
//! `relations.parquet`'s `members`, which is a different pass with a different cost, and
//! the spatial rule ("a part wins inside its own footprint") is what every 3D renderer
//! actually implements anyway because most part relationships are implicit containment
//! rather than a relation. `is_part` is what that rule needs.
//!
//! # Layout
//!
//! Written through [`crate::spatial::pack_batches`] — the ONE packer — so Hilbert order,
//! the outward-rounded `covering.bbox`, `BYTE_STREAM_SPLIT` on the bbox leaves and the
//! **8 192**-row row groups ([`crate::spatial::PACK_ROW_GROUP_ROWS`]) are the same ones
//! korp already prunes with. None of those are tuned here: lane PACK measured 100 000
//! rows/group at 2.03x–3.94x worse and the page index at 1.00x, and a literal in this
//! file would be a second writer of a tuning axis.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use anyhow::Context as _;
use arrow::array::{
Array, ArrayRef, BinaryArray, BooleanArray, Float32Array, Int64Array, RecordBatch,
StringArray, UInt8Array,
};
use arrow::datatypes::{DataType, Field, Schema};
use parquet::arrow::ProjectionMask;
use parquet::arrow::arrow_reader::{
ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
};
use crate::metadata::{self, GeoMeta};
use crate::network::{NetworkRecipe, tag_value};
use crate::shared::height::{self, BuildingHeight, HeightSource};
use crate::spatial::{PackOptions, PackStats, pack_batches};
/// The table's file name (without `.parquet`) — also the recipe's `net=` term.
pub const BUILDINGS_TABLE: &str = "buildings";
/// The manifest a buildings directory carries, beside `networks.json`.
pub const BUILDINGS_MANIFEST: &str = "buildings.json";
// ── the columns ─────────────────────────────────────────────────────────────────
/// **The columns the `buildings` table carries**, and the ONE place they are declared
/// (LAW 5): [`build_batches`] builds its arrays from this and the recipe describes itself
/// from this, so a schema change moves the staleness token with no edit anywhere else.
/// `covering.bbox` is appended by [`crate::spatial::pack_batches`] and is named in the
/// recipe by [`NetworkRecipe::for_table`], because it is that writer's contribution.
pub fn building_fields(roof_shape: bool) -> Vec<Field> {
let mut f = vec![
Field::new("id", DataType::Int64, false),
// **Which base table the row came from.** Way ids and relation ids are separate
// id spaces and both start at 1, so `id` alone is not a key; `(source, id)` is.
// One distinct value per source, so the dictionary makes the column free.
Field::new("source", DataType::Utf8, false),
// The `building=` value (`yes`, `apartments`, `house`, …), or the
// `building:part=` value when there is no `building=`. A renderer colours by it,
// and reading it from `tags` per frame is the `parse_tags` hotspot again.
Field::new("kind", DataType::Utf8, false),
// **This row is a `building:part`.** See the module docs: an extruder that
// ignores this draws the outline AND its parts.
Field::new("is_part", DataType::Boolean, false),
// Free from lane MINZOOM: the one tag → rung decision in this crate, valued by
// `shared::class::min_zoom_of_tags` from the row's own `tags`. It lands in the
// field list, so it is in the recipe automatically.
crate::shared::schema::min_zoom_field(),
// ── the point of the table ────────────────────────────────────────────────
// `f32`, not `f64`: a building height is metres with at most one decimal, and
// `f32` holds 7 significant digits — a 1 000 m tower to the millimetre. It is
// also the type `covering.bbox` already uses, and half the bytes.
// NULLABLE, and the null is the honest 95.7 %: see `height_source` for WHY it is
// null, which is the distinction a default value destroys.
Field::new("height_m", DataType::Float32, true),
Field::new("min_height_m", DataType::Float32, true),
// `building:levels`, rounded. `u8` because the tallest building on Earth has 163
// levels and the tallest in the Sweden cut has 18.
Field::new("levels", DataType::UInt8, true),
// **NON-null.** Every row has a source, including `absent` — a nullable stamp
// would let "we did not classify this" and "this has no height" wear the same
// cell, which is exactly what `min_zoom_field`'s doc argues against from the
// other side. Values are `HeightSource::code()`.
Field::new("height_source", DataType::UInt8, false),
crate::shared::schema::geometry_field(),
Field::new("tags", DataType::Utf8, true),
];
if roof_shape {
// **The RAW `roof:shape` value, never a bucket.** See [`ROOF_SHAPE_FIELD`].
f.push(roof_shape_field());
}
f
}
/// **The `roof:shape` column — OFF by default, and the raw tag value when it is on.**
///
/// # Why the raw string and not a code
///
/// Its one consumer, `facett_map::parse_roof_shape`, collapses ~15 OSM values into THREE
/// buckets (`Flat` / `Gabled` / `Hipped`) because that is what its extruder can build. A
/// `UInt8` bucket column here would be a SECOND copy of that classifier, in a crate that
/// cannot depend on facett (facett-map pulls `egui`; osm-katana is a cutter) — which is
/// exactly the shape `shared::height` is in, and the reason
/// `facett-map/tests/building_height_corpus.rs` exists to hold two texts of one decision
/// together by hand. One such pair is a known cost; two is a habit.
///
/// The raw value costs no more on disk than the bucket would — parquet dictionary-encodes
/// it and the whole Swedish zoo is a few dozen distinct strings — and it leaves the
/// bucket decision with the only crate that knows what its extruder can draw. A new OSM
/// roof value then reaches facett as data, not as a re-cut.
#[must_use]
pub fn roof_shape_field() -> Field {
Field::new("roof_shape", DataType::Utf8, true)
}
/// The row selection, spelled for the recipe. A table cut with a different rule must not
/// read as the same object as this one — the same reason `NetworkSpec::selector` exists.
pub fn buildings_selector() -> String {
format!(
"building|building:part;from:ways+relations.resolved;mpl={};src={}",
height::METRES_PER_LEVEL,
HeightSource::ALL
.iter()
.map(|s| format!("{}={}", s.code(), s.name()))
.collect::<Vec<_>>()
.join("/")
)
}
fn buildings_schema(roof_shape: bool) -> Arc<Schema> {
// Ways give Polygons (3 825 565 of 3 836 471 measured) and a residue of LineStrings
// where a building outline is not closed; relations give MultiPolygons.
let geo = GeoMeta::new(&["Polygon", "MultiPolygon", "LineString"]).to_json();
let mut meta = HashMap::new();
meta.insert(String::from(metadata::GEO_KEY), geo);
Arc::new(Schema::new_with_metadata(building_fields(roof_shape), meta))
}
/// The staleness token, through [`NetworkRecipe::for_table`] — the ONE recipe
/// constructor in the network family (LAW 5), already shared by the four network tables
/// and the two graph tables. A seventh constructor here would be the twinned-descriptor
/// defect `NetworkRecipe`'s doc is written about.
pub fn buildings_recipe(
row_group_rows: usize,
generalise: bool,
roof_shape: bool,
) -> NetworkRecipe {
NetworkRecipe::for_table(
BUILDINGS_TABLE,
&building_fields(roof_shape),
&buildings_selector(),
row_group_rows,
// DELTA 2's `geometry_z6` / `z9` / `z12` are contributed by the packer, exactly
// like `covering.bbox`, so the rung list comes from the SAME function the packer
// asks (`spatial::generalised_rungs_for`) rather than being spelled twice. A
// building footprint is an area, so this table generalises; if it ever stopped,
// the recipe would follow with no edit here.
&crate::spatial::generalised_rungs_for(&buildings_schema(roof_shape), generalise),
)
}
// ── source tables ───────────────────────────────────────────────────────────────
/// Which base table a row was read from. Its own enum rather than
/// [`crate::network::Source`] because that one is Ways/Nodes: a building is never a
/// node worth extruding, and it IS often a relation, which `network::Source` has no
/// spelling for.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum BuildingSource {
Ways,
Relations,
}
impl BuildingSource {
/// The file this source reads.
///
/// `relations.resolved.parquet` and NOT `relations.parquet`. The reason is measured
/// and is worth recording, because the two names look interchangeable: on korp's
/// Sweden cut `relations.parquet` holds **14 632 472** rows and
/// `relations.resolved.parquet` holds **274 206** — a 53x gap — and the resolved one
/// is the table with usable ring geometry. Reading the raw one would select 1 370 053
/// building-tagged relations whose geometry this table cannot draw.
pub fn file(self) -> &'static str {
match self {
BuildingSource::Ways => "ways.parquet",
BuildingSource::Relations => "relations.resolved.parquet",
}
}
/// The `source` column value.
pub fn name(self) -> &'static str {
match self {
BuildingSource::Ways => "way",
BuildingSource::Relations => "relation",
}
}
/// `ways.parquet` must exist; a cut with no resolved relations is a normal cut
/// (`osm-katana convert` writes it only when relations were resolved), so its
/// absence is a skip and not an error.
pub fn required(self) -> bool {
matches!(self, BuildingSource::Ways)
}
}
/// Both sources, in read order.
pub const ALL_BUILDING_SOURCES: &[BuildingSource] =
&[BuildingSource::Ways, BuildingSource::Relations];
// ── options and stats ───────────────────────────────────────────────────────────
/// Knobs for [`extract_buildings`].
#[derive(Clone, Debug)]
pub struct BuildingOptions {
pub compression: String,
/// Rows per row group — the bbox-pruning granularity. Defaults to
/// [`crate::spatial::PACK_ROW_GROUP_ROWS`], which carries the measurement.
pub row_group_rows: usize,
/// Refuse to hold more than this many selected rows in memory. A HARD stop, not a
/// warning — the same ceiling `NetworkOptions::max_rows` exists for, and for the
/// same incident (474.9 GiB RSS on this box, 2026-08-03). Sweden selects 3.84 M, so
/// the default leaves ~10x headroom.
pub max_rows: usize,
/// Include `building:part` rows. On by default: they are the rows that carry
/// `min_height`, and without them a tower's parts vanish.
pub include_parts: bool,
/// Emit DELTA 2's generalisation columns (`geometry_z6` / `z9` / `z12`). On by
/// default and for the same reason the networks have them: a z6 read of 3.8 M
/// building footprints that has only the full-precision column to project pays the
/// full WKB decode. A building footprint is an area, so this table qualifies —
/// `spatial::generalised_rungs_for` is what decides, from the schema, not this flag
/// alone.
pub generalise: bool,
/// Carry the raw `roof:shape` value as its own column ([`roof_shape_field`]).
///
/// **OFF by default and that is a MEASURED decision, not a default nobody chose** —
/// see the row in korp's `korp-v2.md` for the byte figure on the live cut. The 3D
/// lane wants the column; turning it on MOVES the recipe token (the column list is in
/// it), so every cut buildings table reads `Stale` and is re-cut, which is why the
/// switch is here rather than always-on.
pub roof_shape: bool,
}
impl Default for BuildingOptions {
fn default() -> Self {
Self {
compression: String::from("zstd"),
row_group_rows: crate::spatial::PACK_ROW_GROUP_ROWS,
max_rows: 40_000_000,
include_parts: true,
generalise: true,
roof_shape: false,
}
}
}
/// What one buildings cut produced.
#[derive(Clone, Debug)]
pub struct BuildingStats {
pub path: PathBuf,
pub rows: usize,
pub rows_no_geometry: usize,
pub row_groups: usize,
pub bytes: u64,
/// Rows that came from each base table.
pub by_source: Vec<(String, usize)>,
/// `building=` / `building:part=` value → row count.
pub kinds: Vec<(String, usize)>,
/// **The coverage, in the object.** [`HeightSource::name`] → row count, for every
/// variant including the zeroes. This is what an operator reads to learn that 95.7 %
/// of a country's buildings have no height, WITHOUT opening the parquet — and what a
/// later run is diffed against to notice a parser regression (see the
/// `Rejected`-vs-`Absent` argument in [`crate::shared::height::HeightSource`]).
pub height_sources: Vec<(String, usize)>,
/// Rows carrying a `min_height_m`.
pub with_min_height: usize,
/// Rows that are `building:part`.
pub parts: usize,
pub recipe: String,
}
impl BuildingStats {
/// Rows whose height a human actually stated, as a fraction of all rows. The ratio
/// LAW 7 asks for, computed in one place so a report and a manifest cannot disagree.
pub fn surveyed_share(&self) -> f64 {
let n: usize = self
.height_sources
.iter()
.filter(|(k, _)| {
HeightSource::ALL
.iter()
.any(|s| s.name() == k && s.is_surveyed())
})
.map(|(_, n)| *n)
.sum();
n as f64 / self.rows.max(1) as f64
}
/// Rows with ANY height, surveyed or derived.
pub fn any_height_share(&self) -> f64 {
let n: usize = self
.height_sources
.iter()
.filter(|(k, _)| k != HeightSource::Absent.name() && k != HeightSource::Rejected.name())
.map(|(_, n)| *n)
.sum();
n as f64 / self.rows.max(1) as f64
}
}
// ── the rows ────────────────────────────────────────────────────────────────────
/// One selected row, held until the whole scan is done (the Hilbert sort is global).
#[derive(Default)]
struct Rows {
id: Vec<i64>,
source: Vec<&'static str>,
kind: Vec<String>,
is_part: Vec<bool>,
height: Vec<BuildingHeight>,
geom: Vec<Option<Vec<u8>>>,
tags: Vec<String>,
}
impl Rows {
fn len(&self) -> usize {
self.id.len()
}
fn absorb(&mut self, o: Rows) {
self.id.extend(o.id);
self.source.extend(o.source);
self.kind.extend(o.kind);
self.is_part.extend(o.is_part);
self.height.extend(o.height);
self.geom.extend(o.geom);
self.tags.extend(o.tags);
}
}
/// **Is this feature a building, and what kind?** The ONE row-selection rule of this
/// table — `buildings_selector` spells it for the recipe and this decides it, so the
/// token and the data cannot disagree.
///
/// Returns `(kind, is_part)`. `building=no` is refused: it is the documented way to say
/// "there is no building here", and extruding it would put a box on an explicit denial.
pub fn building_of_tags(tags: &str) -> Option<(String, bool)> {
let b = tag_value(tags, "building");
let p = tag_value(tags, "building:part");
let part = matches!(p.as_deref(), Some(v) if v != "no");
match (b.as_deref(), part) {
(Some(v), _) if v != "no" && !v.is_empty() => Some((String::from(v), part)),
(_, true) => {
let v = p.as_deref().unwrap_or("yes");
Some((String::from(v), true))
}
_ => None,
}
}
// ── extraction ──────────────────────────────────────────────────────────────────
/// **Cut the `buildings` table out of a region extract.**
///
/// One pass per source table, one gatling unit per ROW GROUP (ROOT LAW #0 — no rayon),
/// LPT-scheduled heaviest-first by compressed footer size; each unit opens its own
/// projected reader over its single group. Results come back in row-group order, so the
/// selection is a pure function of the input.
///
/// The write goes through [`crate::spatial::pack_batches`], so there is no second layout
/// writer to keep in step with `networks/`.
pub fn extract_buildings(
extract: &Path,
out_dir: &Path,
opts: &BuildingOptions,
) -> anyhow::Result<BuildingStats> {
let mut acc = Rows::default();
for source in ALL_BUILDING_SOURCES {
let table = extract.join(source.file());
if !table.is_file() {
anyhow::ensure!(
!source.required(),
"no {} under {extract:?}",
source.file()
);
continue;
}
let meta = {
let f = std::fs::File::open(&table).with_context(|| format!("open {table:?}"))?;
ArrowReaderMetadata::load(&f, ArrowReaderOptions::new())?
};
let pq_meta = meta.metadata().clone();
let groups: Vec<usize> = (0..pq_meta.num_row_groups()).collect();
let per_group = gatling::gatling_forkjoin::gatling_map_balanced(
&groups,
0,
1,
|rg: &usize| pq_meta.row_group(*rg).compressed_size().max(0) as u64,
|_i, rg: &usize| scan_row_group(&table, &meta, *rg, *source, opts.include_parts),
);
for r in per_group {
acc.absorb(r?);
}
// Charge the ceiling after EVERY source, not only at the end.
anyhow::ensure!(
acc.len() <= opts.max_rows,
"buildings extraction selected {} rows, over the {} row ceiling — refusing to \
build the batches (raise BuildingOptions::max_rows deliberately)",
acc.len(),
opts.max_rows
);
}
std::fs::create_dir_all(out_dir)?;
let by_source = histogram(acc.source.iter().copied());
let kinds = histogram(acc.kind.iter().map(String::as_str));
let height_sources = source_histogram(&acc.height);
let with_min_height = acc.height.iter().filter(|h| h.min_height_m.is_some()).count();
let parts = acc.is_part.iter().filter(|p| **p).count();
let schema = buildings_schema(opts.roof_shape);
let recipe = buildings_recipe(opts.row_group_rows, opts.generalise, opts.roof_shape);
let dst = out_dir.join(format!("{BUILDINGS_TABLE}.parquet"));
let batches = build_batches(&schema, acc, opts.roof_shape)?;
let pack = PackOptions {
compression: opts.compression.clone(),
row_group_rows: opts.row_group_rows,
covering: true,
generalise: opts.generalise,
node_refs_sidecar: false,
};
let stats: PackStats = pack_batches(
batches,
&schema,
&dst,
&pack,
&[(
String::from(crate::network::NETWORK_RECIPE_META_KEY),
recipe.token(),
)],
)?;
let out = BuildingStats {
path: dst,
rows: stats.rows,
rows_no_geometry: stats.rows_no_geometry,
row_groups: stats.row_groups,
bytes: stats.bytes,
by_source,
kinds,
height_sources,
with_min_height,
parts,
recipe: recipe.token(),
};
std::fs::write(out_dir.join(BUILDINGS_MANIFEST), manifest_json(&out))?;
Ok(out)
}
fn scan_row_group(
path: &Path,
meta: &ArrowReaderMetadata,
rg: usize,
source: BuildingSource,
include_parts: bool,
) -> anyhow::Result<Rows> {
let f = std::fs::File::open(path)?;
let builder = ParquetRecordBatchReaderBuilder::new_with_metadata(f, meta.clone());
let mask = ProjectionMask::columns(builder.parquet_schema(), ["id", "geometry", "tags"]);
let reader = builder
.with_row_groups(vec![rg])
.with_projection(mask)
.build()?;
let mut out = Rows::default();
for batch in reader {
let b = batch?;
let ids = col::<Int64Array>(&b, "id")?;
let geoms = col::<BinaryArray>(&b, "geometry")?;
let tags = col::<StringArray>(&b, "tags")?;
for r in 0..b.num_rows() {
let t = if tags.is_null(r) { "" } else { tags.value(r) };
if t.is_empty() {
continue;
}
let Some((kind, is_part)) = building_of_tags(t) else {
continue;
};
if is_part && !include_parts {
continue;
}
out.id.push(ids.value(r));
out.source.push(source.name());
out.kind.push(kind);
out.is_part.push(is_part);
out.height.push(height::of_tags(t));
out.geom
.push((!geoms.is_null(r)).then(|| geoms.value(r).to_vec()));
out.tags.push(String::from(t));
}
}
Ok(out)
}
fn col<'a, T: 'static>(b: &'a RecordBatch, name: &str) -> anyhow::Result<&'a T> {
b.column_by_name(name)
.and_then(|c| c.as_any().downcast_ref::<T>())
.ok_or_else(|| anyhow::anyhow!("column `{name}` missing or wrong type"))
}
/// Build the record batches. Chunked so no single Arrow array has to hold a whole
/// country's geometry contiguously — the same 65 536 `network::build_batches` uses.
fn build_batches(
schema: &Arc<Schema>,
rows: Rows,
roof_shape: bool,
) -> anyhow::Result<Vec<RecordBatch>> {
const CHUNK: usize = 65_536;
let n = rows.len();
let Rows {
id,
source,
kind,
is_part,
height,
geom,
tags,
} = rows;
let mut out = Vec::with_capacity(n.div_ceil(CHUNK).max(1));
let mut at = 0usize;
while at < n {
let end = (at + CHUNK).min(n);
let mut cols: Vec<ArrayRef> = vec![
Arc::new(Int64Array::from(id[at..end].to_vec())),
Arc::new(StringArray::from_iter_values(
source[at..end].iter().copied(),
)),
Arc::new(StringArray::from_iter_values(
kind[at..end].iter().map(String::as_str),
)),
Arc::new(BooleanArray::from(is_part[at..end].to_vec())),
Arc::new(UInt8Array::from_iter_values(
tags[at..end]
.iter()
.map(|t| crate::shared::class::min_zoom_of_tags(t)),
)),
Arc::new(Float32Array::from_iter(
height[at..end].iter().map(|h| h.height_m),
)),
Arc::new(Float32Array::from_iter(
height[at..end].iter().map(|h| h.min_height_m),
)),
Arc::new(UInt8Array::from_iter(height[at..end].iter().map(|h| h.levels))),
Arc::new(UInt8Array::from_iter_values(
height[at..end].iter().map(|h| h.source.code()),
)),
Arc::new(BinaryArray::from_opt_vec(
geom[at..end]
.iter()
.map(|g| g.as_deref())
.collect::<Vec<_>>(),
)),
Arc::new(StringArray::from_iter_values(
tags[at..end].iter().map(String::as_str),
)),
];
if roof_shape {
// Derived from the row's OWN `tags`, the same way `min_zoom` above is: the
// string is already resident, so the column costs no extra RAM in the scan.
cols.push(Arc::new(StringArray::from_iter(
tags[at..end]
.iter()
.map(|t| tag_value(t, "roof:shape").map(|v| v.into_owned())),
)));
}
out.push(RecordBatch::try_new(schema.clone(), cols)?);
at = end;
}
if out.is_empty() {
out.push(RecordBatch::new_empty(schema.clone()));
}
Ok(out)
}
fn histogram<'a>(vals: impl Iterator<Item = &'a str>) -> Vec<(String, usize)> {
let mut m: HashMap<&str, usize> = HashMap::new();
for v in vals {
*m.entry(v).or_insert(0) += 1;
}
let mut v: Vec<(String, usize)> = m.into_iter().map(|(k, n)| (String::from(k), n)).collect();
v.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
v
}
/// The `height_source` histogram, over **every** variant — including the ones with zero
/// rows. A histogram that omits its empty buckets cannot tell "no feet in this country"
/// from "the feet branch is gone", which is the whole reason the stamp exists.
fn source_histogram(hs: &[BuildingHeight]) -> Vec<(String, usize)> {
HeightSource::ALL
.iter()
.map(|s| {
(
String::from(s.name()),
hs.iter().filter(|h| h.source == *s).count(),
)
})
.collect()
}
/// The manifest, beside `networks.json` and in the same spirit: a reader learns what is
/// in the directory and whether it is current WITHOUT opening the parquet.
pub fn manifest_json(s: &BuildingStats) -> String {
let kv = |v: &[(String, usize)]| {
v.iter()
.map(|(k, n)| format!(" {:?}: {n}", k))
.collect::<Vec<_>>()
.join(",\n")
};
format!(
"{{\n \"table\": {:?},\n \"rows\": {},\n \"row_groups\": {},\n \"bytes\": {},\n \
\"rows_no_geometry\": {},\n \"parts\": {},\n \"with_min_height\": {},\n \
\"surveyed_share\": {:.6},\n \"any_height_share\": {:.6},\n \
\"height_sources\": {{\n{}\n }},\n \"by_source\": {{\n{}\n }},\n \
\"kinds\": {{\n{}\n }},\n \"recipe\": {:?}\n}}\n",
BUILDINGS_TABLE,
s.rows,
s.row_groups,
s.bytes,
s.rows_no_geometry,
s.parts,
s.with_min_height,
s.surveyed_share(),
s.any_height_share(),
kv(&s.height_sources),
kv(&s.by_source),
kv(&s.kinds.iter().take(40).cloned().collect::<Vec<_>>()),
s.recipe,
)
}
// ── tests ───────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use crate::spatial::wkb_bbox;
use parquet::file::statistics::Statistics;
/// A closed square ring as WKB Polygon, `size` degrees on a side at `(lon, lat)`.
fn wkb_square(lon: f64, lat: f64, size: f64) -> Vec<u8> {
let mut v = Vec::with_capacity(9 + 4 + 5 * 16);
v.push(1u8); // little endian
v.extend_from_slice(&3u32.to_le_bytes()); // Polygon
v.extend_from_slice(&1u32.to_le_bytes()); // one ring
v.extend_from_slice(&5u32.to_le_bytes()); // five points, closed
for (x, y) in [
(lon, lat),
(lon + size, lat),
(lon + size, lat + size),
(lon, lat + size),
(lon, lat),
] {
v.extend_from_slice(&x.to_le_bytes());
v.extend_from_slice(&y.to_le_bytes());
}
v
}
/// The fixture extent, so the prune guard's control can be checked against it.
const FIXTURE_LON: f64 = 20.0;
const FIXTURE_LAT: f64 = 10.0;
const FIXTURE_AREA: f64 = FIXTURE_LON * FIXTURE_LAT;
/// Scatter `k` over the fixture extent with no spatial locality whatsoever — this is
/// what makes convert order a real control for the Hilbert sort.
fn scatter(k: u64) -> (f64, f64) {
let h = k.wrapping_mul(0x9E37_79B9_7F4A_7C15) ^ (k << 17);
let a = ((h >> 11) & 0xFFFF) as f64 / 65_536.0;
let b = ((h >> 31) & 0xFFFF) as f64 / 65_536.0;
(10.0 + a * FIXTURE_LON, 55.0 + b * FIXTURE_LAT)
}
/// Write a `ways.parquet` in CONVERT order — the layout the packer has to beat.
fn fixture_ways(dir: &Path, n: usize) -> Vec<Vec<u8>> {
std::fs::create_dir_all(dir).unwrap();
let fields = vec![
Field::new("id", DataType::Int64, false),
Field::new("geometry", DataType::Binary, true),
Field::new("tags", DataType::Utf8, true),
Field::new("version", DataType::Int32, true),
];
let mut meta = HashMap::new();
meta.insert(
String::from(metadata::GEO_KEY),
GeoMeta::new(&["Polygon"]).to_json(),
);
let schema = Arc::new(Schema::new_with_metadata(fields, meta));
let mut ids = Vec::with_capacity(n);
let mut geoms: Vec<Vec<u8>> = Vec::with_capacity(n);
let mut tags = Vec::with_capacity(n);
for k in 0..n {
let (lon, lat) = scatter(k as u64);
ids.push(k as i64 + 1);
geoms.push(wkb_square(lon, lat, 0.001));
// A deliberate mix, so every HeightSource variant is exercised.
tags.push(match k % 8 {
0 => String::from(r#"{"building":"yes"}"#),
1 => String::from(r#"{"building":"house","height":"7.5"}"#),
2 => String::from(r#"{"building":"apartments","building:levels":"5"}"#),
3 => String::from(r#"{"building":"yes","height":"25'"}"#),
4 => String::from(r#"{"building":"yes","height":"1;12"}"#),
5 => String::from(
r#"{"building:part":"yes","height":"30","min_height":"12"}"#,
),
6 => String::from(r#"{"highway":"residential"}"#), // NOT a building
_ => String::from(r#"{"building":"no"}"#), // an explicit denial
});
}
let batch = RecordBatch::try_new(
schema.clone(),
vec![
Arc::new(Int64Array::from(ids)),
Arc::new(BinaryArray::from_iter_values(geoms.iter().map(|g| &g[..]))),
Arc::new(StringArray::from_iter_values(tags.iter().map(String::as_str))),
Arc::new(arrow::array::Int32Array::from(vec![1i32; n])),
],
)
.unwrap();
let f = std::fs::File::create(dir.join("ways.parquet")).unwrap();
let mut w = parquet::arrow::ArrowWriter::try_new(f, schema, None).unwrap();
w.write(&batch).unwrap();
w.close().unwrap();
geoms
}
fn read_table(path: &Path) -> Vec<RecordBatch> {
let f = std::fs::File::open(path).unwrap();
ParquetRecordBatchReaderBuilder::try_new(f)
.unwrap()
.build()
.unwrap()
.map(|b| b.unwrap())
.collect()
}
/// ★ **RED-when-broken: the heights survive the ROUND TRIP, off disk.**
///
/// Every assertion here reads the written parquet back through a fresh reader, never
/// the in-memory batch — a batch that never reached a file proves nothing about the
/// arrow → parquet → arrow types, and `Float32` silently becoming `Float64`, or a
/// nullable column becoming non-null, is exactly the class of defect only a disk
/// round trip can see.
///
/// The unit assertions are the point: 25 feet must come back as **7.62 m** and not as
/// 25, and 5 levels must come back as **16.0 m** and not as 5.
#[test]
fn buildings_round_trip_off_disk_with_the_right_units() {
let tmp = tempfile::tempdir().unwrap();
let src = tmp.path().join("extract");
let dst = tmp.path().join("buildings");
fixture_ways(&src, 2_048);
let stats = extract_buildings(&src, &dst, &BuildingOptions::default()).unwrap();
// Six of every eight fixture rows are buildings (one is a highway, one is
// `building=no`).
assert_eq!(stats.rows, 2_048 / 8 * 6, "row selection moved");
assert_eq!(stats.parts, 2_048 / 8, "the building:part rows are missing");
let batches = read_table(&stats.path);
assert!(!batches.is_empty(), "the written table has no batches");
// Column types, as parquet gave them back — not as the writer declared them.
let s = batches[0].schema();
for (name, want) in [
("height_m", DataType::Float32),
("min_height_m", DataType::Float32),
("levels", DataType::UInt8),
("height_source", DataType::UInt8),
("is_part", DataType::Boolean),
] {
let f = s.field_with_name(name).unwrap();
assert_eq!(f.data_type(), &want, "{name} came back as {:?}", f.data_type());
}
assert!(
!s.field_with_name("height_source").unwrap().is_nullable(),
"height_source must be NON-null: a null stamp cannot say why a height is \
missing, which is the entire purpose of the column"
);
assert!(
s.field_with_name("height_m").unwrap().is_nullable(),
"height_m must be nullable: 95.7 % of real buildings have no height and a \
default would paint over that"
);
// ── the values, keyed by way id, read off disk ────────────────────────────
let mut seen: HashMap<i64, (Option<f32>, Option<f32>, Option<u8>, u8, bool)> =
HashMap::new();
for b in &batches {
let ids = col::<Int64Array>(b, "id").unwrap();
let h = col::<Float32Array>(b, "height_m").unwrap();
let mh = col::<Float32Array>(b, "min_height_m").unwrap();
let lv = col::<UInt8Array>(b, "levels").unwrap();
let hs = col::<UInt8Array>(b, "height_source").unwrap();
let ip = col::<BooleanArray>(b, "is_part").unwrap();
for r in 0..b.num_rows() {
seen.insert(
ids.value(r),
(
(!h.is_null(r)).then(|| h.value(r)),
(!mh.is_null(r)).then(|| mh.value(r)),
(!lv.is_null(r)).then(|| lv.value(r)),
hs.value(r),
ip.value(r),
),
);
}
}
assert_eq!(seen.len(), stats.rows, "ids are not unique in the fixture");
// id = k+1, and the tags were chosen by k % 8.
let row = |k: usize| seen[&(k as i64 + 1)];
// k%8 == 0: `building=yes` and nothing else. NULL height, source `absent`.
let (h, _, _, src, part) = row(0);
assert_eq!(h, None, "an untagged building must come back NULL, not 0.0");
assert_eq!(
HeightSource::from_code(src),
Some(HeightSource::Absent),
"an untagged building must be stamped absent"
);
assert!(!part);
// k%8 == 1: `height=7.5` — metres, verbatim.
let (h, _, _, src, _) = row(1);
assert!((h.unwrap() - 7.5).abs() < 1e-4, "height=7.5 came back {h:?}");
assert_eq!(HeightSource::from_code(src), Some(HeightSource::TaggedMetres));
// k%8 == 2: `building:levels=5` — 5 x 3.2 = 16.0 m, and levels = 5.
let (h, _, lv, src, _) = row(2);
assert!(
(h.unwrap() - 16.0).abs() < 1e-4,
"5 levels must come back as 16.0 m (5 x METRES_PER_LEVEL), got {h:?} — a \
column holding the LEVEL COUNT where metres belong is a 5 m apartment block"
);
assert_eq!(lv, Some(5));
assert_eq!(
HeightSource::from_code(src),
Some(HeightSource::DerivedFromLevels)
);
// k%8 == 3: `height=25'` — FEET. 7.62 m off disk, not 25.
let (h, _, _, src, _) = row(3);
assert!(
(h.unwrap() - 7.62).abs() < 1e-3,
"height=25' must come back as 7.62 m, got {h:?} — the unit was lost between \
the parser and the file"
);
assert_eq!(HeightSource::from_code(src), Some(HeightSource::TaggedFeet));
// k%8 == 4: `height=1;12` — present, unusable. NULL height, stamped `rejected`,
// which is what distinguishes it from row 0.
let (h, _, _, src, _) = row(4);
assert_eq!(h, None);
assert_eq!(
HeightSource::from_code(src),
Some(HeightSource::Rejected),
"a height tag that failed to parse must NOT be indistinguishable from no \
height tag at all"
);
// k%8 == 5: a part, lifted off the ground.
let (h, mh, _, src, part) = row(5);
assert!((h.unwrap() - 30.0).abs() < 1e-4);
assert!(
(mh.unwrap() - 12.0).abs() < 1e-4,
"min_height=12 came back {mh:?} — without it the part is drawn from the \
ground and swallows everything below it"
);
assert!(part, "a building:part must be flagged");
assert_eq!(HeightSource::from_code(src), Some(HeightSource::TaggedMetres));
// k%8 == 6 (highway) and 7 (`building=no`) must not be in the table at all.
assert!(!seen.contains_key(&7), "a highway was selected as a building");
assert!(
!seen.contains_key(&8),
"`building=no` is an explicit denial and must not be extruded"
);
// ── the coverage histogram covers every variant, zeroes included ──────────
assert_eq!(
stats.height_sources.len(),
HeightSource::ALL.len(),
"the histogram must name every variant, including the empty ones — otherwise \
`no feet in this country` reads the same as `the feet branch is gone`"
);
let hist: HashMap<&str, usize> = stats
.height_sources
.iter()
.map(|(k, n)| (k.as_str(), *n))
.collect();
assert_eq!(hist["absent"], 256);
assert_eq!(hist["tagged_metres"], 512); // rows 1 and 5
assert_eq!(hist["tagged_feet"], 256);
assert_eq!(hist["derived_from_levels"], 256);
assert_eq!(hist["rejected"], 256);
// surveyed = the two TAGGED forms: 512 metres + 256 feet of 1 536 rows.
assert!(
(stats.surveyed_share() - 768.0 / 1536.0).abs() < 1e-6,
"surveyed_share is {} on a fixture where 768 of 1536 rows carry a tagged \
height (512 metric, 256 feet)",
stats.surveyed_share()
);
// and a DERIVED height is not surveyed — the distinction the column exists for.
assert!(
(stats.any_height_share() - 1024.0 / 1536.0).abs() < 1e-6,
"any_height_share is {}; 768 tagged + 256 derived of 1 536",
stats.any_height_share()
);
// The manifest carries the same numbers, so an operator never has to open the
// parquet to learn the coverage.
let manifest = std::fs::read_to_string(dst.join(BUILDINGS_MANIFEST)).unwrap();
assert!(manifest.contains("\"absent\": 256"), "manifest: {manifest}");
assert!(manifest.contains("\"tagged_feet\": 256"));
assert!(manifest.contains(&stats.recipe));
}
/// ★ **RED-when-broken: the buildings table must PRUNE, and convert order would
/// not.**
///
/// Lane PACK's guard shape, on this table: the mean row-group `covering.bbox` area
/// read out of the WRITTEN file's own statistics — the number a reader prunes with —
/// against the same mean recomputed from the raw WKB in the order convert emitted it,
/// chunked at the same row-group size. The gate is 4x, the same one `network.rs`
/// holds `roads.parquet` to.
///
/// The control is pinned before the 4x is trusted, because "mean row-group bbox
/// area" against "mean row-group bbox area" is one line away from a value compared to
/// a copy of itself:
///
/// 1. the convert-order mean must be a large fraction of the WHOLE fixture extent —
/// an unsorted group really does span everything, so a small number here would
/// mean the control was accidentally computed over the sorted order;
/// 2. the packed file must have several row groups, or there is nothing to prune
/// between.
///
/// RED arm: feed input order. Verified 2026-08-31 by writing the batches straight
/// through `ArrowWriter` instead of `pack_batches` — packed and control converge and
/// the 4x fails.
#[test]
fn the_buildings_table_prunes_and_convert_order_would_not() {
let tmp = tempfile::tempdir().unwrap();
let src = tmp.path().join("extract");
let dst = tmp.path().join("buildings");
let n = 40_000usize;
let rg = 1_024usize;
fixture_ways(&src, n);
let opts = BuildingOptions {
row_group_rows: rg,
compression: String::from("none"),
..BuildingOptions::default()
};
let stats = extract_buildings(&src, &dst, &opts).unwrap();
assert!(
stats.row_groups >= 8,
"need several row groups to prune between, got {}",
stats.row_groups
);
// ── the packed file, through the statistics a reader prunes with ──────────
let f = std::fs::File::open(&stats.path).unwrap();
let b = ParquetRecordBatchReaderBuilder::try_new(f).unwrap();
let md = b.metadata();
let leaf = |name: &str| {
(0..md.file_metadata().schema_descr().num_columns())
.find(|i| md.file_metadata().schema_descr().column(*i).path().string() == name)
.unwrap_or_else(|| panic!("no {name} leaf — the covering column is missing"))
};
let (lx, ly) = (leaf("bbox.xmin"), leaf("bbox.ymin"));
let span = |g: usize, c: usize| match md.row_group(g).column(c).statistics() {
Some(Statistics::Float(s)) => f64::from(*s.max_opt().unwrap() - *s.min_opt().unwrap()),
other => panic!("row group {g} column {c} has no float statistics: {other:?}"),
};
let packed: f64 = (0..md.num_row_groups())
.map(|g| span(g, lx) * span(g, ly))
.sum::<f64>()
/ md.num_row_groups() as f64;
// ── the control: the SAME rows, in the order convert emitted them ─────────
let sf = std::fs::File::open(src.join("ways.parquet")).unwrap();
let sb = ParquetRecordBatchReaderBuilder::try_new(sf)
.unwrap()
.build()
.unwrap();
let mut boxes: Vec<crate::spatial::Bbox> = Vec::new();
for batch in sb {
let batch = batch.unwrap();
let g = col::<BinaryArray>(&batch, "geometry").unwrap();
let t = col::<StringArray>(&batch, "tags").unwrap();
for i in 0..batch.num_rows() {
// the SAME selection the table used, or the two arms are not over the
// same rows and the ratio means nothing
if building_of_tags(t.value(i)).is_some() {
boxes.push(wkb_bbox(g.value(i)).unwrap());
}
}
}
assert_eq!(boxes.len(), stats.rows, "the control is over different rows");
let chunks = boxes.chunks(rg);
let count = chunks.len();
let unsorted: f64 = boxes
.chunks(rg)
.map(|c| {
let x0 = c.iter().map(|b| b.xmin).fold(f64::MAX, f64::min);
let x1 = c.iter().map(|b| b.xmax).fold(f64::MIN, f64::max);
let y0 = c.iter().map(|b| b.ymin).fold(f64::MAX, f64::min);
let y1 = c.iter().map(|b| b.ymax).fold(f64::MIN, f64::max);
(x1 - x0) * (y1 - y0)
})
.sum::<f64>()
/ count as f64;
// (1) the control genuinely spans the fixture.
assert!(
unsorted > FIXTURE_AREA * 0.5,
"the control is not a control: convert order gave a mean row-group area of \
{unsorted:.2} deg² against a fixture extent of {FIXTURE_AREA:.2} deg². A \
small number here means it was computed over the wrong order and the 4x \
below proves nothing"
);
assert!(
packed * 4.0 < unsorted,
"the Hilbert-packed buildings table must shrink the mean row-group bbox area \
by more than 4x against convert order, or the covering column prunes \
nothing a reader can use: packed {packed:.4} deg² vs convert-order \
{unsorted:.2} deg² over {} groups",
md.num_row_groups()
);
}
/// ★ **RED-when-broken: the recipe moves when the table changes.**
///
/// A manifest that cannot tell a current table from a two-schema-old one serves the
/// old one under a Current verdict — `collauto-6`. The token must carry the column
/// list, the selection rule (including [`crate::shared::height::METRES_PER_LEVEL`],
/// because re-tuning it rewrites every derived height in the file and moves not one
/// column name), the row-group size and the `min_zoom` ladder.
///
/// # A defect this test FOUND and did not fix
///
/// `network::short_type` has no `DataType::UInt8` arm, so `min_zoom`, `levels` and
/// `height_source` all spell as `Other` in the token — and `short_type`'s own doc
/// says "`Other` is the one spelling that cannot distinguish two types". Retype
/// `min_zoom` from `UInt8` to `UInt16` today and not one recipe on disk moves. That
/// is `network.rs`, which this lane does not own; the assertions below are on column
/// NAMES so they neither enshrine the defect nor go red when it is fixed. Reported
/// rather than patched, because adding the arm re-cuts every network table on every
/// box (the token changes ⇒ Stale).
#[test]
fn the_recipe_names_what_would_make_the_table_stale_by_name() {
let r = buildings_recipe(8_192, true, false).token();
for term in [
"net=buildings",
"height_m:Float32",
"min_height_m:Float32",
"levels:",
"height_source:",
"is_part:Boolean",
"geometry:Binary",
"bbox:Struct",
"min_zoom:",
"rg=8192",
"mpl=3.2",
// DELTA 2: a building footprint is an area, so this table carries the
// generalisation columns and the recipe must name them — a table cut before
// they existed has to read Stale.
"geometry_z6:Binary",
"geometry_z9:Binary",
"geometry_z12:Binary",
] {
assert!(r.contains(term), "the recipe does not name `{term}`: {r}");
}
// a different row-group size is a different object
assert_ne!(r, buildings_recipe(4_096, true, false).token());
// and so is a table cut without the generalisation columns
assert_ne!(
r,
buildings_recipe(8_192, false, false).token(),
"generalise on and off must not produce the same recipe — the files differ \
by three columns"
);
// …and so is a table cut WITH the `roof:shape` column. That is the whole reason
// the switch is off by default: flipping it re-cuts every buildings table there
// is, and a token that did not move would serve a table missing a column its
// reader now projects.
assert_ne!(
r,
buildings_recipe(8_192, true, true).token(),
"roof_shape on and off must not produce the same recipe"
);
assert!(
buildings_recipe(8_192, true, true).token().contains("roof_shape:Utf8"),
"and the roof column is named in the token when it is carried"
);
// and every HeightSource code is spelled, so adding a variant re-cuts
for s in HeightSource::ALL {
assert!(
r.contains(&format!("{}={}", s.code(), s.name())),
"the recipe does not name source {s:?}: {r}"
);
}
// The token a table on disk declares must be the token the code produces.
let tmp = tempfile::tempdir().unwrap();
let src = tmp.path().join("extract");
let dst = tmp.path().join("buildings");
fixture_ways(&src, 256);
let stats = extract_buildings(&src, &dst, &BuildingOptions::default()).unwrap();
assert_eq!(
NetworkRecipe::of_table(&stats.path).unwrap().as_deref(),
Some(stats.recipe.as_str()),
"the written table declares a different recipe than the code cut it with"
);
}
/// The selection rule, on its own. `building=no` is a denial and `highway=*` is not a
/// building; both are the kind of row a loose `tags.contains("building")` would sweep
/// in, which is how `osm-other` came to hold 57 % of a z6 tile.
#[test]
fn building_selection_refuses_denials_and_lookalikes() {
assert_eq!(
building_of_tags(r#"{"building":"yes"}"#),
Some((String::from("yes"), false))
);
assert_eq!(
building_of_tags(r#"{"building":"apartments"}"#),
Some((String::from("apartments"), false))
);
assert_eq!(
building_of_tags(r#"{"building:part":"yes"}"#),
Some((String::from("yes"), true))
);
// a building WITH parts: `building` wins for `kind`, and the row is still a part
assert_eq!(
building_of_tags(r#"{"building":"church","building:part":"yes"}"#),
Some((String::from("church"), true))
);
for tags in [
r#"{"building":"no"}"#,
r#"{"building:part":"no"}"#,
r#"{"highway":"residential"}"#,
r#"{"building:levels":"4"}"#, // levels without a building is not a building
r#"{"name":"building supplies"}"#, // the substring trap
r#"{}"#,
"",
] {
assert_eq!(
building_of_tags(tags),
None,
"{tags} must NOT be selected as a building"
);
}
}
/// **The coverage measurement, over korp's live Sweden cut.**
///
/// `#[ignore]` because it needs a 902 MB table this repo does not ship, and because
/// the cut is READ-ONLY and shared with a running korp-server. Run it with:
///
/// ```text
/// cargo test -p osm-katana --release --lib -- --ignored --nocapture \
/// height_sources_over_the_live_sweden_cut
/// ```
///
/// It is the source of every Sweden figure in this file's module docs and in
/// `shared::height`'s. The lower bound below is the gate the brief asks for: a parser
/// that returned `None` for everything would be the fastest parser there is, and it
/// would fail here, on real data, before anyone shipped it.
#[test]
#[ignore = "needs korp's live Sweden cut at /home/korp/.korp/work/sweden"]
fn height_sources_over_the_live_sweden_cut() {
let cut = Path::new("/home/korp/.korp/work/sweden");
if !cut.join("ways.parquet").is_file() {
eprintln!("no Sweden cut at {cut:?} — skipping");
return;
}
let out = tempfile::tempdir().unwrap();
let t0 = std::time::Instant::now();
let s = extract_buildings(cut, out.path(), &BuildingOptions::default()).unwrap();
eprintln!(
"\nSWEDEN BUILDINGS {} rows {} row groups {:.1} MB {:.1}s",
s.rows,
s.row_groups,
s.bytes as f64 / 1e6,
t0.elapsed().as_secs_f64()
);
for (k, n) in &s.height_sources {
eprintln!(
" {k:<20} {n:>9} {:>7.4} %",
100.0 * *n as f64 / s.rows as f64
);
}
eprintln!(" by source {:?}", s.by_source);
eprintln!(" parts {}", s.parts);
eprintln!(" with min_height {}", s.with_min_height);
eprintln!(
" surveyed {:.4} % any height {:.4} %",
100.0 * s.surveyed_share(),
100.0 * s.any_height_share()
);
eprintln!(" top kinds {:?}", &s.kinds[..s.kinds.len().min(10)]);
assert!(
s.rows > 3_500_000,
"only {} buildings in the Sweden cut — the selection is broken, not the data",
s.rows
);
// THE LOWER BOUND. Sweden measured 4.32 % with any height (0.24 % surveyed);
// 3 % leaves room for the map to change and none for a parser that quit.
assert!(
s.any_height_share() > 0.03,
"only {:.4} % of {} Swedish buildings got a height. Sweden measured 4.3139 % \
on 2026-08-31; a number this low means the parser stopped parsing, not that \
the mappers stopped mapping",
100.0 * s.any_height_share(),
s.rows
);
assert!(
s.surveyed_share() > 0.001,
"only {:.4} % of buildings carry a TAGGED height (Sweden: 0.2487 %) — the \
`height` branch is not firing",
100.0 * s.surveyed_share()
);
// and the derived branch has to be doing the heavy lifting, or `building:levels`
// is being dropped.
let derived: usize = s
.height_sources
.iter()
.find(|(k, _)| k == HeightSource::DerivedFromLevels.name())
.map(|(_, n)| *n)
.unwrap();
assert!(
derived > 100_000,
"only {derived} buildings derived a height from building:levels; Sweden \
derived 156 167 on 2026-08-31"
);
}
}