osm-katana/src/census.rs
//! **`osm-katana census` — what is actually IN these tables, with books that close.**
//!
//! # The defect this exists to make impossible
//!
//! On 2026-08-31 a census of the tile pyramid reported that **57.0 % of a z6 tile's rows
//! carried no tags**. The figure reached two `.nornir` design documents, a doc comment on
//! [`crate::tiles::ScanLedger::untagged_nodes`], and two commit messages. It was wrong.
//!
//! The census had copied tiles with `cp z6/<x>/<y>.parquet` into ONE FLAT DIRECTORY.
//! Tiles sharing a `<y>` name overwrote each other, so **7 of 19 tiles survived** — and
//! the run printed `7` while the table above it still said nineteen. *A sample that
//! silently loses two thirds of its population still produces a clean-looking
//! percentage.* Several derived per-class figures (78.3 % of water, and others) had to be
//! withdrawn entirely. The corrections are `znippy-zoomies 6d5be1c` and `korp f6aac7e`.
//!
//! The right number — 101 081 991 of 103 488 583 nodes, 97.7 % — came from the tiler's
//! own [`crate::tiles::ScanLedger`], which counts every row of every file **by
//! construction** and closes to `UNACCOUNTED 0`. It cannot be sampled wrong. But a
//! `ScanLedger` only exists *inside a tiling run*: there was no way to ask a question
//! about tables that already sit on disk without hand-rolling exactly the kind of ad-hoc
//! script that produced the wrong answer. **That is the hole this module closes.**
//!
//! # The three properties that make it worth having
//!
//! 1. **The books close.** Every population reported is a [`Partition`] of one file's
//! rows, its denominator is that file's own footer `num_rows`, and
//! [`Partition::unaccounted`] is printed on every line. A partition that does not
//! balance **prints no percentage at all** and the verb exits non-zero — a ratio over
//! books that do not close is precisely the wrong-answer machine.
//! 2. **One accumulator produces both the population size and every ratio.** The bug's
//! signature was a table that said nineteen above a run that read seven; here `files`
//! and `rows` and every bucket come out of the same [`GroupCensus`], so they cannot
//! disagree. There is no copy step to lose a file in: the verb does the directory walk
//! itself and keys every file by its path RELATIVE TO THE ROOT, which is unique by
//! construction.
//! 3. **Path identity is checked against content identity.** In pyramid mode a file at
//! `z<Z>/<X>/<Y>.parquet` must carry a covering bbox that intersects
//! [`crate::tiles::tile_bounds`] of that address. A tile moved to a name it does not
//! belong to — the shape of the original mistake — goes red as `MISPLACED`, and a file
//! that is not at a pyramid address at all goes red as `NONCONFORMING`.
//!
//! # Cost
//!
//! Rows, bytes, row groups, column bytes and covering extents come from the **footer
//! only** — no data page is touched, and that is the whole speed of the tool. The tag
//! census (`--tags`), the geometry-type histogram (`--geometry`) and the per-class
//! breakdown (`--classes`) each need real column data and say so; they are opt-in for
//! that reason.
//!
//! Fan-out is [`gatling`] (ROOT LAW #0 — rayon is forbidden), one unit per *directory*,
//! and each worker folds straight into its own [`GroupCensus`]. **No list of 1.6 M paths
//! is ever built**: the pyramid under `/home/korp/.korp/work/sweden/tiles` is 1 635 971
//! files under 1 417 `x` directories, and a `Vec<PathBuf>` over the leaves would be the
//! largest allocation in the process for no reason.
use std::collections::BTreeMap;
use std::fs::File;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use std::time::Instant;
use anyhow::Context as _;
use parquet::file::metadata::ParquetMetaData;
use parquet::file::statistics::Statistics;
use serde::Serialize;
use crate::tiles::{lonlat_to_tile, tag_is_a_feature, tile_bounds};
// ── a number and its denominator, inseparable ───────────────────────────────────
/// **A count that cannot be printed without the population it is a count OF.**
///
/// The 57.0 % figure was believable precisely because its denominator was invisible: the
/// line said "57.0 % of rows" and the rows it was 57 % of were 7 tiles' worth, not 19.
/// So a percentage is not a `f64` anywhere in this module — it is only ever reachable
/// through [`Ratio::pct`], which owns both halves, and [`Ratio`]'s `Display` always
/// prints `n of d (p %)`.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
pub struct Ratio {
/// The numerator — how many rows fell in this bucket.
pub n: u64,
/// The denominator — the population the bucket was drawn from.
pub d: u64,
}
impl Ratio {
/// `n / d` as a percentage; `0.0` for an empty population, which `Display` still
/// shows as `0 of 0`, so the emptiness is readable rather than inferred.
#[must_use]
pub fn pct(&self) -> f64 {
if self.d == 0 {
0.0
} else {
self.n as f64 * 100.0 / self.d as f64
}
}
/// **`n > d` is not a ratio** — it is a bucket counted against the wrong population,
/// which is the class of error this whole module exists for. Callers that build a
/// ratio from two independently-derived numbers use this and get a `None` they have
/// to handle instead of a percentage over 100 that reads as merely surprising.
#[must_use]
pub fn checked(n: u64, d: u64) -> Option<Self> {
(n <= d).then_some(Self { n, d })
}
}
impl std::fmt::Display for Ratio {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{} of {} ({:.1} %)", self.n, self.d, self.pct())
}
}
// ── the partition: the ScanLedger discipline, for a table already on disk ────────
/// **A disjoint, exhaustive split of one population of rows** — [`crate::tiles::ScanLedger`]'s
/// discipline applied to a file that already exists.
///
/// `total` is the file's own footer `num_rows`; the buckets are every class a row can
/// leave through. [`Self::unaccounted`] is `total` minus what the buckets claim, and it
/// is **signed on purpose** for the same reason the ledger's is: a negative value means a
/// row was counted twice, which is a different bug from a row nobody counted, and a
/// `usize` would print it as 18 446 744 073 709 551 615.
///
/// A partition that does not balance yields **no percentages** — see [`Self::line`].
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize)]
pub struct Partition {
/// What is being split — `"tags"`, `"geometry"`, `"class"`.
pub name: String,
/// The denominator: rows the footer says the file holds.
pub total: u64,
/// Every class a row can fall in, and how many did. Ordered, so two runs print the
/// same table.
pub buckets: BTreeMap<String, u64>,
}
impl Partition {
/// An empty partition of `total` rows, named.
#[must_use]
pub fn new(name: impl Into<String>, total: u64) -> Self {
Self {
name: name.into(),
total,
buckets: BTreeMap::new(),
}
}
/// Add `n` rows to a bucket, creating it.
pub fn add(&mut self, bucket: impl Into<String>, n: u64) {
*self.buckets.entry(bucket.into()).or_insert(0) += n;
}
/// Rows the buckets between them claim.
#[must_use]
pub fn counted(&self) -> u64 {
self.buckets.values().sum()
}
/// **Rows of the file that no bucket claims** — zero when the books close.
#[allow(clippy::cast_possible_wrap)]
#[must_use]
pub fn unaccounted(&self) -> i64 {
self.total as i64 - self.counted() as i64
}
/// Do the books close?
#[must_use]
pub fn balances(&self) -> bool {
self.unaccounted() == 0
}
/// The ratio of one bucket **to this partition's own total** — the only door to a
/// percentage, so a bucket can never be divided by a denominator from somewhere else.
/// `None` when the books do not close: a share of a population that lost rows is the
/// exact figure this module exists to refuse to print.
#[must_use]
pub fn ratio(&self, bucket: &str) -> Option<Ratio> {
if !self.balances() {
return None;
}
Ratio::checked(self.buckets.get(bucket).copied().unwrap_or(0), self.total)
}
/// Fold another partition of the same name into this one — bucket-wise, and the
/// totals add. Merging is how a 1.6 M-file walk stays in constant memory.
pub fn merge(&mut self, other: &Self) {
if self.name.is_empty() {
self.name = other.name.clone();
}
self.total += other.total;
for (k, v) in &other.buckets {
*self.buckets.entry(k.clone()).or_insert(0) += v;
}
}
/// **The one line an operator reads** — every bucket with its own denominator, and
/// the residual last.
///
/// When the books do not close every bucket prints as a bare count and NO percentage
/// appears anywhere on the line. That is deliberate: the failure mode being guarded
/// against is a clean-looking percentage over a population that quietly lost rows,
/// and suppressing the number is the only presentation that cannot be misread.
#[must_use]
pub fn line(&self) -> String {
let mut s = format!("{}: {} rows =", self.name, self.total);
let closed = self.balances();
let mut first = true;
for (k, v) in &self.buckets {
s.push_str(if first { " " } else { " + " });
first = false;
if closed {
// `unwrap` is sound: `closed` is exactly `ratio`'s precondition and the
// bucket is one we just read out of the map.
let r = Ratio {
n: *v,
d: self.total,
};
s.push_str(&format!("{k} {r}"));
} else {
s.push_str(&format!("{k} {v}"));
}
}
if first {
s.push_str(" (no buckets)");
}
s.push_str(&format!(" · UNACCOUNTED {}", self.unaccounted()));
if !closed {
s.push_str(" <-- BOOKS DO NOT CLOSE, no ratio is quotable from this line");
}
s
}
}
// ── options ─────────────────────────────────────────────────────────────────────
/// **A deliberate wrong answer, for proving a guard can go red** (LAW 2).
///
/// A closure check that cannot be made to fail is decoration. Each arm drops or
/// mis-attributes a population inside the REAL counting loop, so the red arm travels the
/// same code the green one does rather than testing a hand-built struct.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Sabotage {
/// Silently drop every tagged row from the tag partition — the "a population
/// vanished" shape.
DropTagged,
/// Silently drop every row whose geometry is a Point from the geometry partition.
DropPoints,
/// Inflate the denominator by 10 % without adding rows to any bucket — the "the
/// table said nineteen while the run read seven" shape, from the other side.
InflateTotal,
}
impl Sabotage {
/// Parse the `OSM_KATANA_CENSUS_SABOTAGE` env value, for a red arm from the CLI.
#[must_use]
pub fn parse(s: &str) -> Option<Self> {
match s.trim() {
"drop-tagged" => Some(Self::DropTagged),
"drop-points" => Some(Self::DropPoints),
"inflate-total" => Some(Self::InflateTotal),
_ => None,
}
}
/// From the environment, or `None`.
#[must_use]
pub fn from_env() -> Option<Self> {
std::env::var("OSM_KATANA_CENSUS_SABOTAGE")
.ok()
.as_deref()
.and_then(Self::parse)
}
}
/// What the walk should look at, and how deep.
#[derive(Clone, Debug)]
pub struct CensusOptions {
/// Count tags — the three spellings of "no tags", and the fail-OPEN rule for a
/// source with no `tags` column at all. Reads the `tags` column.
pub tags: bool,
/// Geometry-type histogram from each WKB header. Reads the `geometry` column, which
/// is the fat one — this is what makes a full-pyramid census minutes rather than
/// seconds.
pub geometry: bool,
/// Per-class breakdown via [`crate::shared::class`]. Implies [`Self::tags`] (it is
/// the same column read once; the classifier is not duplicated here — LAW 5).
pub classes: bool,
/// Fold every group into one, instead of one group per table / per zoom.
pub workers: usize,
/// See [`Sabotage`] — `None` in every non-test caller.
pub sabotage: Option<Sabotage>,
}
impl Default for CensusOptions {
fn default() -> Self {
Self {
tags: false,
geometry: false,
classes: false,
workers: 0,
sabotage: None,
}
}
}
impl CensusOptions {
/// Does anything here need a data page? Footer-only is the fast path and the
/// difference is the whole speed of the tool, so the question has a name.
#[must_use]
pub fn needs_data(&self) -> bool {
self.tags || self.geometry || self.classes
}
#[must_use]
fn wants_tags(&self) -> bool {
self.tags || self.classes
}
}
// ── one file ────────────────────────────────────────────────────────────────────
/// The bytes one column costs, on disk and decoded.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize)]
pub struct ColBytes {
pub compressed: u64,
pub uncompressed: u64,
}
/// A lon/lat extent read from footer statistics — no data page.
#[derive(Clone, Copy, Debug, PartialEq, Serialize)]
pub struct Extent {
pub xmin: f64,
pub ymin: f64,
pub xmax: f64,
pub ymax: f64,
}
impl Extent {
fn area(&self) -> f64 {
(self.xmax - self.xmin).max(0.0) * (self.ymax - self.ymin).max(0.0)
}
fn union(&mut self, o: &Self) {
self.xmin = self.xmin.min(o.xmin);
self.ymin = self.ymin.min(o.ymin);
self.xmax = self.xmax.max(o.xmax);
self.ymax = self.ymax.max(o.ymax);
}
/// Do two extents share any area at all? Inclusive — a shared edge counts.
fn intersects(&self, o: &Self) -> bool {
self.xmin <= o.xmax && o.xmin <= self.xmax && self.ymin <= o.ymax && o.ymin <= self.ymax
}
}
/// **The identity of one input file** — what a quoted figure can be checked against.
///
/// A number without provenance is how this went wrong the first time: the 57.0 % was
/// quoted for months with no way to ask *which files was that over?*
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub struct InputIdentity {
/// Path RELATIVE TO THE CENSUS ROOT — unique by construction, which is the property
/// a flat `cp` destroys.
pub key: String,
pub bytes: u64,
/// Seconds since the epoch, or `None` where the filesystem would not say.
pub mtime_unix: Option<i64>,
/// The `osm-katana:tile-recipe` token from the file's own footer, where it carries
/// one. Two distinct tokens under one pyramid is a MIXED pyramid — a real finding,
/// and one no `du` can report.
pub recipe: Option<String>,
}
/// Everything one parquet file contributes, before it is folded away.
#[derive(Clone, Debug)]
pub struct FileCensus {
pub identity: InputIdentity,
/// The footer's own `num_rows`.
pub rows: u64,
/// Summed over the row-group metadata — an INDEPENDENT count of the same rows, from
/// a different part of the footer. `rows != rows_from_row_groups` is a footer that
/// contradicts itself; it is reported, never averaged.
pub rows_from_row_groups: u64,
pub row_groups: u64,
pub compressed_bytes: u64,
pub uncompressed_bytes: u64,
pub columns: BTreeMap<String, ColBytes>,
/// Union of every row group's covering box, from footer statistics.
pub extent: Option<Extent>,
/// Sum of the per-ROW-GROUP covering areas, and how many groups had one — the pair
/// lane PACK's "does packing work?" question is answered from.
pub rg_area_sum: f64,
pub rg_area_n: u64,
pub partitions: Vec<Partition>,
}
/// Read one parquet file's footer, and its data columns only if `opts` asks.
pub fn census_file(root: &Path, path: &Path, opts: &CensusOptions) -> anyhow::Result<FileCensus> {
let file = File::open(path).with_context(|| format!("open {path:?}"))?;
let fs_meta = file.metadata().with_context(|| format!("stat {path:?}"))?;
let bytes = fs_meta.len();
let mtime_unix = fs_meta.modified().ok().and_then(|t| {
t.duration_since(std::time::UNIX_EPOCH)
.ok()
.map(|d| d.as_secs() as i64)
});
let meta = parquet::file::metadata::ParquetMetaDataReader::new()
.parse_and_finish(&file)
.with_context(|| format!("parquet footer {path:?}"))?;
let rows = meta.file_metadata().num_rows().max(0) as u64;
let row_groups = meta.num_row_groups() as u64;
let recipe = meta
.file_metadata()
.key_value_metadata()
.and_then(|kvs| {
kvs.iter()
.find(|kv| kv.key == crate::tiles::TILE_RECIPE_META_KEY)
})
.and_then(|kv| kv.value.clone());
let mut rows_from_row_groups = 0u64;
let mut compressed_bytes = 0u64;
let mut uncompressed_bytes = 0u64;
let mut columns: BTreeMap<String, ColBytes> = BTreeMap::new();
let mut extent: Option<Extent> = None;
let mut rg_area_sum = 0.0f64;
let mut rg_area_n = 0u64;
for rg in meta.row_groups() {
rows_from_row_groups += rg.num_rows().max(0) as u64;
let mut bb = BboxAcc::default();
for col in rg.columns() {
let name = col.column_path().string();
let e = columns.entry(name.clone()).or_default();
e.compressed += col.compressed_size().max(0) as u64;
e.uncompressed += col.uncompressed_size().max(0) as u64;
compressed_bytes += col.compressed_size().max(0) as u64;
uncompressed_bytes += col.uncompressed_size().max(0) as u64;
bb.absorb(&name, col.statistics());
}
if let Some(e) = bb.finish() {
rg_area_sum += e.area();
rg_area_n += 1;
match &mut extent {
Some(u) => u.union(&e),
None => extent = Some(e),
}
}
}
let key = root_relative_key(root, path);
let mut partitions = Vec::new();
if opts.needs_data() {
partitions = data_partitions(path, &meta, rows, opts)?;
}
Ok(FileCensus {
identity: InputIdentity {
key,
bytes,
mtime_unix,
recipe,
},
rows,
rows_from_row_groups,
row_groups,
compressed_bytes,
uncompressed_bytes,
columns,
extent,
rg_area_sum,
rg_area_n,
partitions,
})
}
/// A file's key is its path RELATIVE TO THE ROOT — `z6/33/15.parquet`, never `15.parquet`.
/// This is the whole anti-flatten property in one function: two different tiles cannot
/// share a key, because the key IS the path the walk found them at.
fn root_relative_key(root: &Path, path: &Path) -> String {
path.strip_prefix(root)
.unwrap_or(path)
.to_string_lossy()
.replace('\\', "/")
}
/// Accumulates the four `bbox.*` leaf statistics of one row group into one [`Extent`].
#[derive(Default)]
struct BboxAcc {
xmin: Option<f64>,
ymin: Option<f64>,
xmax: Option<f64>,
ymax: Option<f64>,
}
impl BboxAcc {
fn absorb(&mut self, name: &str, stats: Option<&Statistics>) {
let Some(stats) = stats else { return };
let (lo, hi) = match stats {
Statistics::Float(v) => (
v.min_opt().map(|x| f64::from(*x)),
v.max_opt().map(|x| f64::from(*x)),
),
Statistics::Double(v) => (v.min_opt().copied(), v.max_opt().copied()),
_ => return,
};
// Leaf paths are `bbox.xmin` … under `crate::metadata::BBOX_COLUMN`.
let leaf = name.rsplit('.').next().unwrap_or(name);
let root = name.split('.').next().unwrap_or(name);
if root != crate::metadata::BBOX_COLUMN {
return;
}
match leaf {
// The MINIMUM of the row group's `xmin` column is the group's west edge;
// the MAXIMUM of its `xmax` column is the east one. Taking `min` of both
// would report a box that does not contain the data.
"xmin" => self.xmin = min_opt(self.xmin, lo),
"ymin" => self.ymin = min_opt(self.ymin, lo),
"xmax" => self.xmax = max_opt(self.xmax, hi),
"ymax" => self.ymax = max_opt(self.ymax, hi),
_ => {}
}
}
fn finish(&self) -> Option<Extent> {
Some(Extent {
xmin: self.xmin?,
ymin: self.ymin?,
xmax: self.xmax?,
ymax: self.ymax?,
})
}
}
fn min_opt(a: Option<f64>, b: Option<f64>) -> Option<f64> {
match (a, b) {
(Some(a), Some(b)) => Some(a.min(b)),
(x, None) => x,
(None, y) => y,
}
}
fn max_opt(a: Option<f64>, b: Option<f64>) -> Option<f64> {
match (a, b) {
(Some(a), Some(b)) => Some(a.max(b)),
(x, None) => x,
(None, y) => y,
}
}
// ── the data-page partitions ────────────────────────────────────────────────────
/// The bucket a row with no tags at all falls in, spelled three ways because **all three
/// occur in a real cut** — see [`crate::tiles::tag_is_a_feature`], whose reasoning this
/// reuses rather than re-deriving (LAW 5).
const TAG_NULL: &str = "untagged: NULL cell";
const TAG_EMPTY: &str = "untagged: empty string";
const TAG_EMPTY_OBJ: &str = "untagged: {}";
const TAG_TAGGED: &str = "tagged";
/// **The fail-OPEN bucket.** A source with no `tags` column at all is a pre-tags source,
/// and there a missing cell means "this file does not say", not "this row carries
/// nothing". Counting it as untagged would attribute every row of such a file to the
/// wrong side — the guard inverted. So it gets a bucket of its own and is never folded
/// into either answer. The rule is [`crate::tiles::ScanLedger::untagged_nodes`]'; this is
/// the same rule, not a second one.
const TAG_NO_COLUMN: &str = "no `tags` column — file does not say";
fn data_partitions(
path: &Path,
meta: &ParquetMetaData,
rows: u64,
opts: &CensusOptions,
) -> anyhow::Result<Vec<Partition>> {
use arrow::array::{Array, BinaryArray, StringArray};
use parquet::arrow::ProjectionMask;
use parquet::arrow::arrow_reader::{
ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
};
let meta = std::sync::Arc::new(meta.clone());
let arm = ArrowReaderMetadata::try_new(meta.clone(), ArrowReaderOptions::new())?;
let builder = ParquetRecordBatchReaderBuilder::new_with_metadata(File::open(path)?, arm);
let schema = builder.schema().clone();
let has_tags = schema.index_of("tags").is_ok();
let has_geom = schema.index_of("geometry").is_ok();
let want_tags = opts.wants_tags();
let want_geom = opts.geometry;
let mut wanted: Vec<&str> = Vec::new();
if want_tags && has_tags {
wanted.push("tags");
}
if want_geom && has_geom {
wanted.push("geometry");
}
let mut tags_p = Partition::new("tags", rows);
let mut geom_p = Partition::new("geometry", rows);
let mut class_p = Partition::new("class", rows);
// **The JOINT question, which neither partition alone can answer.** `korp f6aac7e`
// asserts of the 2026-08-31 recut that the untagged rows left at z6 are "31 981 of
// 375 918 (8.51 %), and NONE of them is a point" — two marginals cannot check that
// sentence, only the cross-tabulation can, and it costs no extra I/O because both
// columns are already in the same batch.
let mut joint_p = Partition::new("geometry × tags", rows);
let want_joint = want_tags && want_geom && has_tags && has_geom;
// A file with no `tags` column: every row goes to the fail-OPEN bucket, and the
// books close on it. No column read at all — the answer is in the schema.
if want_tags && !has_tags {
tags_p.add(TAG_NO_COLUMN, rows);
class_p.add(TAG_NO_COLUMN, rows);
}
if want_geom && !has_geom {
geom_p.add("no `geometry` column — file does not say", rows);
}
if !wanted.is_empty() {
let mask = ProjectionMask::columns(builder.parquet_schema(), wanted.iter().copied());
let reader = builder.with_projection(mask).build()?;
for batch in reader {
let batch = batch?;
let tags_arr = if want_tags && has_tags {
Some(
batch
.column_by_name("tags")
.context("projected `tags` column missing from batch")?
.as_any()
.downcast_ref::<StringArray>()
.context("`tags` is not Utf8")?,
)
} else {
None
};
let geom_arr = if want_geom && has_geom {
Some(
batch
.column_by_name("geometry")
.context("projected `geometry` column missing from batch")?
.as_any()
.downcast_ref::<BinaryArray>()
.context("`geometry` is not Binary")?,
)
} else {
None
};
for i in 0..batch.num_rows() {
let tag_bucket = tags_arr.map(|arr| {
let cell = if arr.is_null(i) { None } else { Some(arr.value(i)) };
let b = match cell {
None => TAG_NULL,
Some(t) if t.trim().is_empty() => TAG_EMPTY,
Some(t) if t.trim() == "{}" => TAG_EMPTY_OBJ,
Some(_) => TAG_TAGGED,
};
(b, cell)
});
let geom_bucket = geom_arr.map(|arr| {
if arr.is_null(i) {
"NULL geometry"
} else {
wkb_type_name(arr.value(i))
}
});
if let Some((bucket, cell)) = tag_bucket {
// ⚠ RED ARM (LAW 2): drop a whole population inside the real loop.
if !(opts.sabotage == Some(Sabotage::DropTagged) && bucket == TAG_TAGGED) {
tags_p.add(bucket, 1);
if opts.classes {
let name = match cell {
Some(t) if tag_is_a_feature(Some(t)) => {
match crate::shared::class::classify_tier(t) {
Some(tier) => format!("{tier:?}"),
None => String::from("unclassified"),
}
}
_ => String::from("untagged"),
};
class_p.add(name, 1);
}
}
}
if let Some(bucket) = geom_bucket {
// ⚠ RED ARM (LAW 2).
if !(opts.sabotage == Some(Sabotage::DropPoints) && bucket == "Point") {
geom_p.add(bucket, 1);
}
}
if want_joint {
let (tb, _) = tag_bucket.expect("want_joint implies a tag bucket");
let gb = geom_bucket.expect("want_joint implies a geometry bucket");
let side = if tb == TAG_TAGGED { "tagged" } else { "untagged" };
joint_p.add(format!("{gb} · {side}"), 1);
}
}
}
}
// ⚠ RED ARM (LAW 2): the "the table said nineteen while the run read seven" shape —
// a denominator that no longer describes the rows that were counted.
if opts.sabotage == Some(Sabotage::InflateTotal) {
let bump = rows / 10 + 1;
tags_p.total += bump;
geom_p.total += bump;
class_p.total += bump;
joint_p.total += bump;
}
let mut out = Vec::new();
if want_tags {
out.push(tags_p);
}
if want_geom {
out.push(geom_p);
}
if opts.classes {
out.push(class_p);
}
if want_joint {
out.push(joint_p);
}
Ok(out)
}
/// **The geometry type from a WKB header** — one byte of endianness and a `u32` code, so
/// this is the question behind *"87.9 % of area placements shipped as WKB Point"* and
/// *"all 57 160 z6 buildings were Point, so 3D had nothing to extrude"*.
///
/// The high bits carry the EWKB Z/M/SRID flags and the ISO 1000/2000/3000 offsets; both
/// are stripped so a `PointZ` still reports as a point dimensionally-annotated rather
/// than as an unknown code nobody can act on.
#[must_use]
pub fn wkb_type_name(wkb: &[u8]) -> &'static str {
if wkb.len() < 5 {
return "malformed (< 5 bytes)";
}
let raw = match wkb[0] {
1 => u32::from_le_bytes([wkb[1], wkb[2], wkb[3], wkb[4]]),
0 => u32::from_be_bytes([wkb[1], wkb[2], wkb[3], wkb[4]]),
_ => return "malformed (bad endian byte)",
};
// EWKB flags live in the top nibble; ISO dimensionality is a 1000/2000/3000 offset.
let base = (raw & 0x0000_FFFF) % 1000;
match base {
1 => "Point",
2 => "LineString",
3 => "Polygon",
4 => "MultiPoint",
5 => "MultiLineString",
6 => "MultiPolygon",
7 => "GeometryCollection",
_ => "unknown WKB type",
}
}
// ── groups: one per table, or one per pyramid zoom ──────────────────────────────
/// **Everything one group of files adds up to** — and the ONE accumulator both the
/// population size (`files`) and every ratio come out of, so a table cannot say nineteen
/// above a run that read seven.
#[derive(Clone, Debug, Default, Serialize)]
pub struct GroupCensus {
/// `nodes.parquet`, or `z6`.
pub label: String,
pub files: u64,
/// On-disk bytes, from the directory entries — NOT from the parquet footers, so
/// footer arithmetic and `du` are two independent numbers here.
pub bytes: u64,
pub rows: u64,
/// The independent row count summed from row-group metadata. See
/// [`FileCensus::rows_from_row_groups`].
pub rows_from_row_groups: u64,
pub row_groups: u64,
pub compressed_bytes: u64,
pub uncompressed_bytes: u64,
pub columns: BTreeMap<String, ColBytes>,
pub extent: Option<Extent>,
pub rg_area_sum: f64,
pub rg_area_n: u64,
/// Every distinct `osm-katana:tile-recipe` token seen, and how many files carried it.
/// More than one entry means a MIXED pyramid.
pub recipes: BTreeMap<String, u64>,
pub mtime_min: Option<i64>,
pub mtime_max: Option<i64>,
pub partitions: Vec<Partition>,
/// **Files whose path address and whose content disagree** — a tile carrying a
/// covering box that does not intersect [`crate::tiles::tile_bounds`] of the address
/// its path names. Pyramid mode only.
pub misplaced: u64,
/// Up to [`SHOW_EXAMPLES`] misplaced keys, named — "the count changed" is not
/// something an operator can act on.
pub misplaced_examples: Vec<String>,
/// Files at a pyramid address whose content could be checked against it.
pub addressed: u64,
/// A few input identities, kept so a quoted figure can be traced. A pyramid has
/// 1.6 M files and they do not all belong in a JSON document; a table has four and
/// they all do — see [`Census::inputs_are_complete`].
pub sample_inputs: Vec<InputIdentity>,
}
/// How many named examples a group keeps of any one class.
pub const SHOW_EXAMPLES: usize = 8;
impl GroupCensus {
fn new(label: impl Into<String>) -> Self {
Self {
label: label.into(),
..Default::default()
}
}
/// Fold one file in, then let it drop. Constant memory over 1.6 M files.
pub fn absorb(&mut self, fc: &FileCensus, keep_identity: bool) {
self.files += 1;
self.bytes += fc.identity.bytes;
self.rows += fc.rows;
self.rows_from_row_groups += fc.rows_from_row_groups;
self.row_groups += fc.row_groups;
self.compressed_bytes += fc.compressed_bytes;
self.uncompressed_bytes += fc.uncompressed_bytes;
for (k, v) in &fc.columns {
let e = self.columns.entry(k.clone()).or_default();
e.compressed += v.compressed;
e.uncompressed += v.uncompressed;
}
if let Some(e) = &fc.extent {
match &mut self.extent {
Some(u) => u.union(e),
None => self.extent = Some(*e),
}
}
self.rg_area_sum += fc.rg_area_sum;
self.rg_area_n += fc.rg_area_n;
if let Some(r) = &fc.identity.recipe {
*self.recipes.entry(r.clone()).or_insert(0) += 1;
}
if let Some(t) = fc.identity.mtime_unix {
self.mtime_min = Some(self.mtime_min.map_or(t, |m| m.min(t)));
self.mtime_max = Some(self.mtime_max.map_or(t, |m| m.max(t)));
}
for p in &fc.partitions {
match self.partitions.iter_mut().find(|q| q.name == p.name) {
Some(q) => q.merge(p),
None => self.partitions.push(p.clone()),
}
}
if keep_identity || self.sample_inputs.len() < SHOW_EXAMPLES {
self.sample_inputs.push(fc.identity.clone());
}
}
/// Fold a whole group in — how per-worker accumulators become one answer.
pub fn merge(&mut self, other: &Self) {
if self.label.is_empty() {
self.label = other.label.clone();
}
self.files += other.files;
self.bytes += other.bytes;
self.rows += other.rows;
self.rows_from_row_groups += other.rows_from_row_groups;
self.row_groups += other.row_groups;
self.compressed_bytes += other.compressed_bytes;
self.uncompressed_bytes += other.uncompressed_bytes;
for (k, v) in &other.columns {
let e = self.columns.entry(k.clone()).or_default();
e.compressed += v.compressed;
e.uncompressed += v.uncompressed;
}
if let Some(e) = &other.extent {
match &mut self.extent {
Some(u) => u.union(e),
None => self.extent = Some(*e),
}
}
self.rg_area_sum += other.rg_area_sum;
self.rg_area_n += other.rg_area_n;
for (k, v) in &other.recipes {
*self.recipes.entry(k.clone()).or_insert(0) += v;
}
if let Some(t) = other.mtime_min {
self.mtime_min = Some(self.mtime_min.map_or(t, |m| m.min(t)));
}
if let Some(t) = other.mtime_max {
self.mtime_max = Some(self.mtime_max.map_or(t, |m| m.max(t)));
}
for p in &other.partitions {
match self.partitions.iter_mut().find(|q| q.name == p.name) {
Some(q) => q.merge(p),
None => self.partitions.push(p.clone()),
}
}
self.misplaced += other.misplaced;
for k in &other.misplaced_examples {
if self.misplaced_examples.len() < SHOW_EXAMPLES {
self.misplaced_examples.push(k.clone());
}
}
self.addressed += other.addressed;
for i in &other.sample_inputs {
if self.sample_inputs.len() < SHOW_EXAMPLES {
self.sample_inputs.push(i.clone());
}
}
}
/// Mean bytes per file — `0` for an empty group rather than a division by zero.
#[must_use]
pub fn mean_file_bytes(&self) -> u64 {
if self.files == 0 {
0
} else {
self.bytes / self.files
}
}
/// Mean rows per row group.
#[must_use]
pub fn mean_row_group_rows(&self) -> u64 {
if self.row_groups == 0 {
0
} else {
self.rows / self.row_groups
}
}
/// Mean covering-box area, in square degrees, over every row group that had one.
#[must_use]
pub fn mean_row_group_area(&self) -> f64 {
if self.rg_area_n == 0 {
0.0
} else {
self.rg_area_sum / self.rg_area_n as f64
}
}
/// **Every reason this group may not be reported on**, named. Empty means clean.
#[must_use]
pub fn faults(&self) -> Vec<String> {
let mut v = Vec::new();
if self.rows != self.rows_from_row_groups {
v.push(format!(
"{}: the footer contradicts itself — file num_rows {} but row groups sum to {}",
self.label, self.rows, self.rows_from_row_groups
));
}
for p in &self.partitions {
if !p.balances() {
v.push(format!(
"{}: partition `{}` does not close — UNACCOUNTED {}",
self.label,
p.name,
p.unaccounted()
));
}
}
if self.misplaced > 0 {
v.push(format!(
"{}: MISPLACED {} of {} addressed file(s) — the path names a tile the content \
does not fall in (e.g. {})",
self.label,
self.misplaced,
self.addressed,
self.misplaced_examples.join(", ")
));
}
v
}
/// The human block for one group. Every ratio carries its denominator; the file count
/// is on the first line, from the same accumulator the ratios are.
#[must_use]
pub fn report(&self) -> String {
let mut s = format!(
"{:<16} files {:>9} · bytes {:>15} ({:.2} GB, mean {} B/file) · rows {:>12} \
· row groups {:>9} (mean {} rows)\n",
self.label,
self.files,
self.bytes,
self.bytes as f64 / 1e9,
self.mean_file_bytes(),
self.rows,
self.row_groups,
self.mean_row_group_rows(),
);
s.push_str(&format!(
"{:<16} rows cross-check: footer num_rows {} vs row-group sum {} — {}\n",
"",
self.rows,
self.rows_from_row_groups,
if self.rows == self.rows_from_row_groups {
"AGREE"
} else {
"DISAGREE <-- the footer contradicts itself"
}
));
s.push_str(&format!(
"{:<16} column bytes: {} compressed / {} uncompressed\n",
"", self.compressed_bytes, self.uncompressed_bytes
));
for (name, cb) in &self.columns {
let share = Ratio::checked(cb.compressed, self.compressed_bytes);
s.push_str(&format!(
"{:<16} {:<24} {:>14} B compressed{} · {:>14} B uncompressed\n",
"",
name,
cb.compressed,
match share {
Some(r) => format!(" ({r} of compressed bytes)"),
None => String::new(),
},
cb.uncompressed,
));
}
if let Some(e) = &self.extent {
s.push_str(&format!(
"{:<16} covering: extent [{:.5},{:.5} .. {:.5},{:.5}] · mean row-group area \
{:.6} deg² over {} group(s)\n",
"",
e.xmin,
e.ymin,
e.xmax,
e.ymax,
self.mean_row_group_area(),
self.rg_area_n,
));
}
if self.addressed > 0 {
s.push_str(&format!(
"{:<16} address check: {} of {} file(s) sit in the tile their path names; \
MISPLACED {}\n",
"",
self.addressed - self.misplaced,
self.addressed,
self.misplaced,
));
}
if !self.recipes.is_empty() {
for (tok, n) in &self.recipes {
s.push_str(&format!(
"{:<16} recipe {} file(s): {}\n",
"",
Ratio {
n: *n,
d: self.files
},
tok
));
}
if self.recipes.len() > 1 {
s.push_str(&format!(
"{:<16} <-- MIXED PYRAMID: {} distinct recipe tokens under one root\n",
"",
self.recipes.len()
));
}
}
for p in &self.partitions {
s.push_str(&format!("{:<16} {}\n", "", p.line()));
}
s
}
}
// ── the walk ────────────────────────────────────────────────────────────────────
/// What was censused, and how the walk read the directory.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
pub enum Mode {
/// A GeoParquet extract: one group per `*.parquet` in the root.
Tables,
/// A tile pyramid: one group per `z<N>`, and every file must sit at
/// `z<Z>/<X>/<Y>.parquet`.
Pyramid,
}
/// **The whole answer** — groups, totals, provenance and a verdict.
#[derive(Clone, Debug, Serialize)]
pub struct Census {
pub tool: &'static str,
pub version: &'static str,
pub generated_unix: i64,
pub root: String,
pub mode: Mode,
pub wall_seconds: f64,
pub read_tags: bool,
pub read_geometry: bool,
pub read_classes: bool,
pub groups: Vec<GroupCensus>,
pub total: GroupCensus,
/// **Files the walk found that are NOT at a legal address for this mode.**
///
/// In pyramid mode this is how a flattened directory announces itself: a `cp
/// z6/<x>/<y>.parquet flat/` leaves files called `15.parquet` at depth 1, which is
/// not `z<Z>/<X>/<Y>.parquet`, and the verb refuses rather than reporting on
/// whichever fraction of the population survived the copy.
pub nonconforming: u64,
pub nonconforming_examples: Vec<String>,
/// True when [`GroupCensus::sample_inputs`] holds EVERY input, not a sample — so a
/// reader can tell "here is the provenance" from "here is some of it".
pub inputs_are_complete: bool,
}
impl Census {
/// **Every reason this census may not be quoted**, named. Empty means quotable.
#[must_use]
pub fn faults(&self) -> Vec<String> {
let mut v = Vec::new();
// NON-VACUITY. A census that read nothing must say so loudly; a clean empty
// table is indistinguishable from a clean full one at a glance, and that is the
// failure this module exists for.
if self.total.files == 0 {
v.push(format!(
"read ZERO parquet files under {} — there is nothing to report a census of",
self.root
));
}
if self.nonconforming > 0 {
v.push(format!(
"NONCONFORMING {} file(s) are not at a `z<Z>/<X>/<Y>.parquet` address (e.g. {}) — \
a flattened copy of a pyramid loses tiles that share a <y> name, so this is \
refused rather than sampled",
self.nonconforming,
self.nonconforming_examples.join(", ")
));
}
for g in &self.groups {
v.extend(g.faults());
}
v.extend(self.total.faults());
v
}
/// Quotable?
#[must_use]
pub fn ok(&self) -> bool {
self.faults().is_empty()
}
/// The human report — groups, then the total, then the verdict.
#[must_use]
pub fn report(&self) -> String {
let mut s = format!(
"census of {} ({:?} mode) — {:.2} s wall, {}\n",
self.root,
self.mode,
self.wall_seconds,
if self.read_tags || self.read_geometry {
"footer + data columns"
} else {
"footer only (no data page was read)"
},
);
for g in &self.groups {
s.push_str(&g.report());
}
s.push_str("─── total ───\n");
s.push_str(&self.total.report());
let faults = self.faults();
if faults.is_empty() {
s.push_str("VERDICT: books close, every population accounted for.\n");
} else {
s.push_str("VERDICT: REFUSED — this census is not quotable:\n");
for f in &faults {
s.push_str(&format!(" <-- {f}\n"));
}
}
s
}
/// The machine-readable form, so a future agent quotes a file instead of re-deriving
/// a figure with `cp`.
pub fn to_json(&self) -> anyhow::Result<String> {
Ok(serde_json::to_string_pretty(self)?)
}
}
/// **The one entry point.** Walks `root`, censuses every parquet under it, and returns an
/// answer whose books close — or one whose [`Census::faults`] say why they do not.
///
/// `mode` decides the grouping AND the legal shape of a path; see [`Mode`].
pub fn census(root: &Path, mode: Mode, opts: &CensusOptions) -> anyhow::Result<Census> {
let t0 = Instant::now();
anyhow::ensure!(root.is_dir(), "census: {root:?} is not a directory");
let (groups, nonconforming, nonconforming_examples, complete) = match mode {
Mode::Tables => census_tables(root, opts)?,
Mode::Pyramid => census_pyramid(root, opts)?,
};
let mut total = GroupCensus::new("ALL");
for g in &groups {
total.merge(g);
}
// The total's sample is not the concatenation of the groups' samples — it is the
// first few, and saying so is cheaper than a reader assuming otherwise.
total.sample_inputs.truncate(SHOW_EXAMPLES);
Ok(Census {
tool: "osm-katana census",
version: env!("CARGO_PKG_VERSION"),
generated_unix: std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs() as i64)
.unwrap_or(0),
root: root.display().to_string(),
mode,
wall_seconds: t0.elapsed().as_secs_f64(),
read_tags: opts.wants_tags(),
read_geometry: opts.geometry,
read_classes: opts.classes,
groups,
total,
nonconforming,
nonconforming_examples,
inputs_are_complete: complete,
})
}
type WalkResult = (Vec<GroupCensus>, u64, Vec<String>, bool);
/// Tables mode: every `*.parquet` directly under `root` is its own group, and EVERY input
/// identity is kept — four files' provenance belongs in the JSON in full.
fn census_tables(root: &Path, opts: &CensusOptions) -> anyhow::Result<WalkResult> {
let mut files: Vec<PathBuf> = Vec::new();
for e in std::fs::read_dir(root).with_context(|| format!("read {root:?}"))? {
let p = e?.path();
if p.is_file() && p.extension().is_some_and(|x| x == "parquet") {
files.push(p);
}
}
files.sort();
// One unit per FILE here — a table dir holds a handful of very large files, and the
// per-file work (a projected column scan of up to 103 M rows) is what needs
// balancing. LPT by on-disk size so the 1 GB table is not started last.
let sizes: Vec<u64> = files
.iter()
.map(|p| std::fs::metadata(p).map(|m| m.len()).unwrap_or(0))
.collect();
let results = gatling::gatling_forkjoin::gatling_for_each_balanced(
files.len(),
opts.workers,
1,
|i| sizes[i],
|i| census_file(root, &files[i], opts),
);
let mut groups = Vec::with_capacity(files.len());
for (i, r) in results.into_iter().enumerate() {
let fc = r?;
let mut g = GroupCensus::new(root_relative_key(root, &files[i]));
g.absorb(&fc, true);
groups.push(g);
}
Ok((groups, 0, Vec::new(), true))
}
/// Pyramid mode: one group per `z<N>`, and every parquet must sit at
/// `z<Z>/<X>/<Y>.parquet`.
///
/// # Why the unit is an `x` DIRECTORY and not a file
///
/// The live Sweden pyramid is **1 635 971 files under 1 417 `x` directories**. A
/// `Vec<PathBuf>` over the leaves would be ~100 MB and the largest allocation in the
/// process, for a walk whose whole job is to not hold anything. So the unit of work is
/// one `x` directory: a worker `read_dir`s it, opens each `y` file, folds the result into
/// its OWN accumulator and drops both the `FileCensus` and the path. Memory is
/// `workers × groups`, independent of the pyramid's size.
///
/// Units are claimed dynamically from gatling's shared cursor, which is what keeps a z15
/// `x` directory of ~1 300 tiles from stalling a worker that drew a z6 one of 4 — 1 417
/// units over 32 cores is ~44 units each, so the longest single unit is a small fraction
/// of a worker's total.
fn census_pyramid(root: &Path, opts: &CensusOptions) -> anyhow::Result<WalkResult> {
// Level 1: `z<N>` directories. Anything else at level 1 is nonconforming.
let mut zdirs: Vec<(u8, PathBuf)> = Vec::new();
let nonconf = Mutex::new((0u64, Vec::<String>::new()));
{
let mut nc = nonconf.lock().unwrap();
for e in std::fs::read_dir(root).with_context(|| format!("read {root:?}"))? {
let p = e?.path();
if p.is_dir() {
match p
.file_name()
.and_then(|n| n.to_str())
.and_then(|n| n.strip_prefix('z'))
.and_then(|n| n.parse::<u8>().ok())
{
Some(z) => zdirs.push((z, p)),
None => note_nonconforming(&mut nc, root_relative_key(root, &p)),
}
} else if p.extension().is_some_and(|x| x == "parquet") {
// A parquet at the ROOT of a pyramid is exactly the flattened-copy
// shape: `cp z6/33/15.parquet z6/34/15.parquet flat/` leaves one
// `15.parquet` where two tiles went in.
note_nonconforming(&mut nc, root_relative_key(root, &p));
}
}
}
zdirs.sort();
// Level 2: `x` directories, one gatling unit each.
let mut units: Vec<(usize, PathBuf)> = Vec::new();
for (gi, (z, zdir)) in zdirs.iter().enumerate() {
let mut nc = nonconf.lock().unwrap();
for e in std::fs::read_dir(zdir).with_context(|| format!("read {zdir:?}"))? {
let p = e?.path();
if p.is_dir() && p.file_name().and_then(|n| n.to_str()).is_some_and(is_num) {
units.push((gi, p));
} else {
let _ = z;
note_nonconforming(&mut nc, root_relative_key(root, &p));
}
}
}
units.sort();
let per_unit = gatling::gatling_forkjoin::gatling_for_each(units.len(), opts.workers, |u| {
let (gi, xdir) = &units[u];
let z = zdirs[*gi].0;
let x: u32 = xdir
.file_name()
.and_then(|n| n.to_str())
.and_then(|n| n.parse().ok())
.unwrap_or(0);
census_x_dir(root, xdir, z, x, opts).map(|(g, nc)| (*gi, g, nc))
});
let mut groups: Vec<GroupCensus> = zdirs
.iter()
.map(|(z, _)| GroupCensus::new(format!("z{z}")))
.collect();
{
let mut nc = nonconf.lock().unwrap();
for r in per_unit {
let (gi, g, unit_nc) = r?;
groups[gi].merge(&g);
for k in unit_nc {
note_nonconforming(&mut nc, k);
}
}
}
let (n, ex) = nonconf.into_inner().unwrap();
Ok((groups, n, ex, false))
}
fn note_nonconforming(nc: &mut (u64, Vec<String>), key: String) {
nc.0 += 1;
if nc.1.len() < SHOW_EXAMPLES {
nc.1.push(key);
}
}
fn is_num(s: &str) -> bool {
!s.is_empty() && s.bytes().all(|b| b.is_ascii_digit())
}
/// One `x` directory: every `<y>.parquet` in it, folded into one accumulator.
fn census_x_dir(
root: &Path,
xdir: &Path,
z: u8,
x: u32,
opts: &CensusOptions,
) -> anyhow::Result<(GroupCensus, Vec<String>)> {
let mut g = GroupCensus::new(format!("z{z}"));
let mut nonconf = Vec::new();
for e in std::fs::read_dir(xdir).with_context(|| format!("read {xdir:?}"))? {
let p = e?.path();
let stem = p.file_stem().and_then(|n| n.to_str()).unwrap_or("");
if !p.is_file() || p.extension().is_none_or(|x| x != "parquet") || !is_num(stem) {
nonconf.push(root_relative_key(root, &p));
continue;
}
let y: u32 = stem.parse().unwrap_or(0);
let fc = census_file(root, &p, opts)?;
// ── PATH IDENTITY vs CONTENT IDENTITY ──
// Two independent sources for the same fact: the address the path NAMES, and the
// covering box the file's own footer statistics REPORT. A tile that was moved to
// a name it does not belong to — the shape of the 2026-08-31 mistake — has a
// content box that does not intersect its named tile at all.
//
// The test is INTERSECTION and not containment on purpose: a feature is placed
// in every tile its bbox overlaps and the stored geometry is the whole feature,
// so a legitimate tile's extent routinely spills past its own bounds. Refusing
// on containment would fail a healthy pyramid, which is the guard shape this
// codebase keeps having to remove.
if let Some(e) = &fc.extent {
g.addressed += 1;
let b = tile_bounds(z, x, y);
let tile = Extent {
xmin: b.min_lon,
ymin: b.min_lat,
xmax: b.max_lon,
ymax: b.max_lat,
};
if !e.intersects(&tile) {
g.misplaced += 1;
if g.misplaced_examples.len() < SHOW_EXAMPLES {
g.misplaced_examples.push(format!(
"{} names z{}/{}/{} [{:.4},{:.4}..{:.4},{:.4}] but holds \
[{:.4},{:.4}..{:.4},{:.4}]",
fc.identity.key,
z,
x,
y,
tile.xmin,
tile.ymin,
tile.xmax,
tile.ymax,
e.xmin,
e.ymin,
e.xmax,
e.ymax
));
}
}
}
g.absorb(&fc, false);
}
Ok((g, nonconf))
}
/// The address a `(lon, lat)` falls in — re-exported reasoning, not a second copy of the
/// projection (LAW 5). Used by the tests that build a misplacement on purpose.
#[must_use]
pub fn address_of(lon: f64, lat: f64, z: u8) -> (u32, u32) {
lonlat_to_tile(lon, lat, z)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_partition_that_loses_a_population_cannot_be_quoted() {
// GREEN: the books close, and a ratio is available.
let mut p = Partition::new("tags", 100);
p.add(TAG_TAGGED, 30);
p.add(TAG_NULL, 70);
assert!(p.balances(), "control arm must close: {}", p.line());
assert_eq!(p.unaccounted(), 0);
assert_eq!(p.ratio(TAG_NULL), Some(Ratio { n: 70, d: 100 }));
assert!(p.line().contains("70 of 100 (70.0 %)"));
// RED: drop a population. The residual must go non-zero, the ratio must vanish,
// and the printed line must carry NO percentage at all.
let mut q = Partition::new("tags", 100);
q.add(TAG_NULL, 70);
assert!(!q.balances(), "guard is blind: {}", q.line());
assert_eq!(q.unaccounted(), 30);
assert_eq!(q.ratio(TAG_NULL), None);
assert!(!q.line().contains('%'), "no ratio may survive: {}", q.line());
assert!(q.line().contains("BOOKS DO NOT CLOSE"));
}
#[test]
fn a_double_count_is_a_different_bug_from_a_lost_row() {
let mut p = Partition::new("tags", 100);
p.add(TAG_TAGGED, 60);
p.add(TAG_NULL, 70);
// Signed, so the operator can tell the two apart — a usize would print
// 18 446 744 073 709 551 586 here.
assert_eq!(p.unaccounted(), -30);
assert!(!p.balances());
}
#[test]
fn a_wrong_denominator_shows_up_as_an_unaccounted() {
// The 2026-08-31 shape from the other side: the buckets are RIGHT and the
// denominator describes a bigger population than was counted. It must not print
// a percentage, and the residual must name the size of the lie.
let mut p = Partition::new("tags", 188_112);
p.add(TAG_NULL, 107_299);
p.add(TAG_TAGGED, 80_813);
assert!(p.balances());
let honest = p.line();
assert!(honest.contains("107299 of 188112 (57.0 %)"));
let mut wrong = p.clone();
wrong.total = 512_000; // the population the run BELIEVED it read
assert!(!wrong.balances(), "a wrong denominator must not pass");
assert_eq!(wrong.unaccounted(), 512_000 - 188_112);
assert!(!wrong.line().contains('%'));
// And the denominator is visible in the line either way — invisibility is what
// made 57.0 % believable.
assert!(wrong.line().contains("512000"));
}
#[test]
fn ratio_refuses_a_numerator_bigger_than_its_population() {
assert_eq!(Ratio::checked(7, 19), Some(Ratio { n: 7, d: 19 }));
assert_eq!(Ratio::checked(19, 7), None);
assert_eq!(Ratio { n: 7, d: 19 }.to_string(), "7 of 19 (36.8 %)");
// The identity-value trap: 0/0 must not read as 100 %.
assert_eq!(Ratio { n: 0, d: 0 }.to_string(), "0 of 0 (0.0 %)");
}
#[test]
fn wkb_headers_are_read_as_types_not_guessed() {
let le = |t: u32| {
let mut v = vec![1u8];
v.extend_from_slice(&t.to_le_bytes());
v
};
assert_eq!(wkb_type_name(&le(1)), "Point");
assert_eq!(wkb_type_name(&le(2)), "LineString");
assert_eq!(wkb_type_name(&le(3)), "Polygon");
assert_eq!(wkb_type_name(&le(6)), "MultiPolygon");
// ISO dimensionality: PointZ is 1001, and reporting it as "unknown" would have
// hidden the "all 57 160 z6 buildings were Point" finding behind a code.
assert_eq!(wkb_type_name(&le(1001)), "Point");
// Big-endian WKB.
let mut be = vec![0u8];
be.extend_from_slice(&3u32.to_be_bytes());
assert_eq!(wkb_type_name(&be), "Polygon");
// Red arms.
assert_eq!(wkb_type_name(&[1, 2, 3]), "malformed (< 5 bytes)");
assert_eq!(wkb_type_name(&[9, 1, 0, 0, 0]), "malformed (bad endian byte)");
}
#[test]
fn an_extent_intersection_is_not_a_containment() {
let tile = Extent {
xmin: 0.0,
ymin: 0.0,
xmax: 10.0,
ymax: 10.0,
};
// A way that straddles the tile edge — legitimate, must NOT be misplaced.
let straddle = Extent {
xmin: -5.0,
ymin: 5.0,
xmax: 5.0,
ymax: 15.0,
};
assert!(straddle.intersects(&tile));
// A tile from the other side of the world — misplaced.
let elsewhere = Extent {
xmin: -180.0,
ymin: -80.0,
xmax: -170.0,
ymax: -70.0,
};
assert!(!elsewhere.intersects(&tile));
}
#[test]
fn sabotage_parses_only_the_arms_that_exist() {
assert_eq!(Sabotage::parse("drop-tagged"), Some(Sabotage::DropTagged));
assert_eq!(Sabotage::parse("inflate-total"), Some(Sabotage::InflateTotal));
assert_eq!(Sabotage::parse("nonsense"), None);
}
}