crates/nornir-parquet-delta/src/lib.rs
//! **Mirror a remote parquet file into a sparse local one, a row-group column chunk at
//! a time** — plan the byte ranges, remember which ones landed, and read the
//! partially-filled file with the ORDINARY parquet reader.
//!
//! # The problem this is the general form of
//!
//! A server holds one big parquet table. A client asks it questions that each touch a
//! few row groups of a few columns, and the server answers with cooked rows. The client
//! therefore re-asks forever, and never converges on anything it could answer itself.
//!
//! The alternative is that the client keeps **the same file the server has**, and fills
//! it in. A parquet file is already shaped for exactly that: its footer says where every
//! (row group × column) chunk lives, the chunks are independent byte ranges, and a
//! reader told `with_row_groups(..)` + a projection mask touches only the ranges it was
//! promised. So a client that holds the footer, the four magic bytes, and *some* of the
//! chunks holds a file the stock reader can answer *those* queries out of — and every
//! answered query leaves the client permanently closer to holding the whole table.
//!
//! That is the whole crate:
//!
//! 1. [`plan`] — footer + [`Need`] + what is already held ⇒ the byte [`Range`]s to fetch.
//! 2. [`Mirror`] — the sparse file, the presence bitmap over (row group × column), and an
//! honest [`Mirror::holds`].
//! 3. [`Mirror::open`] against a **source version token** — a changed token is
//! [`Error::StaleSource`], never a silently-wrong read.
//! 4. **The transport is not this crate's job.** It emits ranges and consumes bytes. The
//! caller carries them over gRPC, an HTTP range GET, S3, a USB stick. That separation
//! is what makes it reusable, and it is why there is no `async` here and no client.
//!
//! # The one property everything rests on
//!
//! **A hole is an ERROR, never a zero.** A sparse file reads unwritten bytes as `0x00`,
//! and `0x00` is a perfectly plausible parquet page — so a reader pointed at a hole
//! would return a *wrong answer* rather than fail. [`MirrorReader`] therefore checks
//! every range the parquet reader asks for against the presence set and returns
//! [`Error::Hole`] for anything not held. That is the difference between this being a
//! cache and being a corruption engine, and `reader.rs`'s
//! `a_read_that_runs_one_byte_past_the_held_span_is_a_hole` proves it red — its own
//! comment names the injection: written as `end <= self.spans[i-1].1` against an
//! UNMERGED set it passes, and removing the merge makes it fail. Its three neighbours
//! (`touching_ranges_merge_so_a_span_across_two_chunks_is_held`,
//! `a_gap_between_two_spans_stays_a_gap`,
//! `an_empty_mirror_holds_nothing_but_still_answers_a_zero_length_read`) carry the rest
//! of the boundary.
//!
//! ⚠ Until 2026-09-03 this sentence cited an integration test named after the hole
//! itself, which has never existed in this crate — the `tests/` directory holds
//! `leaf_shape.rs` and `mirror.rs` and nothing else. The property was proven the whole
//! time and the pointer was imaginary, which is worse than no pointer: a reader who
//! sees a citation stops looking for the coverage. `nornir-testmatrix`'s
//! `citation_guard` is what caught it, and it was the LAST phantom in the repo.
//! (The old path is deliberately not spelled here — writing it would be a citation,
//! and the guard would flag this paragraph.)
//!
//! # What it deliberately does NOT do
//!
//! * It does not decode a page, so it needs no codec, no arrow and no schema knowledge.
//! It moves opaque bytes at offsets the footer named.
//! * It does not decide WHICH row groups you need. That predicate is the caller's — it
//! is the caller's statistics, the caller's query. [`Need::row_groups_where`] is a
//! convenience over the caller's own closure, not a query planner.
//! * It does not fetch. See point 4.
//! * It does not write parquet. The source file is authored elsewhere; this mirrors it.
//!
//! # Freshness, and the one case where the caller has to choose (2026-09-03)
//!
//! [`Mirror::open`] takes the version it EXPECTS and refuses a mirror that disagrees.
//! That is the right shape whenever the source can be asked, and it is not negotiable: a
//! re-written parquet moves every byte offset in its footer, so a mirror of the old one
//! is not "mostly right", it is arbitrary bytes at plausible addresses.
//!
//! But a consumer that is OFFLINE has no source to ask, and refusing to open the file it
//! already holds turns "the server is down" into "you have nothing" — which is the one
//! situation a mirror exists for. So [`Mirror::held_version`] reads the token out of the
//! sidecar on its own, and the CALLER decides: an offline reader opens on that version
//! and knows exactly what it has assumed, and the first answer from a live source re-runs
//! [`Mirror::open`]'s comparison and discards the mirror if the cut moved. The assumption
//! is bounded by the next successful connection and it is never silent.
//!
//! It is two functions and not a flag on one because the two callers are different
//! people: everything with a wire uses `open`, and only a deliberately offline read
//! reaches for `held_version`. A `bool` parameter would let the offline behaviour arrive
//! at the online call site by accident.
//!
//! # Driven live, 2026-09-03 (korp lane MT)
//!
//! The crate's first real consumer — korp's `map_mirror` over a `MapFile` gRPC — was
//! driven end to end for the first time on this date: a thin client on a private X
//! display against a real server holding the live Sweden cut, 1.5 GB of ways, 1.0 GB of
//! nodes and 379 MB of relations. **415 range requests moved 43.8 MB**, the presence
//! bitmap came back at 315/14 807 · 84/442 · 7/113 697 chunks, the sparse claim held
//! (2.8 GB apparent, 43 MB on disk), a repeat fill of the same viewport moved **0 bytes
//! and made 0 requests**, and with the server process killed and the client restarted
//! cold the ordinary parquet reader answered 20 000 ways per viewport out of the holes.
//! Nothing in this crate changed for it except [`Mirror::held_version`] — which is the
//! result worth recording.
#![deny(missing_docs, rust_2018_idioms)]
mod error;
mod plan;
mod presence;
mod reader;
mod store;
pub use error::Error;
pub use plan::{plan, Need, Piece, Plan, Range};
pub use presence::Presence;
pub use reader::MirrorReader;
pub use store::{Mirror, Source};
/// The parquet file magic, at BOTH ends of every parquet file.
///
/// It is `pub` because a mirror's first four bytes are the caller's problem the moment
/// it stops using [`Mirror::create`] — and because the fact that a sparse file's byte 0
/// is `0x00` and not `P` is the first thing that breaks when this is re-implemented.
pub const PARQUET_MAGIC: [u8; 4] = *b"PAR1";
/// Bytes of fixed trailer after the footer's thrift: a `u32` length + [`PARQUET_MAGIC`].
pub const FOOTER_TRAILER_LEN: u64 = 8;
/// **The last 8 bytes of the source file** — fetch these first; they say how long the
/// footer is. Step one of the two-step footer handshake ([`footer_range`] is step two).
#[must_use]
pub fn tail_probe(source_len: u64) -> Range {
Range { offset: source_len.saturating_sub(FOOTER_TRAILER_LEN), len: FOOTER_TRAILER_LEN }
}
/// **The whole footer's byte range**, given the 8 bytes [`tail_probe`] named.
///
/// Returns [`Error::NotParquet`] when the magic is absent — which is what a caller
/// pointed at an HTML error page, a truncated download or a text file gets, and is the
/// only chance to notice before the ranges become nonsense offsets.
pub fn footer_range(source_len: u64, tail8: &[u8]) -> Result<Range, Error> {
if tail8.len() != FOOTER_TRAILER_LEN as usize {
return Err(Error::NotParquet { why: format!("tail probe is {} bytes, want 8", tail8.len()) });
}
if tail8[4..8] != PARQUET_MAGIC {
return Err(Error::NotParquet { why: "no PAR1 trailer".into() });
}
let meta_len = u64::from(u32::from_le_bytes([tail8[0], tail8[1], tail8[2], tail8[3]]));
let len = meta_len + FOOTER_TRAILER_LEN;
if len > source_len {
return Err(Error::NotParquet {
why: format!("footer claims {len} bytes of a {source_len}-byte file"),
});
}
Ok(Range { offset: source_len - len, len })
}