znippy-common/src/precompressed.rs
//! Deciding whether a payload is **already compressed**, and so should be stored
//! raw instead of handed to the codec.
//!
//! ## Where the probe lives
//!
//! In [`ldeflate::precompressed`], not here. This module is a **re-export** plus
//! the one leg that cannot live there: the path/extension fallback.
//!
//! The direction is forced, not chosen. `znippy-common` already depends on
//! `znippy-zoomies` (`Cargo.toml:96`, plus `ljar`/`lbzip2`/`lgz`), so an
//! `ldeflate -> znippy-common` edge would close a cycle; and `znippy-common`
//! drags `arrow`, `blake3` and an `openzl-sys` whose `build.rs` fetches a tarball
//! over the network — for 88 lines of `std`-only byte matching. The dependency
//! arrow already pointed at zoomies, which makes `ldeflate` the home: it is the
//! leaf, and it is the *compressor*, the only thing that ever needs to ask the
//! question.
//!
//! What is left here is genuinely znippy's: [`SkipPolicy::skip_by_path`] consults
//! [`crate::index::is_probably_compressed`], an extension table that `ldeflate`
//! has no business knowing about — it is handed `&[u8]`, never a `Path`.
//! [`SkipPolicy`] is therefore a thin wrapper that adds the path leg and
//! **delegates** the byte leg to `ldeflate::precompressed::SkipPolicy::skip`.
//! There is one magic table, one `looks_compressed`, one `is_zlib_stream`, and
//! one hint-to-verdict mapping for bytes — in `ldeflate`.
//!
//! ## The resolution order
//!
//! | # | mechanism | cost | how it can be wrong |
//! |---|---|---|---|
//! | 1 | an explicit caller [`ContentHint`] | free | only if the caller lies |
//! | 2 | the extension table ([`crate::index::is_probably_compressed`]) | one `str` compare | the name can lie |
//! | 3 | a [magic-byte probe](looks_compressed) over the first [`SNIFF_PREFIX_LEN`] bytes | ≤ 16 byte compares | a container whose magic is not at offset 0 |
//! | 4 | default | — | — |
//!
//! **The hint is the primary mechanism, not the fallback.** A caller sealing a
//! git packfile *knows* what it is holding; no amount of inspection beats being
//! told. It is deterministic by construction, costs nothing, and carries
//! knowledge the bytes themselves do not.
//!
//! ## Why there is no entropy estimate
//!
//! An entropy estimate was considered and **rejected**. It is a heuristic with a
//! tunable threshold, it costs a scan proportional to the data, and it misjudges
//! in both directions — high-entropy-but-compressible (a table of hashes with a
//! compressible envelope) and low-entropy-but-already-compressed (a short zstd
//! frame) both exist. A wrong guess wastes CPU or wastes bytes and reports
//! neither. Every probe in `ldeflate` is exact: a fixed byte pattern at a fixed
//! offset, no threshold, no sample size to tune.
//!
//! ## What a probe does *not* claim
//!
//! [`looks_compressed`] reads only the head of a buffer. For a file sliced into
//! chunks, only chunk 0 carries the magic. The big-file pass in `znippy-compress`
//! therefore takes chunk 0's verdict and carries it over the whole file — but
//! **excluding** [`is_zlib_stream`], which is a two-byte test accepting 1 in 2048
//! two-byte heads (measured exhaustively over all 65 536, not sampled) and so is
//! not a claim to make about 8 GiB. See `slot_packer::carries_file_wide`. The
//! extension table covers the whole file at once and so has no such gap, which is
//! why `.pack` / `.idx` belong there too.
//!
//! ## The git case, precisely
//!
//! Two different things are called "a git object" and they need opposite
//! answers:
//!
//! * a **loose object file** in `.git/objects/ab/cdef…` is a zlib stream, has no
//! extension at all, and must be stored raw. [`is_zlib_stream`] catches it.
//! * a `znippy-plugin-git` archive entry is the object's **canonical form**
//! (`"<type> <size>\0<content>"`) — inflated bytes that compress perfectly
//! well and *must* be compressed. It begins `commit `/`tree `/`blob `/`tag `,
//! matches no probe there, and is correctly left alone.
//!
//! A rule of the shape "an oid-keyed entry is already compressed" would get the
//! second case backwards and silently store compressible data raw. Probing the
//! bytes at offset 0 gets both right. Both halves are asserted against a real
//! encoder in `ldeflate::precompressed`'s own tests.
use std::path::Path;
/// The probe itself, re-exported so `znippy_common::precompressed::*` keeps
/// naming exactly what it always named. These are `ldeflate`'s items, not copies
/// of them — breaking one there breaks every znippy caller, which is the point
/// of the move.
pub use ldeflate::precompressed::{ContentHint, SNIFF_PREFIX_LEN, is_zlib_stream, looks_compressed};
/// A batch-level compression policy: one [`ContentHint`] covering everything in
/// a run, plus the fallbacks for when the hint says nothing.
///
/// Batch-level because that is how the knowledge actually arrives — a caller
/// sealing a pack knows it for the whole pack, not entry by entry.
///
/// The two decision points are deliberately split, because the compress path has
/// two moments with different information available:
///
/// * [`skip_by_path`](Self::skip_by_path) runs during enumeration, before a
/// single byte is read;
/// * [`skip_by_bytes`](Self::skip_by_bytes) refines it inside the compressor,
/// once a chunk's real bytes are in hand, and is
/// `ldeflate::precompressed::SkipPolicy::skip` under a name that says which of
/// the two moments it belongs to.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct SkipPolicy(ldeflate::precompressed::SkipPolicy);
impl SkipPolicy {
/// No claim: decide per entry from its name, then from its bytes.
pub const fn resolve() -> Self {
Self(ldeflate::precompressed::SkipPolicy::resolve())
}
/// Everything in this batch is already compressed — store it all raw.
pub const fn already_compressed() -> Self {
Self(ldeflate::precompressed::SkipPolicy::already_compressed())
}
/// Compress everything, whatever it looks like. The meaning of `--no-skip`.
pub const fn compress_everything() -> Self {
Self(ldeflate::precompressed::SkipPolicy::compress_everything())
}
/// Bridge from the historical `no_skip: bool`, which is precisely the
/// two-valued subset of [`ContentHint`] that the CLI already exposed.
pub const fn from_no_skip(no_skip: bool) -> Self {
if no_skip { Self::compress_everything() } else { Self::resolve() }
}
pub const fn from_hint(hint: ContentHint) -> Self {
Self(ldeflate::precompressed::SkipPolicy::from_hint(hint))
}
pub const fn hint(&self) -> ContentHint {
self.0.hint()
}
/// The same policy as `ldeflate` sees it — for a caller handing bytes
/// straight to `ldeflate::compress_objects` rather than to znippy's packer,
/// so the two cannot disagree about what a hint means.
pub const fn as_ldeflate(&self) -> ldeflate::precompressed::SkipPolicy {
self.0
}
/// The decision available before any bytes are read: hint, else the
/// extension table.
///
/// This is the leg `ldeflate` cannot have — it is handed `&[u8]`, never a
/// `Path`.
///
/// `true` means "store raw". A `false` here is **not** final — the caller
/// should still offer the bytes to [`skip_by_bytes`](Self::skip_by_bytes)
/// once it has them.
pub fn skip_by_path(&self, path: &Path) -> bool {
match self.0.hint() {
ContentHint::AlreadyCompressed => true,
ContentHint::Compressible => false,
ContentHint::Unknown => crate::index::is_probably_compressed(path),
}
}
/// The refinement once real bytes are in hand: a magic-byte probe.
///
/// Only meaningful when [`skip_by_path`](Self::skip_by_path) returned
/// `false`; calling it otherwise is harmless but pointless. Honours the hint
/// in both directions, so a `Compressible` batch is never vetoed by a probe.
pub fn skip_by_bytes(&self, prefix: &[u8]) -> bool {
self.0.skip(prefix)
}
}
impl From<ldeflate::precompressed::SkipPolicy> for SkipPolicy {
fn from(p: ldeflate::precompressed::SkipPolicy) -> Self {
Self(p)
}
}
impl From<SkipPolicy> for ldeflate::precompressed::SkipPolicy {
fn from(p: SkipPolicy) -> Self {
p.0
}
}
#[cfg(test)]
mod tests {
use super::*;
// The probe's own behaviour — the magic table, the bzip2/WebP special cases,
// the zlib header against streams from a real encoder, canonical git object
// bytes — is asserted in `ldeflate::precompressed`, which is where the code
// now is. Repeating it here would be the twin this change removed.
//
// What is asserted here is what this crate adds: the path leg, the bridge
// from `no_skip`, and — first, because everything else assumes it — that the
// re-export really does reach `ldeflate` rather than a leftover copy.
/// The dedup itself. If `znippy-common` ever grows a second magic table
/// again, these calls stop agreeing.
#[test]
fn the_probe_is_ldeflates_and_not_a_second_copy() {
for bytes in [
b"PACK\x00\x00\x00\x02".as_slice(),
b"\x1f\x8b\x08\x00".as_slice(),
b"\x28\xb5\x2f\xfd\x00\x58".as_slice(),
b"RIFF\x24\x00\x00\x00WEBPVP8 ".as_slice(),
b"\x78\x9c".as_slice(),
b"SELECT * FROM njord WHERE id = 42;".as_slice(),
b"x = 1;\n".as_slice(),
b"".as_slice(),
] {
assert_eq!(
looks_compressed(bytes),
ldeflate::precompressed::looks_compressed(bytes),
"the re-export must BE ldeflate's probe: {bytes:?}"
);
assert_eq!(
is_zlib_stream(bytes),
ldeflate::precompressed::is_zlib_stream(bytes),
"the re-export must BE ldeflate's zlib probe: {bytes:?}"
);
// And the policy's byte leg must be the same decision, not a
// parallel one.
assert_eq!(
SkipPolicy::resolve().skip_by_bytes(bytes),
ldeflate::precompressed::SkipPolicy::resolve().skip(bytes),
"skip_by_bytes must delegate: {bytes:?}"
);
}
assert_eq!(SNIFF_PREFIX_LEN, ldeflate::precompressed::SNIFF_PREFIX_LEN);
}
/// The wrapper must not lose the hint on the way in or out — the failure
/// that would make `--no-skip` silently stop meaning anything.
#[test]
fn the_hint_survives_the_wrapper_in_both_directions() {
for (p, want) in [
(SkipPolicy::resolve(), ContentHint::Unknown),
(SkipPolicy::already_compressed(), ContentHint::AlreadyCompressed),
(SkipPolicy::compress_everything(), ContentHint::Compressible),
] {
assert_eq!(p.hint(), want);
assert_eq!(p.as_ldeflate().hint(), want, "the ldeflate view must carry the same hint");
assert_eq!(SkipPolicy::from(p.as_ldeflate()), p, "round-trip through ldeflate");
assert_eq!(ldeflate::precompressed::SkipPolicy::from(p), p.as_ldeflate());
assert_eq!(SkipPolicy::from_hint(want), p, "from_hint must agree with the constructor");
}
assert_eq!(SkipPolicy::default(), SkipPolicy::resolve(), "the probe is the default");
}
// ── the hint, both directions ────────────────────────────────────────────
#[test]
fn an_already_compressed_hint_is_honoured_without_inspection() {
let p = SkipPolicy::already_compressed();
// Plain compressible bytes under a plain name: only the hint can decide
// this, and it must.
assert!(p.skip_by_path(Path::new("objects/00ff")), "hint must win on the path");
assert!(p.skip_by_bytes(b"SELECT * FROM njord;"), "hint must win on the bytes");
}
#[test]
fn an_unhinted_batch_of_compressible_data_is_still_compressed() {
// The other direction: a hint that swallows everything is as broken as
// one that is ignored.
let p = SkipPolicy::resolve();
assert!(!p.skip_by_path(Path::new("dbdump/njord.dump")));
assert!(!p.skip_by_bytes(b"SELECT * FROM njord WHERE id = 42;"));
}
#[test]
fn a_compressible_hint_overrules_both_fallbacks() {
let p = SkipPolicy::compress_everything();
assert!(!p.skip_by_path(Path::new("photo.png")), "--no-skip must beat the extension table");
assert!(!p.skip_by_bytes(b"\x89PNG\r\n\x1a\n"), "--no-skip must beat the probe");
}
#[test]
fn no_skip_maps_onto_the_hint() {
assert_eq!(SkipPolicy::from_no_skip(true), SkipPolicy::compress_everything());
assert_eq!(SkipPolicy::from_no_skip(false), SkipPolicy::resolve());
}
/// The whole point of a sniffer: a name that lies.
#[test]
fn an_extension_that_lies_is_caught_by_the_bytes() {
let p = SkipPolicy::resolve();
let notes = Path::new("release-notes.txt");
let zstd_bytes = b"\x28\xb5\x2f\xfd\x00\x58\x2d\x00\x00";
assert!(!p.skip_by_path(notes), "the name says text, so the fast path lets it through");
assert!(p.skip_by_bytes(zstd_bytes), "the bytes say zstd, and the bytes are the authority");
}
/// And the same trap in reverse: a `.pack` name over compressible bytes is
/// still skipped by the fast path. Documented so the asymmetry is a decision
/// rather than a surprise — the extension table is trusted, by design.
#[test]
fn the_extension_fast_path_covers_a_whole_file_not_just_its_head() {
let p = SkipPolicy::resolve();
assert!(p.skip_by_path(Path::new("objects/pack/pack-9f2c.pack")));
assert!(p.skip_by_path(Path::new("objects/pack/pack-9f2c.idx")));
}
/// A git loose object is the case with **no extension at all**, so only the
/// byte leg can reach it — and the path leg must say "not from the name",
/// which is what makes the byte leg necessary rather than decorative.
#[test]
fn an_oid_named_loose_object_is_reached_only_by_the_bytes() {
let p = SkipPolicy::resolve();
let oid = Path::new(".git/objects/9f/2c1e0b7a4d3f8c6e5a2b9d0c7f4e1a8b3d6c9e2f");
assert!(!p.skip_by_path(oid), "an oid has no extension for the table to match");
assert!(p.skip_by_bytes(b"\x78\x01\x4b\xca\x49"), "so the zlib header must catch it");
}
}