nordisk/znippy

RULE: Never add Co-Authored-By or any AI attribution to commits. Ever.

znippy workspace

Local environment

Run benchmarks with output going to ~/work/ to avoid thrashing the HDD. OSM benchmark inputs read from T9 (sequential, HDD is fine for large reads).

Dependency graph

znippy-zoomies/linflate          (DEFLATE core)
├── znippy-zoomies/lgz           (gzip)
├── znippy-zoomies/ljar          (JAR/ZIP)
└── znippy-zoomies/lzip-parallel (ZIP)

znippy-zoomies/ldeflate          (DEFLATE write side; owns the already-compressed probe)
└── znippy-common                 (re-exports it as znippy_common::precompressed)

znippy-zoomies/lbzip2            (bzip2, independent)
znippy-zoomies/zstd-sys-rs       (zstd FFI, independent)
znippy-zoomies/osm-katana        (OSM→GeoParquet/PBF, uses lgz + lbzip2 + zstd-sys-rs)

znippy-zoomies                   (fast primitives: OSM index, gatling, psort)
znippy-common / znippy-compress / znippy-decompress / znippy
znippy-plugin-maven              (uses ljar)
znippy-plugin-python             (uses lzip-parallel)
znippy-cli

Release gate protocol

Before cargo publish for any crate:

1. Run benchmarks: cargo bench -p <crate>
2. Compare result against crate's bench_history.json — no result may drop >10% vs last entry for same machine
3. If valid: append result to bench_history.json, commit, git tag vX.Y.Z, cargo publish

Publish order (when linflate changes, cascade)

ldeflate is not on crates.io yet (index.crates.io has no ld/ef/ldeflate, checked 2026-08-03) and znippy-common now depends on it, so cargo publish -p znippy-common will fail until ldeflate 0.1.0 is published. It is a leaf — gatling/rotaryengine, flate2, anyhow — so it can go up at any time, ahead of the znippy cascade below.

ldeflate → znippy-common → …            (the probe's home; publish first)

linflate → lgz, ljar, lzip-parallel → znippy-zoomies → osm-katana
                                                       → znippy-plugin-maven
                                                       → znippy-plugin-python
                                                       → znippy

Benchmark history files

Benchmark column schema — decompressors

Each entry must include reference_mbs, single_core_mbs, and multi_core_mbs so the two axes of improvement (pure-Rust vs C, single-core vs parallel) are always visible.

{
  "date": "2026-05-30",
  "version": "0.3.2",
  "crate": "lgz",
  "machine": "Ryzen 9 7900",
  "cores": 12,
  "results": [
    {
      "name":            "test_500mb_gz",
      "input_mb":        500,
      "reference_tool":  "gzip -d",
      "reference_mbs":   120.5,
      "single_core_mbs": 185.3,
      "multi_core_mbs":  2840.2,
      "cores_used":      12
    }
  ]
}

• reference_mbs — system tool, single-threaded (gzip/bzip2/unzip baseline)
• single_core_mbs — this crate, 1 thread (pure-Rust advantage over reference)
• multi_core_mbs — this crate, all cores (parallel advantage on top)

Benchmark column schema — osm-katana

Same three-column principle; reference tool is typically bzip2 -d | osmconvert or osmium.

{
  "name":            "liechtenstein_bz2_to_geo",
  "input_format":    "bz2",
  "input_mb":        5.2,
  "reference_tool":  "bzip2 -d | osmconvert",
  "reference_mbs":   8.5,
  "single_core_mbs": 14.2,
  "multi_core_mbs":  95.0,
  "cores_used":      16
}

Contracts

decompressors → znippy-plugin-maven

• ljar: parallel JAR decompression, feature host-decompressors

decompressors → znippy-plugin-python

• lzip-parallel: parallel ZIP decompression, feature host-decompressors

decompressors → osm-katana

• lgz: gzip OSM input
• lbzip2: bzip2 OSM input (most common distribution format)
• zstd-sys-rs: zstd OSM input

Doc update rule at release

At every release, before tagging:

• CLAUDE.md — update architecture sections if code changed
• README.md — update the headline number and benchmark table to current version; move the previous entry to a ## Benchmark history section at the bottom (append, never overwrite history)
• bench_history.json — append new result line (never edit existing lines)

README.md structure:

# znippy
<3-line description>
**<best number> · <cores> cores · v<version>**
<benchmark table — current version only>

## Benchmark history
| version | date | best number | machine |