nordisk/znippy-zoomies

znippy-zoomies — design (f0, the whole)

Class: design · Function axis: f0 (the toolbox) + gatling · stree · decompressors · chunk_revolver · vann · scanners · psort. Strict: this file states only the shape we ship. Every why, every rejected alternative, lives in reasoning.md and the per-function *-reasoning.md files.

f0 — how the parts wire into one thing

znippy-zoomies is a single library crate (src/lib.rs) plus a workspace of decompressor member crates (Cargo.toml:17). Everything shares one substrate and one law: the no-barrier gatling engine, and no rayon in the shipped tree (guarded by tests/rayon_free_law.rs).

The dependency spine, top to bottom:

The reuse graph (uses): decompressors.uses = [gatling, chunk_revolver]; scanners.uses = [gatling_forkjoin]; psort, stree, vann are leaf primitives that the engine feeds (a sort/search/score inside a worker) but do not depend on it. vann.uses = [psort] conceptually (top-k and any future k-means both want the parallel sort). This graph is why they live in one crate: the exact-key tree, the fuzzy-vector index, the no-barrier driver, and the parallel sort are one toolbox, reusable by skade, nornir, and znippy without pulling a whole application repo.

The crate also exposes a single introspection seam, functional_status(component, check, ok, detail) (lib.rs:35) — a compiled-out no-op unless the testmatrix feature is on, when each engine run appends a nornir test-matrix row.

gatling — the no-barrier engine

src/gatling.rs (streaming), src/gatling_forkjoin.rs (fixed 0..n), src/gatling_io.rs (async I/O), src/gatling_ordered.rs (item-oriented ordered sink). One idea in several bodies: a worker claims its next unit with a single atomic fetch_add, runs it to completion without waiting on any sibling, and writes into a disjoint output slot. The reader only reads; the collector only stitches in order; all heavy work is in the middle. Full architecture in gatling-design.md.

stree / stree32 — AVX2 static search tree

src/stree.rs (STree64, i64, B=8) and src/stree32.rs (STree32, u32, B=16, Ragnar's original width). An implicit cache-line-sized S(+)-tree: children by arithmetic (no pointers), a branchless SIMD count_lt node compare, and a software-pipelined prefetched batch (batch_stream::<P>) that hides the miss latency. Two concrete types, never a generic STree<T>. Full architecture in stree-design.md.

decompressors — the parallel (de)compressor family

Workspace members lgz (multi-member gzip), lbzip2 (bzip2), linflate (DEFLATE core), ljar (JAR), lzip-parallel (ZIP). Each parallel-decodes a stock archive format across every core through the gatling + chunk_revolver pipeline. These are the subjects of the bake-off (see benchmarks.md); their shapes are in decompressors-design.md.

chunk_revolver — zero-allocation slot pool

gatling/src/revolver/ (revolver.rs, coalesce.rs, ring.rs). Two modes. Fixed-slot: ChunkRevolver::new(chunk_size, total_chunks, num_workers) owns per-worker rings of fixed-size slots; try_get_chunk() hands the reader a free slot, workers borrow it as Chunk<'a> (&[u8]), and return_chunk(ring, index) recycles it. Coalescing: ChunkRevolver::with_coalescing(num_slots, slot_size, slice_size) packs many variable-width payloads into ONE slot and reference-counts it, so a memory budget buys an order of magnitude more small files in flight. Either way, 1-reader/N-worker streaming with no per-chunk Vec<u8> allocation. Details in chunk_revolver-design.md.

scanners — vtd / xml / json

src/vtd.rs (VTD-style parallel OSM-XML → compact byte-offset ElemIndex + zone-map summaries), src/xml.rs (the tag-agnostic scan machinery underneath), src/json.rs (zero-copy parallel NDJSON scan + escape-aware find_key). All three compute sub-chunk borders up front, then map them across gatling_forkjoin::gatling_for_each and merge per-chunk results in index order.

psort — parallel sample sort

src/psort.rs: samplesort_aos_by_i64_key(&mut [[u8; RECORD_SIZE]]) (:62) sorts fixed 16-byte array-of-structs records by their leading little-endian i64, all cores at once; radix_sort_aos_by_i64_key (:257) is the reference LSD radix.

vann — exact vector index

src/vann/mod.rs: VectorIndex is an exact brute-force f32 nearest-neighbour oracle (100 % recall). search_i8_rerank (:266) runs a provably-lossless int8/VNNI cosine prefilter (score_i8_batch, vpdpbusd) then an f32 rerank that is byte-identical to the flat oracle; search_batch (:353) overlaps prefetch across queries. Runtime SIMD detection via active_simd() (:518). Details in vann-design.md.

LAW 6 — the canonical branch, and the guard that makes the trap impossible

Ask the repository which branch is canonical. Never assume, never hardcode, never memorise a table. Several repositories in this constellation carry BOTH a main and a master on the remote and they do not agree on which is canonical — one repository's trunk is the next one's abandoned branch. Enumerate them; do not learn them:

for r in ~/git/*/; do
  both=$(git -C "$r" ls-remote --heads origin main master | wc -l)
  [ "$both" = 2 ] && printf '%-20s canonical=%s\n' "$(basename "$r")" \
      "$(git -C "$r" symbolic-ref --short refs/remotes/origin/HEAD)"
done

On 2026-08-30 that listed exactly three, and the answer was not the same for all three:

Two lanes pushed to a non-canonical trunk in one week. The gunnar one was harmless only by luck — that tip happened to be an ancestor of master, so nothing was stranded; the znippy one was not.

What the hook refuses: a push whose destination is a trunk name (main, master, trunk, develop) that is not this repository's canonical trunk. A deliberate push to a feature branch is untouched — roadmap/obruten-kedja, lane/whatever, tags: all pass. The trap is the second default-looking branch, not branches in general.

How it knows: git symbolic-ref refs/remotes/<remote>/HEAD — the remote's own answer — falling back to asking the server with git ls-remote --symref. The canonical name is never written in the hook; the same bytes therefore give opposite answers in znippy and znippy-zoomies, which is the whole design. A fourth place for the repositories to disagree would be the bug, not the fix. If neither source answers, the hook says so and allows: a hook that blocked every push on a repository it could not resolve would be deleted within the day, and then nothing would guard anything.

The refusal names the branch that is right, the command that pushes there (git push <remote> HEAD:<canonical>), the enumeration above, and the deliberate override (GIT_ALLOW_NONCANONICAL_TRUNK_PUSH=1) for the one honest case — maintaining or retiring the non-canonical trunk on purpose.

Install (hooks live outside the working tree, so a clone does not get them):

cp .githooks/pre-push .git/hooks/pre-push && chmod +x .git/hooks/pre-push

The hook chains to git lfs pre-push itself when .gitattributes tracks LFS paths (it buffers stdin and replays it), so it replaces the LFS hook rather than fighting it.

Proven both ways in all three repositories (LAW 2 — never accept a green you have not seen red), 2026-08-30:

and end to end: git push --dry-run origin HEAD:main is refused in gunnar and in znippy, while git push --dry-run origin master / origin main still runs the LFS half.

Which recipe reader this repo uses, and why (lane AJ, 2026-08-30)

Reader: nornir-build-thing 0.1.12, via xtask's [patch.crates-io] onto the edda path.

Why: in-constellation; recipe is schema 1. The requirement had been the floating "0.1" while the lock still pinned 0.1.4 — seven generations stale, with nothing to force it forward. The [patch] (not a direct path =) is deliberate and documented in the manifest: it is overridable per-checkout and it keeps the graph on ONE copy.

The rule itself — [nornir-build-thing] schema = N, a refusal that names the version to upgrade to, and a loader that refuses to DROP a key it cannot read — lives once, in edda .nornir/build-design.md, "THE COMPATIBILITY RULE". The cross-version guard that would have caught the 2026-08-30 skew is edda/nornir-build-thing-skew.