nordisk/znippy-zoomies

znippy-zoomies — for idiots (f0, the whole)

Class: idiot. Every step, nothing assumed. Goal: run the decompressor bake-off and see znippy-zoomies win, from a cold shell to a printed table. If you already know Rust, read guide.md instead — this one spells out every keystroke.

What you need first

• A terminal (the black window where you type commands).
• The repo on disk. In the terminal, type this and press Enter:

cd /run/media/rickard/T9/git/znippy-zoomies

• The Rust toolchain (cargo) and the nornir command already installed (they are, on this machine).

Step 1 — build everything, optimised

Type this and press Enter. It takes a while the first time; that is normal.

cargo build --release

--release means "make it fast." Without it the numbers are meaningless. Red text at the end = an error; fix what it names. Finished = you are good.

Step 2 — check the fast primitives work

Type and press Enter:

cargo test --release stree32_bench -- --nocapture

You will see lines comparing binary_search against STree32. The speedup column should say something like 9× to 20×. That is the search tree beating the standard library. If it says test result: ok, it works.

Step 3 — run the big bake-off

This is the main event. It compresses a big file into every format, then makes every decompressor on the machine race to unpack it. Type and press Enter:

nornir bench run znippy-zoomies

It will churn for a few minutes — it is using all your cores and a lot of memory on purpose (that is what "HEAVY" means). Let it finish. When it is done it has saved one row of numbers into nornir's warehouse.

If you want a smaller, faster run the first time, set the corpus smaller before the command (still on one line, then Enter):

NORNIR_ZOOMIES_CORPUS_MB=256 nornir bench run znippy-zoomies

For the real, quotable numbers, leave it at the default (1024 MB).

Step 4 — turn the numbers into a document

Type and press Enter:

nornir docs book znippy-zoomies

This writes three PDFs into the docs/ folder. The one you want is docs/manual.pdf — it contains the big comparison table.

Step 5 — read the table

Open docs/manual.pdf (double-click it, or type xdg-open docs/manual.pdf and press Enter). Find the Decompressor bake-off table. Each row is one tool. The bold cell in each column is the winner for that measurement. Look at the percore_mbs column (throughput per core): the znippy-gz and znippy-bz2 rows — those are ours — leave the plain serial tools (bzip2, pbzip2) far behind.

That is the whole thing. You built it, you ran a real all-cores benchmark, and you read the result — without writing a line of code.

osm-katana census — for idiots

You have a folder of map tables and someone asked "how many of them have no tags?" Do NOT write a script. This is the whole job.

Step 1 — build it once

Copy this line, paste it into a terminal, press Enter.

cargo build --release -p osm-katana

Step 2 — ask the question

Point it at the folder. That is it.

./target/release/osm-katana census ~/.korp/work/sweden --tags

It prints a block per file. Three seconds for two billion-row tables.

Step 3 — read exactly two things

First: the last line.

VERDICT: books close, every population accounted for.

If it says VERDICT: REFUSED, stop. Do not copy any number off the screen. The lines under it say what is wrong in plain words. The commonest one is that you pointed it at the wrong folder, and it read no files at all.

Second: the line that answers your question.

tags: 103488583 rows = tagged 2406592 of 103488583 (2.3 %)
      + untagged: {} 101081991 of 103488583 (97.7 %) · UNACCOUNTED 0

Read it left to right:

• 103488583 rows — how many rows there are. This is the file's own count.
• 101081991 of 103488583 (97.7 %) — the answer. The percentage always comes with the two numbers it is made of. If you quote it anywhere, quote all three.
• UNACCOUNTED 0 — the important one. It means every single row landed in one of the buckets on this line. Nothing fell off the edge.

If UNACCOUNTED is anything but 0, the percentages disappear from the line on purpose. That is not a bug. A share of a population that lost rows is a wrong number that looks right, and this tool will not print one.

Step 4 — if you are going to quote it, save it

./target/release/osm-katana census ~/.korp/work/sweden --tags --json census.json

census.json holds the same numbers plus which files they came from — every path, its size, its timestamp and the recipe the tables were cut by. Link to that file. Someone six months from now can then check your figure instead of believing it.

The mistake this exists to stop you making

Someone once wanted the tag numbers for nineteen map tiles. They copied the tiles into one folder first. The tiles were named 33/15.parquet, 34/15.parquet, 35/15.parquet … and copying them flattened those names to 15.parquet — so twelve of the nineteen silently overwrote each other and seven were left. The percentage that came out looked completely normal. It went into two design documents and two commit messages before anyone noticed.

So: never copy the tiles first. Point census --pyramid at the real folder and let it do the walking. If you point it at a flattened copy it will refuse and tell you why — which is the one thing the script that caused all this could not do.

Other things you can ask

Add these to the same command:

Without any of them it still gives you rows, file sizes, and which column is eating your disk — and that costs almost nothing, because it only reads the little index at the end of each file.