.nornir/gatling-for-idiots.md

Gatling for idiots — the 1 → N → 1 idea in plain words

Class: idiot. No jargon, nothing assumed.

The story of why Gatling and which one to call. The proper design is in gatling-design.md; the how-to with signatures is gatling-guide.md; the real bug this is all based on is written up with the osm work in the sibling repos (znippy / znippy-osmfeed .nornir/osm-katana-gatling-vs-rayon.md).

The picture

Your computer has lots of cores — little workers that can each do a job at the same time. You want them all busy. If only 1 is busy and the rest sit idle, you're using a few percent of the machine and everything crawls.

Think of a kitchen:

   1 waiter   →   N cooks   →   1 plater
  brings the      each cooks     puts finished
  raw food        a full dish    dishes on the
  (does NOT        start to       pass, in order
  cook anything)   finish, never  (does NOT cook
                   waiting for     anything)
                   the others

1 waiter carries raw food in. Never cooks.
N cooks each grab a ticket and cook a whole dish by themselves. Nobody waits for anybody — the moment a cook finishes, they grab the next ticket.
1 plater lines finished dishes up in the right order. Never cooks.

If all the cooking happens in the cooks, the kitchen flies. The disaster is making the waiter or the plater cook — there's only one of each, so one person does all the hard work while everyone else stands around. That's the bug that once dropped osm-katana from "all 32 cores" to "~3 cores": we'd made the ends cook (read the whole 8 GB then sort it one-at-a-time, walked the whole file to find boundaries, assembled the final result at the pass). Funny twist — while broken, the old rayon way looked faster, and people almost blamed Gatling. Gatling was fine; we'd just told the waiter to cook. We moved all the cooking back into the cooks and it went back to a flat 100% on all cores, even on the entire planet's map (10.58 billion points).

The one thing to remember: the waiter and the plater never cook. All the hard work — finding boundaries, decoding, sorting, parsing, encoding — happens in the cooks in the middle, with nobody waiting on anybody.

Which one do I call?

There's one idea (1 → N → 1) in a few shapes. Pick by your situation:

Do you already have all the work in hand (a list, an array, a grid)? Or is it streaming in (bytes off a file/socket)?

I already have the list → fork-join (`gatling_forkjoin`)

I want the results back, jobs are all about the same size → gatling_for_each(n, 0, |i| ...). Hands you a Vec back in order. (0 = "use every core".)
I just want side-effects, no result list → gatling_run(n, 0, |i| ...).
Some jobs are WAY bigger than others (one giant file among hundreds of tiny ones) → gatling_for_each_balanced(n, 0, batch, weight, f). This is the new one. Two tricks:
- Big ones first. You tell it each job's size (weight) — say, the file's byte count. It hands out the biggest jobs first, so no cook is left grinding on the one monster dish at the very end while everyone else has gone home. (Picture the head chef handing out the wedding cakes before the side salads.)
- Grab a handful at a time. Instead of taking one ticket per trip to the pass, a cook grabs batch tickets (say 8) — fewer trips, same balance. Great when there are thousands of tiny jobs. You still get the results back in original order.
Same as above, but my work is a Vec of mixed jobs and I want to sort the results into buckets afterwards (parse every file across many projects, then file each result under its project) → gatling_map_balanced(&items, 0, batch, size_of, f). Same "big ones first, grab a handful at a time" tricks, but it hands each cook the job itself (f(index, &item)) instead of just a ticket number, and the size comes straight off the job. You get a Vec of results back in original order, ready to drop into buckets by key. (Picture the head chef handing each cook the actual order slip, not just "order #37".)
It's a picture / a grid of rows → gatling_scanlines(...). Each cook owns whole rows; nobody steps on anyone's row.

It's streaming in → the streaming engine (`gatling`)

Bytes that need stitching back together (decompress a .bz2/.gz into XML) → gatling::run. The cooks decode chunks; the plater glues them at safe seams.
Each chunk becomes a finished record (parse files into structured data) → gatling::run_typed. The plater just lines the records up in order.

It's network / disk I/O, not CPU → the async sibling (`gatling::io`)

Uploading a pile of files to S3, doing many network calls, lots of fsyncs? Those cooks aren't cooking, they're standing at the phone waiting for someone to pick up. Threads would just block. Use gatling::io::run (results as they land) or gatling::io::run_ordered (results in order). It keeps a fixed number of calls "in flight" and starts the next the instant one finishes.

The two rules

The ends never cook. Reader reads, plater stitches-in-order, all heavy work is in the cooks.
Never use rayon. It's banned in this codebase — its hidden shared pool is exactly what bit osm-katana. Gatling is the replacement everywhere.

How to tell if you broke it

Watch your cores while it runs (htop, btop, whatever). One core pegged and the rest asleep = you made an end cook, or you wrote a plain for loop. All cores busy = you did it right. If everything's busy at the start but it sags to one core at the end, you've got one monster job — switch to gatling_for_each_balanced and give it a weight.