All-core footer-stat build — for idiots
The ground-up version. No Iceberg background assumed.
The kitchen
Imagine skade writing a bulk load as a restaurant kitchen with 12 cooks.
Step 1 — cook the dishes. Each "dish" is one Parquet file. All 12 cooks cook at once — one file each, in parallel. Fast. Everybody busy. (This is the encode.)
Step 2 — write the little index card for each dish. Before a dish goes out, we write a card describing it: how many rows, the smallest and largest value in each column, how big it is. A waiter (the scan planner) later reads these cards to skip dishes it doesn't need — that's what makes reads fast. (This is build_data_file parsing the Parquet footer.)
The bug
We used to make one cook write all the index cards, one after another, after everyone finished cooking. So for a few seconds: 1 cook scribbling cards, 11 cooks standing around doing nothing.
That idle is silly. Writing a card is real work, and we had 11 free cooks.
The fix
Now each cook writes their own dish's index card the moment they finish cooking it — still all 12 in parallel. When the cooking is done, the cards are done too. The only thing left is the waiter carrying plates out (writing the bytes to disk), which is just carrying, not cooking.
Did anything about the food change?
No. Same dishes, same cards, down to the last byte. We only stopped one cook from doing what twelve could do together. A test (write_output_is_byte_identical) literally compares "cards written the old way" against "cards written the new way" and fails if a single byte differs — and we checked it really does fail if we break it on purpose.
What do I do?
Nothing. Call ingest_parallel or ingest_multi exactly as before. You just get your load done with all the cooks working the whole time.