nordisk/edda

What gunnar's CPU per request is doing under concurrency: localised

Opened 2026-08-04, against origin/master at 71b7bc5. Companion to .nornir/performance-design.md and .nornir/transport-encoding-design.md: nothing here edits either file. New entries use the same block format and continue the numbering at P-023, so they merge unchanged.

The answer, first

The bake-off measured gunnar's CPU per served request rising 2.54× between 1 and 32 concurrent clients (0.418 → 1.063 CPU-seconds) while gitea's fell (0.283 → 0.256), with gunnar's throughput peaking at 16 clients and regressing at 32. It could not localise the cause.

The cause is not in gunnar's request path. It is where gunnar's threads are allowed to run.

One binary, one year corpus (28 893 objects), one 8-CPU budget, smart HTTP, oden 2026-08-04. The only difference between the two arms is how that identical budget is applied:

Pinned, the curve is flat. gunnar's per-request cost does not in fact grow with concurrency. P-004's "concurrency comes from serving many requests at about one core each" is correct, and this document does not overturn it — it names the one thing P-004 did not anticipate, in P-023 below.

The bake-off's condition is podman run --cpus 8 on a 32-core host, which is quota-without-cpuset, so the measured ramp is the top row. It is a real and very common deployment; it is simply not a statement about gunnar's request path.

How it was measured

gunnar serve --http-bind provisioned exactly as bench/examples/forge/mod.rs provisions it — private repository, principal, grant, and a personal access token on every request — in a systemd --user scope whose cpu.max is read back out of the kernel, not out of the request that set it. Clients are stock git 2.53.0.

Two guards, both seen red, the same two the bake-off's concurrent.rs carries and for the same reason — clients that all fail instantly produce a magnificent ops/sec:

Two instruments in this work were blind and were caught, and both are worth recording because both looked exactly like results:

1. Context switches read from /proc/<pid>/status. That file reports the main thread only, and gunnar's main thread sleeps in the accept loop, so the counter read a constant 0 however hard the server worked. Reading /proc/<pid>/task/*/status instead then produced a negative delta, because threads that exit take their counts out of the sum — which is how the second version was caught too.
2. A parser test that could not fail. Cpus_allowed_list on oden is the single range 0-31, so a version of count_cpu_list that counted ranges and silently dropped singletons passed on this box while answering 0 allowed CPUs for a server pinned to one core — the condition read backwards. The test now runs on literals, not on this host.

And one measurement condition worth stating plainly: oden's load ran 6.8 → 99 inside this session under other lanes' work. Every timing figure here is therefore from an interleaved A/B — the two arms alternate inside each rung and the order swaps every round — and the two absolute levels quoted for the same comparison (2.7× on a quiet box, 1.35× at load ~40) differ because a loaded box starves the clients and they press the server less hard. Counts (futex syscalls, objects, bytes) are quoted in preference to times wherever a count would answer the question, because a count is a property of the server and a millisecond on this box was a property of the box.

P-023 — One served request is three threads that hand work to each other, and under a CPU quota every hand-off crosses a core

• Found: 2026-08-04

• Status: DIAGNOSED — one warning landed, no behaviour changed

• Severity: blocker (it is the whole of the measured concurrency regression)

• Where: crates/gunnar-cli/src/server/transport.rs:238 (spawn_blocking + SyncIoBridge) · crates/gunnar-pack/src/copy.rs:374-384 (P-020's thread_limit: Some(1)) · crates/gunnar-cli/src/server/serve.rs (report_cpu_budget, the line this produced)

• What: P-004 decided the serving tier stays serial and that a request costs "about one core". The CPU figure is right. The thread figure is not: serving one clone occupies three threads, and they synchronise with each other continuously rather than running one after another.

  1. a runtime worker, carrying hyper and the socket;
  2. the spawn_blocking thread, which runs the whole of gunnar-wire's blocking state machine for the transfer's entire duration; and
  3. a third OS thread gix_features::parallel::reduce::stepped spawns inside iter_from_counts — which thread_limit: Some(1) does not prevent. P-020 stops the fan-out; it does not stop the one worker, and every chunk of pack entries still crosses a channel to it.

So at N concurrent transfers the scheduler has roughly 3N runnable threads. Given 32 CPUs of affinity and 8 CPUs of quota it spreads them over all 32, and a hand-off between two threads on two different cores is a futex.

Measured, perf record on the serving process at 32 clients:

Futex syscalls per clone scale with the object count, not with bytes and not with writes — which is what says the traffic is the per-object pipeline and not the socket:

With frame pointers the single largest call site resolves to one line: iter_from_counts's worker → entry_by_location → to_vec → Vec::try_allocate_in → glibc's lock. One heap allocation per object served, on a thread that is not the one the transfer runs on. That is ~50% of all futex calls; the tokio blocking pool's own task hand-off is most of the rest.

• Why it matters at load: it is the entire measured regression. Pinned to a cpuset of the same size the curve is flat to 32 clients, so this is not a limit of gunnar's design — it is a limit of running gunnar's threads across four times as many cores as it is allowed to use.

• What was NOT the cause, each eliminated by measurement rather than by reading:

  • A lock on the object store or the odb handle. OdbPool's idle mutex and store rwlock are taken per object lookup, but they are Rust futex mutexes and appear nowhere in the profile: an uncontended one issues no syscall, and confining the same threads to 8 CPUs removed the cost without touching them.
  • The two caches added in 9ce1cca (archive handle, delta). Both are on the znippy path, GUNNAR_COLD_TIER was off in the ramp, and the profile contains no gunnar_coldtier frame.
  • glibc's arena count. MALLOC_ARENA_MAX=1 — every thread forced onto one arena — gave 12 349 futex/clone against 12 291 at =64 and 9 326 and 13 644 on two unmodified runs. The red probe did not fire: run-to-run variance exceeded the effect, so the glibc lock in that stack is recorded as not shown to be arena contention, and the allocation site is reported without a claim about which of glibc's locks it takes.
  • touch_token on the request path. Every authenticated request writes the identity table (AuthTable::touch_token → DocRef::update → object write + ref compare-and-swap, retried up to MAX_ATTEMPTS), and with every client holding the same token this looked like a certain contention point. It does not appear in the profile at all. Recorded because it is the most plausible wrong answer, and reading the code would have shipped it.

• Fix: the operator's, and gunnar must not attempt it itself — a server inside a container does not know which CPUs are its to take, and choosing wrong is worse than the condition. Start the container with --cpuset-cpus naming as many CPUs as --cpus grants, or the unit with AllowedCPUs= alongside CPUQuota=. What landed is Server::start's report_cpu_budget: gunnar compares its own affinity against its own quota, both read from the kernel, and says so once at startup when the first is twice the second.

• Verify: spread_budget is pure and asserted on both sides of its threshold, count_cpu_list on literals rather than on this host, and the line itself observed on a live server in all three states — quota-only (fires), quota+pin (silent), no quota (silent). Seen RED four ways: >= weakened to >, the singleton branch of the CPU-list parser removed, the range width off-by-one, and a reversed range accepted.

• Open, and honestly not attempted here — DONE 2026-08-05. The third thread, and the fourth. Stepwise now runs inline when num_threads(thread_limit) is 1, so a served request spawns zero threads for pack generation: strace -f -c -e trace=clone3 over one served 12 186-object clone counts 2 before, 0 after. It was not upstream in gix-pack as this entry said — Stepwise lives in gix-features, the crate gunnar already vendors, so it was ~40 lines away the whole time. See vendor/gix-features/PATCH.md fix 2 and .nornir/upstream-gix-stepwise-serial.md.

And the per-object hand-off vanishing did not buy anything measurable, which is the honest half. Process CPU per 12 186-object clone is flat (min of 9, taskset-pinned: 79.0 ms both), the 32-client wall moved 1.4% against a ~1.4× spread at loadavg 141, and wire bytes are byte-identical on all 19 corpus repositories × {clone, negotiated fetch}, both hash kinds — 38/38. The hypothesis this entry supports — that the cross-thread to_vec/free pair was buying the futex traffic and therefore the throughput gap — is not confirmed: removing the cross-thread half of it left CPU where it was. What the change is worth is P-004 becoming true of the code, and every per-thread instrument in the repository becoming honest (see below).

• Answered 2026-08-04 by P-025's lane, and the count above is one short. strace -f -c -e trace=clone3 over one iter_from_counts call at thread_limit: Some(1): 2 clone3, not 1, on both a 12 186-object and a 69-object request. gix_features::parallel::reduce::Stepwise::new spawns num_threads(thread_limit) consumers plus an input-feeder thread, and neither is conditional. So a served request is four threads.

The resolve phase — which resolves every count to a pack location, and which P-025 found was inflating the whole repository — is already inline at Some(1), because in_parallel_if takes the serial branch when num_threads(thread_limit) is not > 1. MEASURED: the whole of P-025's saving landed on the calling thread's stage tally (pack_write exclusive CPU 61.84 → 16.73 ms on a 12 186-object clone), which it could not have done had that phase been on a worker. P-020's Some(1) does more than this entry credited it with.

Cost of the two threads, MEASURED oden 2026-08-04 under taskset -c 0-7: 64 µs of process CPU per spawn+join pair-of-two — 0.08% of a 12 186-object clone, ~7.1% of the smallest real fetch in the corpus (bifrost, 69 objects, 0.90 ms). Removing them from gunnar's side means not calling iter_from_counts at all and re-implementing ~90 lines of gix's counts sort, counts_range_by_pack_id index and thin-base resolution — which must reproduce gix's ordering exactly or the wire bytes move. Not done, on LAW 5 and on that ratio. The upstream patch, which costs gitoxide nothing because the serial Stepwise is already in the file, is written out ready to file in .nornir/upstream-gix-stepwise-serial.md.

The per-object hand-off this entry names as the largest futex call site — entry_by_location → to_vec — is the entry phase and survives P-025 untouched. It is now the largest single item left on the read path.

Superseded 2026-08-05, in both halves. The hand-off is gone (the entry phase is inline), and the to_vec that remains is not the largest item left on the read path. Profiled with perf record -F 4999 over 25 served clones of the corpus's nornir.git, now that single-threading makes the profile attributable:

Deflate on a path whose entire purpose is to copy already-deflated entries is the finding. MEASURED with packwrite-probe: a clone of that repository copies 12 150 of 12 186 entries and re-deflates 36 — 0.30% of the objects. Those 36 are deltas whose base is inside the pack but outside the clone's selected object set, which Entry::from_pack_entry cannot re-point and therefore explodes into a whole object.

A/B against the same 12 186 objects served out of a repository repacked so the selection is the pack (git clone --single-branch + git repack -adq):

24 objects — 0.20% of the clone — were 29.5% of its CPU. That is roughly four times the entire allocation-and-copy budget the to_vec represents, and it is the next thing to take on the read path. The fix is the one git already makes and gix's Mode::PackCopyAndBaseObjects is named for: when a selected delta's base is in the pack but unselected, add the base to the selection rather than expand the delta. It is gunnar's own gunnar_wire::graph that chooses the set, so this needs no gix fork — but it changes wire bytes by construction, so it is a deliberate improvement to be measured against stock git, not a byte-neutral one.

P-024 — The smart-HTTP response body took 16 KiB bites out of a 256 KiB window, so the window controlled nothing

• Found: 2026-08-04

• Status: DONE 2026-08-04

• Severity: smell (it is a real defect and it is not P-023's cause)

• Where: crates/gunnar-http/src/body.rs — Streamed::poll_frame

• What: HttpConfig::stream_window is documented as "the in-flight window between the pack writer and the socket" and "smaller costs wakeups", and it defaults to 256 KiB. poll_frame read into a hardcoded 16 KiB buffer. So the writer could get 256 KiB ahead and the reader still took sixteen bites to collect it, and the documented knob could not move the wakeup rate in either direction. Each of those wakeups is a futex pair: the spawn_blocking thread parks in SyncIoBridge, a runtime worker wakes it.

• Fix: the frame is the window. Measured, same corpus, 8 concurrent clones, four interleaved rounds:

−25% syscalls for +10% peak RSS, four rounds each, and the two sets do not overlap. Both halves are the same trade in opposite directions, which is why it is one number: an operator for whom the memory ceiling binds can now lower stream_window and actually get fewer bytes in flight.

No throughput claim is made. Against a box whose load moved 6.8 → 99 the effect on ops/sec was inside the noise, and it is shipped as a defect fixed and not as a win.

• A second bug fell out. tokio::io::duplex(0) accepts no byte ever, so stream_window: 0 — a public field with no floor — was not a small buffer but a hang: the transfer blocks on its first write and the client waits for a response that cannot begin. Streamed::pair now floors the window at one byte, which is absurdly slow and finishes, which is the right shape for a misconfiguration.

• Verify: a_body_hands_out_frames_the_size_of_its_window_not_of_a_constant asserts the frame boundary, not the collected bytes — every wrong frame size in the world collects to the same bytes, so a collect() there would have passed on the 16 KiB constant, on a 1-byte frame, and on anything a later edit put in. Seen RED by restoring 16 * 1024 (frame carried 16 384 of 262 144) and, for the floor, by removing the max(1) (the body hangs and the test times out).