nordisk/edda

Upstream report, FILED: Stepwise spawns two threads even at thread_limit: Some(1)

FILED 2026-08-05: https://github.com/GitoxideLabs/gitoxide/issues/2881 (as under the Ignalina account — see .nornir/upstream-gix-base-size.md for the precedent and the account rule).

Applied locally 2026-08-05 in vendor/gix-features, where gunnar already carries the watcher-sleep fix — see "What gunnar did" at the bottom, and vendor/gix-features/PATCH.md fix 2. The report below is unchanged and is still what should be filed.

Found 2026-08-04 while measuring gunnar's serve path. gunnar is not blocked on it: the cost is ~64 µs of process CPU per request on oden, which is 0.08% of a 12 186-object clone and ~7% of the smallest real fetch in gunnar's 19-repository corpus. It is filed because the fix costs gitoxide nothing — the serial implementation is already in the tree.

Title

parallel::reduce::Stepwise spawns threads unconditionally; thread_limit: Some(1) does not make it serial

Body

gix_features::parallel::in_parallel_if takes care to stay on the calling thread when one thread is all that was asked for:

// gix-features/src/parallel/mod.rs
if num_threads(thread_limit) > 1 && condition() {
    in_parallel(input, thread_limit, new_thread_state, consume, reducer)
} else {
    serial::in_parallel(input, thread_limit, new_thread_state, consume, reducer)
}

parallel::reduce::Stepwise::new — one module over, and the other half of the same pipeline — does not:

// gix-features/src/parallel/reduce.rs, mod stepped, #[cfg(feature = "parallel")]
let num_threads = num_threads(thread_limit);
for thread_id in 0..num_threads {
    let handle = std::thread::spawn({ … });   // ← at least one, always
    threads.push(handle);
}
threads.push(std::thread::spawn(move || {     // ← and the input feeder, always
    for item in input { if send_input.send(item).is_err() { break; } }
}));

So thread_limit: Some(1) yields two spawned threads, not zero: one consumer plus the input-feeder. MEASURED with strace -f -c -e trace=clone3 over one gix_pack::data::output::entry::iter_from_counts call at Options { thread_limit: Some(1), .. }: exactly 2 clone3 per call, on both a 12 186-object and a 69-object request.

An entire #[cfg(not(feature = "parallel"))] mod stepped already sits in the same file and is a lazy, in-thread Stepwise with the identical public surface — new, finalize, Iterator, Finalize.

Why it matters to a caller

A server that serves one request per connection and deliberately holds thread_limit: Some(1) — because it has no shared pool and per-request fan-out at forge concurrency is what it is avoiding — pays two thread creations and two joins per request, plus two thread stacks live for the length of the pack write, for a pipeline it asked to be serial. in_parallel_if's existing behaviour is a promise that Some(1) means "do not spawn", and Stepwise is the one place in the pipeline that does not keep it.

Suggested fix

Make Stepwise an enum over the two implementations and pick at runtime, the way in_parallel_if already picks:

pub enum Stepwise<Reduce: super::Reduce> {
    Threaded(threaded::Stepwise<Reduce>),
    Serial(serial::Stepwise<…>),
}

impl<Reduce: super::Reduce> Stepwise<Reduce> {
    pub fn new<…>(input: InputIter, thread_limit: Option<usize>, …) -> Self {
        if num_threads(thread_limit) > 1 {
            Stepwise::Threaded(threaded::Stepwise::new(input, thread_limit, …))
        } else {
            Stepwise::Serial(serial::Stepwise::new(input, thread_limit, …))
        }
    }
}

Iterator::next and Finalize::finalize forward. Both variants are in the file today; nothing new is written and no bound changes, because the serial variant's bounds (Iterator<Item = I>, Fn(I, &mut S) -> O) are a subset of the threaded one's.

The one visible difference is when the work happens: the threaded variant starts computing at new() and the serial one at the first next(). For Stepwise, whose whole documented purpose is step-wise consumption, that is the intended semantics either way — and it is already the observable behaviour of any build without the parallel feature.

Environment

• gix-features 0.49 (parallel on), gix-pack 0.73
• observed through gix_pack::data::output::entry::iter_from_counts, the only caller of Stepwise in gitoxide
• Linux 6.x, strace -f -c -e trace=clone3

What gunnar did — TAKEN 2026-08-05, in the crate it already vendors

The two threads are gone. The fix above is applied in vendor/gix-features/src/parallel/reduce.rs; see vendor/gix-features/PATCH.md fix 2 for the diff and its measurements.

This entry previously said "Nothing. The two threads stay.", on the reasoning that the only alternative was to stop calling iter_from_counts and re-implement ~90 lines of gix's counts sort, counts_range_by_pack_id index and thin-base resolution — which must reproduce gix's ordering exactly or the wire bytes move, in exchange for 64 µs. That reasoning was sound and its premise was wrong. Stepwise is not in gix-pack; it is in gix-features, which gunnar has vendored and patched through [patch.crates-io] since the watcher-sleep fix. The change is ~40 lines in a file already in the tree, and no reimplementation of anything.

Two ways of reading that are worth keeping:

• "Upstream" was never the same word as "unreachable". The report was filed under a heading that assumed a gitoxide release was the only delivery vehicle, and the vendored copy sitting one directory away went unconsidered for a day. Whenever an entry says a fix is upstream, check the patch table first.
• The 64 µs was the wrong number to weigh it against. Thread creation is what 64 µs measures, and it is genuinely negligible. What the two threads cost is that every chunk of pack entries crossed two channels, so every served object's compressed_data was allocated and memcpy'd on one core and read and freed on another. Which is a much larger claim, and — MEASURED, see PATCH.md — is not worth anything either: process CPU per request is flat and the 32-client wall did not move outside its spread.

What it is worth is written down rather than inflated: P-004's "the serving tier stays serial" is now true of the code and not only of the intent, and every per-thread instrument in the repository stopped lying — stage-probe's attributed_frac on a 12 186-object clone went from 0.097–0.356 to 0.982–1.000. bench/examples/packwrite-probe.rs exists only because that attribution was partial; its own header records the 0.545 it saw.

crates/gunnar-pack/tests/serve_runs_inline.rs is the guard, and it asserts the property rather than a thread count: every entry_by_location a served pack makes must happen on the thread that asked for the pack. Seen red by restoring upstream reduce.rs — "72 of 73 pack entries were produced off the calling thread".

CORRECTION 2026-08-30 (lane AW) — edda has not carried the patch since 2026-08-23

Two things above are no longer true of this repository, and one number was wrong.

The number: the red is 72 of 73, not 72 of 72. The 73rd is the 12-byte pack header PackedObjects::objects_in_pack reads through the same seam, and it is the caller's.

The situation: edda dropped its [patch.crates-io] in cbe0fb8 (2026-08-23) — a path patch this repository does not produce reads as a foreign fork to the release gate and held gunnar-pack/gunnar-delta off crates.io, and they held gunnar-client and dwarves. The fork was not lost: it is gunnar/vendor/gix-features, still fix 2 alone on 0.49.1, still reached through gunnar's own two patch tables and still guarded by gunnar/xtask/src/vendor_patch.rs. What was lost is edda's access to it, which left this test asserting a property this workspace cannot supply and red for seven days — a guard that can no longer report the thing it exists for.

The guard now discovers the pipeline shape through a different door (it drives a bare Stepwise with thread_limit: Some(1) and one item) and asserts the exact shape of whichever pipeline is present, plus one unconditional claim that holds in both: Some(1) bounds the work to at most one thread besides the caller, which is the P-004 claim that survives the fork being absent.

And it quantifies the fork for the first time in something other than CPU. MEASURED oden 2026-08-30, same binary, same 72-object fixture, the only variable being the patch table:

gix clones the consume closure once per worker and PackedObjects::clone opens a fresh gix-odb handle each time, so the two spawned threads cost one extra open, mapped copy of the repository's packs per served request. That is an fd and a mapping on a box that announces transfers 64, and it is a better argument for fix 2 than the ~64 µs of thread creation ever was.

P-020's thread_limit: Some(1) does more than it was credited with, and this is worth writing down because the opposite was believed on 2026-08-04: the resolve phase (in_parallel_if, which resolves every count to a pack location) is already inline at Some(1), because num_threads(Some(1)) is not > 1. MEASURED: the whole of P-025's inflate saving landed on the calling thread's stage tally (pack_write exclusive CPU 61.84 → 16.73 ms on a 12 186-object clone), which it could not have done had that phase been on a worker.