docs/v2/git-sessions.md · nordisk/gunnar

Git sessions: bounds, limits and how to read a running server

No client, live or dead, can keep another client from git. Every wait on a client has a bound, every limit has a number, and an operator can read the running server without a shell. This page says which ones and where they act.

The dataflow of a git session

client ── front forward :2222 ── main :2222
            (idle 15 min,          │ accept: connection ceiling, SO_KEEPALIVE,
             512 connections)      │         TCP_USER_TIMEOUT, auth grace starts
                                   ▼
                          SSH handshake and auth        (off the async workers)
                                   ▼
                          exec git-upload-pack          (the decision: off the workers)
                                   ▼
                          admission slot   ── wait, then refused by name
                                   ▼
                          worker permit    ── wait, then refused by name
                                   ▼
                          fan worker ── blocking thread ── store, pack
                                   │      reads and writes bounded by the idle timer
                                   ▼
                          last words: exit status, close (bounded; else the socket is cut)
                                   ▼
                          worker, slot, store handle and status row given back

Smart HTTP takes the same steps from the slot down. It answers 429 where SSH refuses in the git protocol, and it does not wait for a slot or a worker.

Timeouts

Bound	Default	Flag	What it ends
Auth grace	30 s	`--auth-grace-secs`	a connection that has not authenticated, banner included
Exec grace	60 s		a signed-in session that asks for no command
Transfer idle	180 s	`--transfer-idle-secs`	a transfer waiting on its client with no byte either way
TCP stall	180 s	`--tcp-stall-secs`	a socket that takes nothing, also at a zero window (`TCP_USER_TIMEOUT`)
SSH keepalive	15 s × 3		a session whose peer answers nothing, 60 s
Inactivity	600 s		a connection that sends nothing, keepalive replies included
Last words, failed transfer	5 s		then the session is cut at its socket
Last words, clean transfer	the transfer idle bound		the client may still be taking the last window
Slot wait	2 s anonymous, 30 s with a key	`--admission-wait-ms`	then refused by name
Worker wait	30 s	`--worker-wait-secs`	then refused by name
Session age	24 h	`--session-max-secs`	any connection, however well it behaves

For the -secs flags of the bounds that can be turned off, 0 turns the bound off.

A real big clone is never ended by these. Every byte either way resets the transfer idle timer and the TCP stall timer, and the time the server spends building a pack is not timed at all. Only the 24 h session age does not move: a 100 GB clone at 2 MB/s takes 14 h.

Limits

standing workers per class = clamp(4 × cores, 16, 256)
worker ceiling per class   = max(standing, transfer ceiling)     --max-fan-workers
transfer ceiling           = min(16 × cores, descriptor room, 256), at least 8
per-identity floor         = 32, and its fair share once the server is contended
anonymous ceiling          = 5
SSH connections            = 1024                                --max-ssh-connections
HTTP connections           = 512
dead client released after ≤ max(keepalive 60 s, TCP stall 180 s)
stalled client released after ≤ transfer idle 180 s + last words 5 s

A class has its standing workers from boot. When all of them are held, the next transfer gets a spare thread, up to the ceiling, and a spare leaves after 30 s without work. The ceiling follows the transfer ceiling, so a transfer the gate admitted always has a worker and the gate alone decides who waits. git-lfs can run many channels on one slot, and each channel needs a worker, so only LFS can reach the worker wait.

Reading a running server

Both reads need gunnar-admin on the server, and they answer on the control plane, which does not use the git workers.

gunnar control git-state --at <addr> -i <key>
gunnar control logs --at <addr> -i <key> -n 500 --level warn --grep git.transfer

git-state (Admin.ReadGitState) lists every open transfer with door, state, age, idle time, bytes, principal, repository, service and peer, then the slots, then each class's workers. logs (Admin.ReadLogs) reads the last 4096 lines at INFO or louder.

The lines live in memory, so read them before you reboot.

What to look for:

You see	It means
`waiting-worker` rows, `busy` at the ceiling	workers are held; the oldest `serving` rows are who holds them
`serving` rows with a large idle time	transfers waiting on their clients; they end at the idle bound
`closing` rows	a client that takes nothing; it is cut at the last-words bound
`panicked` above 0	an operation panicked; its line is `git.pipeline.panicked`
`ssh.session.cut` in the log	the server ended a session; `why` says which bound
`git.transfer.cut` in the log	a transfer's last words were not taken