nordisk/gunnar

Disks, caches and rendering

This chapter covers three things. It says which disks each Gunnar box has and what lives on each mount. It lists every cache with its key, its budget, its eviction rule and what survives a restart. It explains how a cache knows that its source has changed. It ends with the lessons of the 2026-11-06 incident and a table of the owner's decisions.

The chapter was checked against gunnar df519085 (master, 2026-11-06), gunnar-ui 1075bc0 (main), znippy 29ec553 (znippy-plugin-git 0.2.5), and the live view of gunnar.rs (gunnar control storage and gunnar control topology, read on 2026-11-06). Work that has not merged yet is marked in progress.

1. The boxes and their disks

A Gunnar estate has three boxes. The front answers the internet. The main holds the repositories. The twin follows the main.

 internet ──► front :443/:2222 ──WireGuard──► main :50051/:2222 ──replication──► twin
              one system disk                 system disk + 4 data disks       system disk + 1 data disk

The two areas on the main are two failure domains, not "fast" and "slow" (crates/gunnar-server/src/slow.rs:9-20):

• /data is a stripe. It is fast and has zero redundancy. If one member disk dies, all of /data is gone. The twin is the copy of the repositories.
• /serialdat is the boot disk's third partition. Its value is that it is another spindle and another failure domain. The copies that must survive the loss of a data disk live there.

korp-installer decides the layout at install time and writes it to /boot/korp/layout.conf. Gunnar reads dir.<name>.path from that file and writes exactly there. Gunnar adds no placement logic of its own (slow.rs:21-33).

2. What lives where

The installer lays some directories physically on /serialdat and makes /data/<name> a symlink to each ("slow-linked"). A write to /data/log therefore lands on /serialdat without Gunnar knowing. The rows below are from gunnar.rs's live layout config and from the code.

On the main

Hot and cold archives. The znippy engine keeps one archive per repository. Every push lands in it, and control storage counts it as hot. There is no cold tier today: the cold column is 0 with cold_measured: false, which means "not a reading", so the console shows N/A (control_plane.rs:5810-5818). On gunnar.rs, 47 repositories hold 2.1 GiB hot and 0 B cold.

The admin page's per-mount breakdown of /serialdat measures these known paths: metadata copy, crash record, audit trail, logs, grow journal, repository mirror and, when it is there, the read cache. Whatever the mount uses beyond them is other (control_plane.rs:5948-5975, capacity.rs:542-600).

On the front

2.1 Is gc/repack scratch on /serialdat? No.

The answer on 2026-11-06 is no. /data/gc exists and links to /serialdat/gc, but nothing writes there.

• Gunnar's placement table marks the row wired: false (slow.rs:201-207), and gc_dir() says "nothing writes here yet" (slow.rs:455-458). The live view on gunnar.rs prints git gc / repack scratch /data/gc NOT WIRED.

• The znippy engine stages every gc file beside the archive, inside the repository directory on /data:

  • gc.rs:437 hard-links the archive to <archive>.gc-<pid>-<ns> (unique_sibling, gc.rs:399-407);
  • compact_archive stages <archive>.compact-<pid>-<ns> and renames it over the work name (znippy-common/src/meta_sink_append.rs:732-748);
  • an index fold writes objects.index.tmp beside objects.index and renames it (read_stack.rs:1095-1103).

• This placement is deliberate. A hard link and an atomic rename(2) work only inside one filesystem. Staging on /serialdat would turn the rename into a whole-archive copy across disks. Moving it is a storage-engine change, and it needs the owner's go.

Push quarantine is not wired for the same reason. The incoming-*.pack spool is created in the repository's own directory (storage.rs:1161-1164), so the final rename stays on one filesystem and costs nothing.

3. Every cache

 browser ──HTTP cache──► front ──────────────────────────────► main ──────────────────────────► disk
          (immutable     │ public-site pages   (memory)          │ handle cache  (memory)        /data/repos  archives
           hashed        │ prerendered parts   (memory)          │   └ redb page cache per handle
           assets)       │ git object cache    (memory)          │ explode read cache ──────────► /serialdat/explode
                         │ README render cache (state.redb)      │
                         │ render store (in progress, disk)      │

3.1 The explode read cache

What it is. A read of a git object that sits deep in a delta chain costs several inflates. The cache keeps the inflated object so that the next read is one lookup. Every row can be rebuilt from the pack, so the cache is never data (exploded_cache.rs:1-16).

Two pools. The eager pool objects.exploded holds the commits and tags that the indexer writes at push time. It is never evicted, because reach and the graph walks need it. The lazy pool objects.exploded.c1, .c2, and so on is filled on the first read of an object. An object is cached only when it is worth it: delta depth over 2 (or an external base) and at most 1 MiB inflated (exploded_cache.rs:202-221).

Where. Since gunnar df519085 (release b52684ea, znippy-plugin-git 0.2.5), the cache lives at <serialdat.mountpoint>/explode when the layout config names a /serialdat that exists. Without one (a dev or KVM box), it stays beside each repository as before (explode_home.rs:1-34). The store keeps only its key file objects.exploded.key on /data.

Budget. Let S be the size of the home's filesystem, and let f = 50 %:

budget per repository  B = min(f · S, 8 GiB)          (floor 64 MiB)
line for all of them   G = f · S
disk line              trips when used(S) ≥ 80 %, releases when used(S) < 75 %

Without a home, the old rule applies on /data: f = 10 % (exploded_cache.rs:245-247, exploded_home.rs:56).

Eviction. The lazy pool is a row of segment files in first-read order. Eviction is one unlink of the oldest segment. An object that is still hot misses once, is rebuilt and is written into the newest segment. That makes it FIFO over segments with re-entry on a miss, which behaves like an LRU for a working set (exploded_cache.rs:39-62). Past the disk line the oldest segments go whatever the budget says, and the eager pool is never touched (exploded_cache.rs:270-291).

What empties it. A repo gc unlinks the lazy cache (gc_queue.rs:3-8). A purge unlinks the repository's directory under the home, and every boot sweeps the caches of repositories that are gone. A repository deleted and created again under the same name gets a new key, so its old cache is never read (exploded_home.rs:27-33).

Restarts. /serialdat is P3, which korp-installer keeps on every keep boot, and a live update does not touch it. A restart adopts the segments that the last process wrote. The in-memory oid index of the lazy pool is rebuilt at the first miss.

The move. The first open under the new release moves the old cache from /data into the key directory. Across two filesystems that is a copy (staged as .part, then renamed) and an unlink. A copy that fails drops the old cache, which is re-derivable.

Worked example (gunnar.rs main).

S = 47.9 GiB                      f · S = 23.9 GiB
B = min(23.9 GiB, 8 GiB) = 8 GiB per repository
G = 23.9 GiB for all 47 repositories together
disk line: trips at 0.80 · 47.9 = 38.3 GiB used, releases under 0.75 · 47.9 = 35.9 GiB
in use on 2026-11-06: 1.8 GiB

control storage prints the budget as "376.0 GiB". That figure is the sum of the per-repository caps (47 × 8 GiB). It is not a real limit. The line G binds first, at 23.9 GiB.

3.2 The store handle cache

What it is. An open repository handle holds the folded index, the commit graph and a redb page cache. Opening a cold repository took 27 s once on gunnar.rs. The cache keeps handles open between requests.

Price of one handle (budget.rs:205-222, measured 2026-08-31):

price(h) = 0.5 MiB + objects(h) · 768 B + min(tail_bytes(h), 64 MiB)

objects is the repository's own object count. tail_bytes is the size of its objects.tail database, because the redb page cache never holds more than the file.

Budget.

share    = MemAvailable at boot · 20 %
capacity = clamp((nofile − 256) / 3, 8, 256)          handles, by descriptors

Eviction rule (stores.rs:496-524). Sort the idle handles by last use. Drop the oldest while the count is over capacity, or while the summed price is over share and more than 8 handles are open. A handle in use is never dropped. When MemAvailable falls under 25 % of MemTotal, the bound is halved and the least recently used handles go (stores.rs:2425-2452). It never shrinks below 8.

Restarts. The cache is memory only. Every restart, keep boot or version bump starts it empty.

Worked example. On the 8 GB main, suppose MemAvailable is 7 GiB at boot.

share = 7 GiB · 0.20 = 1.4 GiB
nordisk/gunnar-sized repository: 12 849 objects, objects.tail 4 MiB
price = 0.5 + 12 849 · 768 B (9.4 MiB) + 4 = 13.9 MiB
1.4 GiB / 13.9 MiB ≈ 103 such handles fit; the descriptor capacity is 256
alice/linux-mirror: 13 948 244 objects → price ≈ 10.7 GB → over the share alone;
                    the floor of 8 still keeps it open

Before gunnar df519085, every handle was priced as a 100 000-object repository (137 MiB), so the same box kept about 11 handles. The 30 public repositories could not all stay open.

3.3 The warm-up of the public repositories

After a boot, the jera job warm-public-repos waits for the boot's disk test (readiness::is_ready). It then opens every public repository, one at a time, sorted by path, with a 2 s pause between opens (warm.rs:1-70). The handle cache's ledger decides what stays open. The first visitor of a public repository, often a crawler that read llms.txt, then pays no open.

3.4 The README render cache and the public-site prerender

README render cache (gunnar-ui-server/src/readme/cache.rs). It holds the typeset SVG of a README. The key is (namespace, name, blob oid, path, shape, renderer). The budget is 256 entries or 64 MiB, whichever binds first. A typeset SVG costs about 105 bytes per character of README, so a 3 KB README is about 350 KB. Eviction is in insertion order, because a true LRU would need a redb write on every hit. The cache lives in the front's state.redb, so it survives a front restart.

Prerendered page parts (gunnar-public-site/src/shell.rs:87). The front keeps the rendered parts of /<ns>/<repo>, /commits and /issues in memory: at most 4096 parts, 64 MiB, 2 MiB a part. A push or issue event drops the repository's parts and renders the three pages again. Parts older than 1 h expire at every refresh.

Public-site built pages (gunnar-public-site/src/lib.rs). llms.txt, the sitemap, robots.txt and the per-repository readme.md, tree.txt and issues.md are built in memory. The first build starts at the first request, then runs hourly, and at once on a push event. A refresh that fails or comes back empty is tried again after 10 s, doubling up to 5 min (lib.rs:156-162, RETRY_FIRST, RETRY_MAX).

The render store on the front (in progress, lane grender). The plan moves both renders to an on-disk store in the front's typeset dir:

budget      = 75 % of the front's filesystem
safety line = trips at 90 % used, releases at 85 % used (oldest entries go first)
key         = git blob oid (files and READMEs) or the commit oid a ref points to (pages)
              + the renderer version

Worked example: the front has one 40 GB disk, so the budget is about 30 GB. Eviction past the line starts at 36 GB used and stops under 34 GB. There is one copy only: the store replaces the in-memory and state.redb copies, it does not sit beside them.

3.5 HTTP caching

A hashed name changes when its content changes, so the browser can keep it for a year. A name that points at a moving ref gets five minutes.

4. How a cache knows the source changed

Every cache that holds git content is keyed by a git object hash. A git object never changes under its hash.

new byte in a file  ⇒  new blob oid  ⇒  new key  ⇒  a miss, then one fresh render
new commit on a ref ⇒  the ref points at a new commit oid  ⇒  new page key
old key             ⇒  still right for the old content; it ages out by eviction

• Files and READMEs are keyed by the blob oid. A repository with a thousand commits and one unchanged README has one entry. A reader on an old revision gets the warm answer.
• Pages are keyed by the commit oid that the ref points to (render store, in progress). Today the prerendered parts are keyed by path and are dropped by the push event instead.
• The explode cache is keyed per store, and each row by its object oid. A recreated repository gets a new store key.
• Nothing is ever invalidated for being stale, because nothing can be. Only a renderer version bump clears everything: the version is part of the key (readme/cache.rs:93, RENDERER = "docbook-<version>+pictures2"), so after a gunnar-ui release that bumps docbook, every key is new and each README is typeset again once.

5. Lessons of 2026-11-06

Sources: ~/plan/GUNNAR-RS-ROOT-CAUSE-2026-11-06.md and ~/plan/GUNNAR-PAGES-FABLE-2026-11-06.md.

• The seccomp filter killed pid 1. The approved TRAP profile lacked restart_syscall (nr 219). The kernel issues that call itself, when an io_uring notification interrupts a thread in a timed futex or sleep. The filter answered SIGSYS, Gunnar exited, pid 1 died and the box rebooted (panic=10) 1 to 5 minutes after boot. The kernel-implied base set is now enforced in korp-installer (tunnr main 375e31e7) and in Heimdall 0.4.2. Gunnar ships with no filter (gunnar 14fefa05, release 8275d4d6) until the owner's KVM-lifecycle rule passes.
• The front's kept bundle. The front held the bundle that re-admits the main only in process memory. A front restart lost it, and the front could only refill it from the main. The front now reads the main's slot first and then the twin's (gunnar-front/src/recharge.rs:242, held_or_slots, gunnar 7bb4abb0).
• Re-admission after a keep boot. A keep boot clears the main's pairing, and re-admission happened only inside an update step that could fail first. monetize 96783cd3 resumes a stopped update and retries 408, 429 and 5xx. gunnar 9fd6209e re-admits a box at its install listener even when the shop call fails.
• The 4-worker stall. Every control-plane call ran on a runtime of (cores / 4).clamp(4, 8) workers, 4 on the main. Four cold opens held all of them, and every other call queued behind them. Each call now runs on its own thread of a blocking pool of 128 (serve.rs:544, CONTROL_CALL_THREADS). A WARN line behind an INFO flood waits at most 10 ms. The :2222 dial window went from 60 s to 5 s (gunnar-front/src/forward.rs:37). gunnar-ui gives console calls a deadline by class (sign-in 5 s, 1075bc0) and asks a read that met a silent Gunnar once more after 1 s (c669e27).
• The public-site refresh retry. A failed refresh used to leave the public site empty for an hour. It is now tried again after 10 s, doubling up to 5 min.

6. Decisions