nordisk/skade

The read path — reasoning

Class: reasoning · Function: read-path. The descriptive what is in read-path-design.md.

Why three layers instead of one cache

Each layer removes a different, measured cost, so collapsing them would leave one of those costs on the warm path:

• L1 pointer mirror removes the begin_read() + open_table() cost of touching redb just to learn a table's current location — measured at ~16 µs. A table_exists or resolve_location that hit redb would pay it on every call.
• L0 metadata cache removes the real cost of load_table: the FileIO round-trip + JSON parse of the metadata file (~150 µs on NVMe, 10–50 ms on object storage), replacing it on a warm hit with an Arc clone (~100–300 ns).
• L1.5 handle cache removes a cost that surprised us: Table::builder().build() is dominated (~14 µs, ~99% of warm load_table) by iceberg-rust constructing a fresh per-Table ObjectCache on every call. disable_cache() does not avoid it (it still builds a capacity-0 moka cache, same ~14 µs) and there is no public way to inject a pre-built one. But Table is Clone, and a clone shares the Arc<ObjectCache> at ~100 ns — ~140× cheaper. So we build once per location and hand out clones.

Three separate costs → three separate layers, each keyed so it's coherence-free.

Why no TTL, no invalidation

The tempting simplification — one cache with a TTL — is wrong here, and worse than the layered design, because it would reintroduce a staleness window that the keys make impossible. Every layer is keyed by the immutable, content-addressed metadata location (or an immutable snapshot id). A commit that changes a table writes a new file at a new location; the old key is simply never requested again. So an entry can only be evicted, never stale. A TTL would add latency (re-read after expiry) and a correctness cliff (a window where you might serve old-then-new inconsistently) to buy nothing.

Why the pointer mirror can be lock-free (and why that's safe)

Correctness rests on single-process ownership: redb takes an exclusive file lock, so every mutation of the tables keyspace flows through this process. Each write path updates the mirror write-through after its redb commit while still holding the redb write lock — so mirror updates are serialised with each other and ordered after the durable commit. The mirror is therefore always complete and never stale; a key absent from it means the table genuinely does not exist. The redb fallback on a miss is a belt-and-braces safety net that never runs on the warm path. If skade were multi-process this reasoning collapses — which is exactly why multi-process is a non-goal (see reasoning.md).

Why foldhash, why imbl

• foldhash over SipHash — the mirror's keys are internal, content-addressed table identifiers, not attacker-controlled input. SipHash's HashDoS resistance buys nothing here but latency; foldhash is fast, well-distributed, and needs no AES-NI.
• imbl::HashMap (persistent) over a plain HashMap + lock — the mirror publishes a whole new immutable snapshot per mutation via ArcSwap, so readers never see a torn map. imbl's clone is O(1) (structural sharing) and the insert/remove only copies the touched path, so a write-through is cheap even with the mirror holding every table.