nordisk/skade

skade-katalog — the TRI-MODAL catalog (embedded · REST · gRPC)

Design (chosen path only; rationale/alternatives live in reasoning.md). skade-katalog is an iceberg::Catalog over a single-file redb. Today it ships embedded (in-process, the hot path) and has a complete but bench-filed REST front. This design promotes it to a first-class tri-modal catalog: the SAME RedbCatalog fronted three ways — embedded (link + call), REST (Iceberg OpenAPI, ecosystem interop), gRPC (nordisk-native, the fast wire nobody else ships).

f0 — the whole idea

RedbCatalog implements the iceberg::Catalog trait and already owns the hard machinery: create/load/drop/list, namespaces, and commit_table with optimistic group-commit + conflict detection (CatalogCommitConflicts). Because the trait is the single choke-point, every serving mode is a thin front over one RedbCatalog:

                 ┌────────────── embedded (in-proc, no hop) ── co-located hot paths
   RedbCatalog ──┼────────────── REST  (axum, Iceberg OpenAPI) ── Spark/PyIceberg/DuckDB/Trino
   (redb, ACID,  └────────────── gRPC  (tonic, native protobuf) ── nornir/dwarves/korp/modgunn
    single-writer)

The serving process owns the redb write-lock and serializes every commit; remote clients are thin and lock-free. Embedded callers co-located with that process keep bypassing the network entirely. This is what makes "the ONE live catalog" real and retires the whole single-writer-lock / modgunn-cycle thread (see PLAN "CATALOG DOCTRINE").

Crate layout (additive — embedded path untouched)

• skade-katalog (exists) — the core RedbCatalog. No change to its embedded API.
• skade-katalog-rest (NEW) — promote bench/src/rest_shim.rs (already a full, tested Iceberg-REST front: /v1/config, namespaces CRUD, tables create/load/list/head/drop/commit, proven by bench/tests/rest_commit.rs against a stock iceberg-rust RestCatalog) out of bench/ into a shipped library crate.
• skade-katalog-grpc (NEW) — a tonic service mirroring the SAME handlers over protobuf (proto sketch below). One-to-one with the REST endpoints.
• skade-katalog-serve (NEW bin) — skade-katalog serve [--rest ADDR] [--grpc ADDR] --warehouse ROOT. Opens ONE RedbCatalog (holds the write-lock), runs REST and/or gRPC over it, serializes commits. Pure-Rust, zero-shell (L9).

gRPC proto sketch (skade.katalog.v1)

service Catalog {
  rpc Config          (ConfigRequest)        returns (ConfigResponse);
  rpc ListNamespaces  (ListNamespacesRequest)returns (NamespaceList);
  rpc CreateNamespace (CreateNamespaceReq)   returns (Namespace);
  rpc NamespaceExists (NamespaceIdentMsg)    returns (Exists);
  rpc GetNamespace    (NamespaceIdentMsg)    returns (Namespace);
  rpc ListTables      (NamespaceIdentMsg)    returns (TableIdentList);
  rpc CreateTable     (CreateTableReq)       returns (LoadTableResult);
  rpc LoadTable       (TableIdentMsg)        returns (LoadTableResult);
  rpc TableExists     (TableIdentMsg)        returns (Exists);
  rpc DropTable       (TableIdentMsg)        returns (Empty);
  rpc CommitTable     (CommitTableReq)       returns (CommitTableResult);   // requirements + updates
}

Table metadata rides as the on-disk canonical Iceberg metadata JSON bytes (same splice trick the REST shim uses — iceberg's TableMetadata isn't Serialize, but the file IS the wire form). CommitTableReq carries repeated TableRequirement + repeated TableUpdate, handed straight to RedbCatalog::commit_table (mirrors the REST path).

Why gRPC should beat REST (and where all three land)

gRPC over REST for a catalog: binary protobuf (no JSON encode/parse per call), HTTP/2 multiplexing + persistent connections (no per-call TCP/TLS setup), typed/codegen clients, smaller frames. For the tiny-payload, high-QPS catalog calls (table_exists, load_table, resolve_metadata) that overhead IS the cost, so gRPC should show materially lower p50/p99 than REST. Expected ordering:

   embedded   <<   gRPC   <   REST        (latency; embedded = no hop at all)
   2.87M ops/s     ~?           ~?        table_exists — gRPC/REST arms are what we're adding

Embedded stays untouchably fastest (no serialization, no socket). gRPC is the fast wire. REST is the interop wire. The bench will fill the ~? with real Loki numbers.

Bench plan — the matrix you asked for

Skade arms (self, 3 modes): skade_katalog_embedded, skade_katalog_rest, skade_katalog_grpc — identical scenario code, three fronts over the same catalog.

Opponents (columns):

Two capability rows the matrix must surface (extends bench/src/capabilities.rs):

• Embedded single-file ACID catalog — skade ✅; Nessie/Polaris/Hive/Glue ✗ (already encoded).
• Native gRPC catalog — skade ✅; everyone else ✗ (novel; no mainstream Iceberg catalog ships a gRPC surface — Nessie had gRPC early and dropped it for REST).

Add the Iceberg file-based (Hadoop/filesystem) catalog as the serverless opponent so the embedded column has a fair no-service rival (Nessie/Polaris can't do embedded at all → those cells stay NA, which is the point).

modgunn / consumers (ties back to CATALOG DOCTRINE)

Once served: modgunn stops embedding and becomes a generic Iceberg REST client (iceberg-rust RestCatalog) — no nornir/dwarves dep, no cycle, no lock fight. dwarves / korp / funnel likewise become clients (REST for foreign, gRPC for our Rust mesh). nornir's Warehouse{Tables,Scan} gRPC stays as its read-only viz peephole (a subset view), distinct from the full catalog service. Ownership: skade owns the catalog AND its serving; every other repo is a client.

Definition of done (L3/L4/L7/L8)

• Embedded API unchanged (L2, no regression); REST promoted out of bench; gRPC added.
• skade-katalog serve runs REST and/or gRPC over one lock-owning RedbCatalog.
• Bench arms for embedded/REST/gRPC + the file-based opponent + the two capability rows, HEAVY, host-stamped, plain numbers; README matrix regenerates and CRUSHES (more columns).
• Each front robot/asserted: a stock RestCatalog round-trips REST; a generated gRPC client round-trips gRPC; RED-when-broken commit-conflict test on each wire.
• design (this) + guide + idiot-guide; thin book.pdf; pushed SSH main.