nordisk/skade

skade — complete overview

The real README: the full pitch, the competitive matrix, design, quickstart and every measured benchmark. The repo-root README.md is the slim face (title · mashup · pointers); this chapter is the heavy detail, and it is also folded into the manual docs/manual.pdf by nornir docs book.

Why skade — the whole Iceberg stack as one in-process Rust dependency, where the catalog is a single redb file rather than a JVM service — the rows marked NA are things the rivals have no concept for:

Full numbers below — generated by nornir bench run skade-katalog.

What it does

• skade — the data plane: Arrow RecordBatch in, SQL out; gatling multi-core ingest, zero-copy recast, identity-partitioned + compressed writes, V3 tables. (skade/)
• skade-katalog — the catalog: pure-Rust, redb-backed, single-file ACID iceberg::Catalog. No SQL, no C deps.

skade-katalog

crates.io docs.rs License

A pure-Rust iceberg::Catalog with cross-table transactions. A lock-free static search tree (Ragnar STree64) out front, continuously regenerated from a redb ACID backend. In-process — no network hop, no JVM, no JSON round-trip.

• table_exists — 2.0M ops/s, ~492× Nessie · ~677× Polaris (0.31 µs p50). Every core reads the front lock-free.
• Ingest saturates the box — 100% CPU, fully multicore. All cores encode Parquet in parallel; a gatling no-barrier pool keeps the writes overlapping, even over S3.
• ACID. Every mutation is one redb WriteTransaction. Atomic multi-table commits via RedbCatalog::atomic_release.

Full leaderboard (skade-katalog vs Nessie vs Polaris, every storage variant, all four capabilities) → Benchmarks.

table_exists — catalog reads (ops/sec, log scale · higher is better)

Best-of, at a glance

The tight head-to-head — skade-katalog vs Nessie vs Polaris, side by side so the gap is obvious (the rival's number is right there). Full per-storage detail (every NVMe/RAM/S3 variant, all four capabilities) is under Benchmarks.

Reads — table_exists (storage-free catalog RPC, higher ops/s better):

v0.1.3 · loki · 12 cores · 2026-07-12

Writes — data_pipe (create→ingest→scan, higher rows/s better):

v0.1.3 · loki · 12 cores · 2026-07-12

Status

The Iceberg Catalog trait is fully implemented and tested against iceberg = "0.9.1". One gap: schema evolution (the upstream Transaction actions aren't public until iceberg-rust 0.10). See Known shortcomings.

skade — the data-plane companion crate

This repo also ships skade ("winter queen" — Skaði): fast Iceberg table writing/reading + ergonomic DataFusion SQL on top of RedbCatalog. One directory = catalog (catalog.redb) + warehouse; Arrow RecordBatch in, SQL out (wh.sql("SELECT … FROM a JOIN b …")). It packages the writer stack, read-to-Arrow primitive, schema bridges (incl. the unsigned widen/unwiden reinterprets znippy needs), and the windowed-Parquet bulk-ingest helpers that were proven in bench/. Like bench/, it is a detached cargo workspace (skade/), so this crate's manifest and lockfile stay untouched. See skade/README.md.

Read & write paths

Every call resolves left-to-right through these layers, falling through only on a miss. Latest-state reads ride the L1 pointer mirror; time-travel (snapshot-id) reads ride the Ragnar STree64 index. redb is always the source of truth behind them — the in-memory layers are keyed by immutable, content-addressed identifiers, so they can only be evicted, never stale.

The same routing in full — including the time-travel, write, and redb-direct paths the diagram leaves out:

LATEST READS
  table_exists(id) ───────► L1 mirror ──hit──► true
                                └─miss─► redb `tables` ─────► bool

  resolve_metadata(id) ───► L1 mirror ──► loc ──► L0 cache ──hit──► Arc<TableMetadata>
       (skips Table::build)     └─miss─► redb         └─miss─► FileIO read + parse JSON

  load_table(id) ─────────► L1 mirror ──► loc ──► L1.5 cache ──hit──► Table  (~100 ns clone)
                                └─miss─► redb          └─miss─► L0 cache ──► Table::build() + insert

TIME-TRAVEL READS  (by snapshot_id: i64)
  load_table_at / resolve_metadata_at(id, sid)
        ──► Ragnar STree64 ──hit──► loc ──► L1.5 / L0 ──► Table / Arc<TableMetadata>
                 └─miss (sid above cutoff)─► redb `commits` live tail ──► loc ──► …

  resolve_many(id, [sid; N])
        ──► Ragnar batch probe (1 pipelined pass) ──► redb `commits` (1 txn, misses only) ──► L0 per loc

WRITES  (create · register · update · drop · rename)
  …──► FileIO write …/<uuid>.metadata.json ──► redb WriteTransaction { tables CAS · commits log · meta++ }
        ──commit──► L1 mirror write-through (insert / remove) ──► maybe rebuild Ragnar (background)

  update_table ──► group-commit: N concurrent commits coalesce into 1 redb txn / 1 fsync   ← commit-burst lever

REDB-DIRECT  (no cache layer)
  list_namespaces · get_namespace · namespace_exists · list_tables ──► redb read txn (range scan)
  create / update / drop_namespace ──────────────────────────────────► redb WriteTransaction

Benchmarks

All numbers below are auto-filled by nornir docs render from the latest nornir bench run (machine/cores/version in each header). Don't hand-edit inside the generated regions — re-run the bench instead. Reproduce with the harness in bench/ (cargo run --bin bench-containers -- up in bench/, then nornir bench run skade-katalog from workspace_skade/).

Catalog read RPC — table_exists (storage-free, runs on every backend)

table_exists is a pure catalog RPC (no object storage), the apples-to-apples comparable; Nessie/Polaris go through the same iceberg::Catalog REST client.

v0.1.3 · loki · 12 cores · 2026-07-12

Embedded read fast paths (nornir-only)

resolve_metadata skips Table::build() (the lock-free L1+L0 floor); load_table is the full Catalog trait path after the L1.5 handle cache.

v0.1.3 · loki · 12 cores · 2026-07-12

Data plane — single-writer / many-processor ingest

All cores encode Parquet; one writer streams + commits (data-pipe). Backend- agnostic and storage-bound (shared client-side Parquet + S3), so rows/s converge across backends — this lifts all boats, it is not a catalog lever. Nessie runs the same shared RustFS S3 warehouse. Scan verifies the round-trip. The nornir local-FS row runs two file destinations — _nvme (PCIe-4.0 NVMe) and _ram (/dev/shm tmpfs) — to expose the pure storage floor.

v0.1.3 · loki · 12 cores · 2026-07-12

Commit-bursty — the catalog lever (group-commit)

Many small concurrent metadata commits — where skade-katalog coalesces commits into one redb txn/fsync. This is where the embedded catalog genuinely pulls ahead of a REST/JVM catalog (throughput and tail latency).

v0.1.3 · loki · 12 cores · 2026-07-12

TPC-H — analytical SQL (all 22 queries via DataFusion)

The full 8-table TPC-H schema loaded into Iceberg tables, queried through DataFusion (iceberg-datafusion) — per-query latency for all 22 canonical queries over the catalog's scan/manifest path. Authentic tpchgen data; TPCH_SF sets the scale (small by default). Answer values aren't validated here (that needs SF=1) — this measures plan + scan + execute latency end-to-end.

v0.1.3 · loki · 12 cores · 2026-07-12

TPC-H large warehouse (opt-in, big iron)

A full-scale run: ingest the 8-table warehouse at a large scale factor across all cores (partitioned parallel ingest), then run all 22 queries over it — reporting build vs query time, ingest rate, and the slowest query. Runs by default at a moderate scale; TPCH_WAREHOUSE_SF tunes it (default 10 ≈ 60 M lineitem rows / a few min; 100-200 for a 10-60 min big-iron workout), TPCH_WAREHOUSE=0 disables it, TPCH_WAREHOUSE_PARTS sets ingest partitions (default = all cores).

v0.1.3 · loki · 12 cores · 2026-07-12

TPC-H across catalogs × storage (skade-katalog vs Nessie vs Polaris)

The same SF warehouse built and all 22 queries run over each (catalog × storage) combination through DataFusion — the analytical-SQL path compared apples-to-apples (not just the table_exists RPC). The matrix covers every combo that's possible: nornir on file-NVMe, file-RAM, and S3; Nessie S3-only (it rejects a local file warehouse); Polaris on file (its S3 credential vending 301s against the RustFS endpoint — see bench/src/containers.rs). The REST targets need their container + the shared RustFS S3 warehouse (bench-containers up all), else they skip. TPCH_COMPARE_SF sets the scale (default 1.0 ≈ 8.66M rows).

Parallel scan patch: stock iceberg-datafusion 0.9.1 scans a table as a single DataFusion partition (one serial Parquet-decode stream), so on a 32-core box only ~1 core works and queries are ~7-8× slower than they should be. The repo vendors a patched iceberg-datafusion (bench/vendor/, relative-path [patch] used by both bench/ and skade/) that exposes one partition per file, so DataFusion decodes files across all cores. SF100 FILE query dropped ~1660s → ~197s (8.4×). These numbers reflect that patch; published skade builds get stock 0.9.1 until the patch is upstreamed.

At SF100 (866M rows) both S3-backed paths (skade_s3, nessie_s3) fail with hyper: connection closed before message completed finishing the Parquet writer — an S3 write-robustness limit under sustained heavy PUT load (rust-s3 / RustFS), not a catalog issue; the FILE-backed paths complete cleanly.

v0.1.3 · loki · 12 cores · 2026-07-12

OSM GeoParquet ingest (data plane)

End-to-end ingest of authentic OSM data: a nodes.parquet produced by the katana-osm / osm2geoparquet converter (OSM .osm/.bz2/.pbf → GeoParquet) is read, its schema mapped to Iceberg, and ingested into a fresh RedbCatalog table via the single-writer / many-processor pipeline, then scanned back to verify. NVMe vs RAM destinations isolate storage cost. Point OSM_GEOPARQUET at the file (OSM_MAX_ROWS caps rows); the osm_ingest_* benchers record a failure if it's unset, like the container-backed targets.

v0.1.16 · oden · 32 cores · 2026-07-22

ZSTD decode — zstd-sys-rs vs the zstd crate

Aggregate (all-core) decompression throughput on a real OSM corpus (WKB geometry bytes from the converted Europe extract, ~parquet-page-sized frames), decoding the same frames two ways: the stock zstd crate (one-shot, allocates per frame) and zstd-sys-rs's zero-copy path (a reused ZSTD_DCtx + reused output buffer). Both link the same static libzstd 1.5.7, so this validates the bindings and the zero-copy API on real data rather than chasing a codec difference. Needs OSM_GEOPARQUET (ZSTD_CHUNK_KB sets frame size).

v0.1.16 · oden · 32 cores · 2026-07-22

Ours vs rivals — skade vs reference Iceberg vs raw Parquet (one dataset)

The data-plane head-to-head over ONE identical dataset and ONE storage root. Read throughput is meaningless beside a core count, so every read arm reports its own per-phase read_cores_busy and read_rows_per_sec_per_core, and skade's scan appears twice — single-core AND all-core — so its own single↔multi scaling is visible, not just skade-vs-rivals:

• skade (ours) — skade::Table::{ingest_pipelined,read}: all-core gatling encode + snapshot commit, and the all-core gatling scan (manifest parse, footer load, and Parquet decode fanned across every core on the ONE fork-join engine);

• skade_serial (ours, the single-core control) — the SAME Table::read scan over the SAME layout, run through skade::with_scan_serial: identical code, identical row order, but the decode dispatched on one core. skade ÷ skade_serial is skade's measured scan scaling on this box, the exact analogue of the two raw-Parquet arms below;

• iceberg — upstream iceberg-rust's stock writer stack (DataFileWriterBuilder

  • ParquetWriterBuilder + fast_append) and stock scan (table.scan()…to_arrow()), over the same RedbCatalog;

• parquet — the floor: raw ArrowWriter files, no catalog / no snapshots / no ACID, read back single-threaded with ParquetRecordBatchReaderBuilder;

• parquet_par — the SAME raw files read through the SAME gatling row-group fan-out skade uses. parquet_par − parquet is the fan-out contribution; skade − parquet_par is what the catalog/snapshot layer costs.

Held equal: the byte-identical deterministic dataset; the storage root and tempdir-per-pass discipline; the UNCOMPRESSED+dictionary codec; the 8192-row decode batch; the row-group count (the read fan-out's work units, equalised at OXR_ROWS/OXR_BATCH); and the read output contract (every arm materialises the full Vec<RecordBatch> Table::read returns). Not equal, and intrinsic to each writer: the file cadence (skade writes one file per input batch; the rivals one per commit) and write parallelism (skade's encode is all-core, the rivals single-writer) — which is why write and read core counts are reported separately and per phase (CLOCK_PROCESS_CPUTIME_ID), never one whole-pass number that charges skade's all-core encode against its scan. Correctness gate: every pass must scan back exactly the rows it wrote. OXR_ROWS/OXR_BATCH/OXR_COMMIT_EVERY/ OXR_ITERS tune it; OXR_RAM=1 roots on tmpfs.

Single↔multi, measured on oden (32c, oxr-default 1M rows / 20 units, UNCOMPRESSED), 2026-07-24:

skade's scan scales 4.6× single→all-core here (51.2M/1 core → 234.1M/9.5 cores) — but note the per-core figure falls 51.2M → 24.6M as cores are added, and pushing the unit count up (20 → 160 units) raises read_cores_busy 8.8 → 15.5 while total throughput drops slightly (222M → 185M, examples/oxr_scaling). That is the honest verdict for this dataset: the all-core scan of a cheap, UNCOMPRESSED 4-column shape is DRAM-bandwidth bound at ~9 busy cores, not fan-out bound — the extra cores stall on memory, they are not idle for want of work units. The fan-out is real (4.6× is real, and the manifest-parse + footer-load prologues are now fanned across gatling, not serial); the ceiling above ~9 cores is this box's memory subsystem, not the layout. When the decode is instead compute-bound — the scan_core_saturation test's ZSTD, high-entropy data — the identical scan keeps ~21 of 32 cores busy at a 13× single→all-core speedup, because there the cores have real per-byte work and are not waiting on DRAM. "Need enough data" means enough work units to reach the cores and enough per-unit work that the cores are compute-bound; give it both and it fills them.

v0.1.4 · oden · 32 cores · 2026-07-24 — measured (OXR_RAM tmpfs, OXR_ITERS=7). Hand-filled from the oxr bench run on this host (the gen pipeline persists to the nornir warehouse, which this workstation's harness is not wired to; the numbers are the harness's own JSON output, NORNIR_BENCH_ONLY=oxr NORNIR_MACHINE=oden). files ≠ equal by design (skade writes one file per input batch; the rivals one per commit); the row-group count is equalised at 20 — the read fan-out's work units.

Reading the table honestly: skade wins the WRITE (44.4M — 4.4× iceberg, 4.8× raw Parquet — its all-core gatling encode) and scales its READ 4.6× single→all-core (51.2M → 234.1M). oxr_parquet_par reads faster (393.7M at 12.7 cores) than skade (234.1M at 9.5): it is the SAME gatling row-group fan-out over raw files with no catalog and no snapshot, and it reads 3 physical files (20 row groups) where skade reads 20 files — so skade − parquet_par is the catalog/snapshot/per-file-footer cost, not a decode deficit. Every read number sits beside its own per-phase read_cores_busy, because 234.1M at 9.5 cores (24.6M/core) and 62.0M at 1 core (62.0M/core) are the SAME per-core decode — the difference is purely how many cores each engine puts on the job.

Quickstart

,no_run
use std::collections::HashMap;
use std::sync::Arc;

use iceberg::io::LocalFsStorageFactory;
use iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation};
use skade_katalog::RedbCatalogBuilder;

# async fn run() -> anyhow::Result<()> {
let catalog = RedbCatalogBuilder::default()
    .db_path("/var/lib/nornir/catalog.redb")
    .warehouse_location("file:///var/lib/nornir/warehouse")
    .with_storage_factory(Arc::new(LocalFsStorageFactory))
    .load("nornir", HashMap::new())
    .await?;

let ns = NamespaceIdent::new("bench".to_string());
catalog.create_namespace(&ns, HashMap::new()).await?;

let schema = Schema::builder()
    .with_schema_id(0)
    .with_fields(vec![
        NestedField::required(1, "run_id", Type::Primitive(PrimitiveType::String)).into(),
        NestedField::required(2, "ops_sec", Type::Primitive(PrimitiveType::Double)).into(),
    ])
    .build()?;

let table = catalog
    .create_table(
        &ns,
        TableCreation::builder()
            .name("bench_runs".to_string())
            .schema(schema)
            .build(),
    )
    .await?;
# Ok(()) }

Multi-table atomic commits

Iceberg's per-table Transaction::commit only gives single-table atomicity. When a release logically spans several tables (e.g. publishing bench_runs, dep_graph, and components from a single CI run), readers can otherwise observe a half-published state.

redb gives every write transaction global atomicity across all of its tables. RedbCatalog::atomic_release(commits) exploits this: each TableCommit is staged (metadata blobs written to object storage), then a single redb write transaction performs an optimistic-concurrency check on every base pointer and either flips them all or none.

       prepare metadata files       single redb txn
TableCommit ──┐                    ┌─ check base = current ┐
TableCommit ──┼─► fileio.write_to ─┤  …for every table…    ├─► commit
TableCommit ──┘                    └─ swap pointer         ┘

If iceberg-rust later exposes TableCommit construction publicly, the ergonomic story improves. Until then, atomic_release is most useful when you build commits via crates that have direct access to internal helpers (e.g. a nornir release driver).

Storage layout

Inside the redb file:

Where <ns_path> is the dot-joined namespace ident ("a.b.c"). One redb file may host multiple logical catalogs by reusing different <catalog> prefixes; in practice you'll point each catalog at its own file.

Known shortcomings (0.1.3)

• Schema evolution is not callable yet. iceberg-rust 0.9.1 seals the TransactionAction trait (pub(crate)), so downstream code can't construct AddSchema / SetCurrentSchema. The on-disk metadata already supports schema evolution — only the writer API is sealed — so it works here unchanged once upstream exposes it (0.10). These Transaction actions do work today: update_table_properties, fast_append, replace_sort_order, update_location, update_statistics, upgrade_format_version.

• No orphan-file garbage collection. A commit that fails after writing metadata blobs to object storage but before the redb pointer-swap commits leaves those blobs behind — harmless but unreferenced. Reap them with Iceberg's standard orphan-file procedures; there is no built-in sweeper.

License

Licensed under Apache License, Version 2.0, matching the upstream Apache Iceberg project.