Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/skade

.nornir/tutorial.md

skade — tutorial

A hands-on walk through skade, from an empty directory to SQL over time-travel snapshots. Assumes you know Rust and roughly what a columnar table is. If words like catalog, Iceberg or snapshot are new, read dummy-guide.md first, then come back.

There are two layers, and you pick by how much control you want:

1. Install

The data plane bundles and re-exports everything (arrow, parquet, iceberg, datafusion), so one dependency is enough:

[dependencies]
skade = "0.4"                       # data plane (Arrow in, SQL out)
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
anyhow = "1"

Want SQL? It's behind a feature:

skade = { version = "0.4", features = ["sql"] }

Only need the catalog (no Arrow/SQL/datafusion weight)? Depend on the lean crate directly:

skade-katalog = "0.1"
iceberg = "0.9.1"

Everything is async (tokio).

2. Your first warehouse (data plane)

One directory becomes the whole lakehouse — a redb catalog file plus the Parquet data. Open it, create a table from an Iceberg schema, append Arrow batches:

,no_run
use std::sync::Arc;
use skade::iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use skade::arrow_array::{RecordBatch, Int64Array, Float64Array};
use skade::arrow_schema::{Field, Schema as ArrowSchema, DataType};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. Open (creates ./lake/catalog.redb + ./lake/<ns>/<table>/… on demand).
    let wh = skade::open("./lake").await?;

    // 2. Describe the table with an Iceberg schema (field ids are required).
    let schema = Schema::builder()
        .with_schema_id(0)
        .with_fields(vec![
            NestedField::required(1, "id",    Type::Primitive(PrimitiveType::Long)).into(),
            NestedField::required(2, "score", Type::Primitive(PrimitiveType::Double)).into(),
        ])
        .build()?;

    // 3. Create it. Tables live in the default namespace `main` unless you
    //    dot-qualify the name (e.g. "logs.events" → namespace `logs`).
    let mut events = wh.create_table("events", &schema).await?;

    // 4. Append Arrow data. Each append is one Iceberg commit (a new snapshot).
    let arrow = Arc::new(ArrowSchema::new(vec![
        Field::new("id", DataType::Int64, false),
        Field::new("score", DataType::Float64, false),
    ]));
    let batch = RecordBatch::try_new(arrow, vec![
        Arc::new(Int64Array::from(vec![1, 2, 3])),
        Arc::new(Float64Array::from(vec![0.5, 1.0, 1.5])),
    ])?;
    events.append(&[batch]).await?;

    // 5. Read it straight back as Arrow.
    let batches = events.read().await?;
    let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
    assert_eq!(rows, 3);
    Ok(())
}

That's a complete, durable, ACID Iceberg table in one file tree. Copy ./lake and you've copied the whole lakehouse.

3. Query with SQL (features = ["sql"])

skade speaks DataFusion. Query a single table, or the whole warehouse with joins across tables and namespaces:

,no_run
# async fn run(wh: &skade::Warehouse, mut events: skade::Table) -> anyhow::Result<()> {
// Table-scoped:
let res = events.sql("SELECT count(*) AS n, sum(id) AS s FROM events WHERE id > 1").await?;

// Warehouse-scoped — tables are addressable bare or fully-qualified
// (`skade.<namespace>.<table>`, default namespace is `main`):
let res = wh.sql("SELECT count(*) AS n FROM events").await?;
let res = wh.sql("SELECT count(*) AS n FROM skade.main.events").await?;

// Joins across tables / namespaces:
let res = wh.sql("SELECT count(*) FROM a JOIN b ON a.id = b.id").await?;
let res = wh.sql("SELECT * FROM skade.logs.c").await?;
# Ok(()) }

Results come back as Arrow RecordBatches.

4. Reopen, and open-or-create

Nothing is in memory — reopen the directory and the tables are there:

,no_run
# async fn run() -> anyhow::Result<()> {
let wh = skade::open("./lake").await?;
let events = wh.table("events").await?;                 // open existing
let events = wh.table_or_create("events", &schema()).await?;  // idempotent
# Ok(()) }
# fn schema() -> skade::iceberg::spec::Schema { todo!() }

5. Bulk ingest (when append is too slow)

For loading lots of data, the data plane has higher-throughput paths than append:

These are backend-agnostic and storage-bound — they lift all boats; they are not a catalog lever (see §7). See skade/ANALYSIS.md for measured numbers.

6. The catalog directly (skade-katalog)

Drop to the catalog when you need raw Iceberg control or a non-local FileIO. Build a RedbCatalog; it implements iceberg::Catalog in full:

,no_run
use std::collections::HashMap;
use std::sync::Arc;
use iceberg::io::LocalFsStorageFactory;
use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation};
use skade_katalog::RedbCatalogBuilder;

# async fn run(schema: iceberg::spec::Schema) -> anyhow::Result<()> {
let catalog = RedbCatalogBuilder::default()
    .db_path("/var/lib/app/catalog.redb")
    .warehouse_location("file:///var/lib/app/warehouse")
    .with_storage_factory(Arc::new(LocalFsStorageFactory))
    .load("app", HashMap::new())
    .await?;

let ns = NamespaceIdent::new("bench".to_string());
catalog.create_namespace(&ns, HashMap::new()).await?;

let table = catalog.create_table(
    &ns,
    TableCreation::builder().name("bench_runs".to_string()).schema(schema).build(),
).await?;
# Ok(()) }

Swap LocalFsStorageFactory for an S3/object-store factory to put data on remote storage while the catalog pointer file stays local and ACID.

Time-travel reads

The catalog has inherent (non-trait) methods for reading historical snapshots, routed through the Ragnar STree64 index (see design.md):

,no_run
# async fn run(catalog: skade_katalog::RedbCatalog, id: &iceberg::TableIdent, sid: i64) -> anyhow::Result<()> {
let meta  = catalog.resolve_metadata_at(id, sid).await?;   // one snapshot
let table = catalog.load_table_at(id, sid).await?;
let many  = catalog.resolve_many(id, &[sid, sid - 1]).await?; // one pipelined batch probe
# Ok(()) }

Multi-table atomic commits

Iceberg's per-table Transaction::commit is single-table. When a logical release spans several tables and readers must never see a half-published state, stage the commits and flip every pointer in one redb write transaction:

,no_run
# async fn run(catalog: skade_katalog::RedbCatalog, commits: Vec<iceberg::TableCommit>) -> anyhow::Result<()> {
catalog.atomic_release(commits).await?;   // all pointers swap, or none do
# Ok(()) }

7. Mental model: which knob does what

8. Gotchas

Next: dummy-guide.md for the gentle version, or design.md / plan.md for how it works underneath.