nordisk/skade

skade for absolute beginners

No prior data-lake knowledge assumed. By the end you'll know what skade is, why it exists, and you'll have run your first query. When you want the real API, go to tutorial.md; for how it works inside, design.md.

The one-sentence version

skade turns a single folder on your disk into a real, transactional, queryable analytics database — no server to run, no JVM, no Postgres. You hand it columns of data; you get SQL back.

The problem it solves (in plain words)

Imagine you have a lot of tabular data — millions of rows, growing over time — and you want to:

1. append to it safely (no half-written messes if the program crashes),
2. query it with SQL,
3. keep its history so you can ask "what did this table look like last Tuesday?",
4. and do all of that without standing up a database server.

The industry's answer to "big tables in files, with history and transactions" is a format called Apache Iceberg. Iceberg is great, but the usual way to run it drags in a catalog server (often a JVM service backed by Postgres) whose only job is to remember "which file is the current version of each table." That's a lot of moving parts for something that could be a single file.

skade is that catalog — and a writer/reader — as an embedded library. The "which file is current" bookkeeping lives in one small transactional file (catalog.redb) right next to your data. Your whole lakehouse is one directory.

An analogy

Think of a library:

• The books are your data files (Parquet — compressed columns of rows).
• The card catalog is what tells you which book is the current edition of each title. In a normal Iceberg setup the card catalog is a whole separate building with staff (a server). skade keeps the card catalog as one tidy index card box on the same shelf as the books — instant to consult, and it never disagrees with itself because every update is atomic.
• Editions = snapshots. Every time you add data, Iceberg writes a new edition and updates the catalog to point at it. The old editions stay on the shelf, so you can always go back and read last week's edition (time travel).

What you actually get

• One directory = one lakehouse. Copy it, back it up, rsync it — done.
• ACID. A write either fully happens or doesn't; a crash never leaves a table half-updated.
• SQL. SELECT … FROM my_table JOIN other … over your data.
• Time travel. Read any past snapshot.
• Fast. Reading "the current version of table X" is a ~100-nanosecond in-memory lookup, not a network round-trip.
• No server, no JVM. It's a Rust crate you link into your program.

Your first five minutes

Add it to a Rust project:

[dependencies]
skade = { version = "0.4", features = ["sql"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
anyhow = "1"

Open a folder, make a table, put three rows in, count them with SQL:

,no_run
use std::sync::Arc;
use skade::iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use skade::arrow_array::{RecordBatch, Int64Array};
use skade::arrow_schema::{Field, Schema as ArrowSchema, DataType};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // 1. The folder "lake" becomes your whole lakehouse.
    let wh = skade::open("./lake").await?;

    // 2. Say what columns the table has (the "schema"). Each column needs an id.
    let schema = Schema::builder()
        .with_schema_id(0)
        .with_fields(vec![
            NestedField::required(1, "id", Type::Primitive(PrimitiveType::Long)).into(),
        ])
        .build()?;
    let mut t = wh.create_table("greetings", &schema).await?;

    // 3. Hand it some data (Arrow = the in-memory columnar format).
    let arrow = Arc::new(ArrowSchema::new(vec![Field::new("id", DataType::Int64, false)]));
    let batch = RecordBatch::try_new(arrow, vec![Arc::new(Int64Array::from(vec![1, 2, 3]))])?;
    t.append(&[batch]).await?;        // this is one durable commit / snapshot

    // 4. Ask a question.
    let answer = wh.sql("SELECT count(*) AS n FROM greetings").await?;
    println!("{answer:?}");           // n = 3
    Ok(())
}

Run it. Look in ./lake: there's catalog.redb (the index-card box) and a greetings/ folder with .parquet data and .metadata.json editions. That's a production-shaped Iceberg lakehouse you can hand to Spark, DuckDB, or anything else that reads Iceberg.

Tiny glossary

Two layers (so the docs make sense)

skade is two crates, and you'll see both names:

• skade — the friendly part you just used (open, create_table, append, sql). Start here.
• skade-katalog — the bare catalog engine underneath (implements the Iceberg Catalog trait over redb). You only touch it directly for advanced things like custom storage backends or multi-table atomic releases.

When skade is the right tool — and when it isn't

Great fit: one program (or one machine) owning a dataset; embedding a lakehouse in an app or CLI; fast local analytics; CI pipelines that publish result tables; anywhere you'd otherwise resent running a catalog server.

Not a fit: many separate processes/hosts writing the same catalog at once. redb is a single-file lock — one writer process at a time. (It's a library, not a shared server. For that, you'd want a REST catalog like Nessie/Polaris — which skade is benchmarked against, and beats on single-node latency.)

Where to go next

• Hands-on, all the real API + bulk ingest + time travel → tutorial.md
• How the speed actually works (the cache layers, the Ragnar search tree) → design.md, then plan.md
• What's planned / not built yet → TODO.md