nordisk/nornir

SQL over the warehouse — design

Class: design · Function axis: f0 (the one read path) + f1 lazy schema · f2 native provider · f3 open + format surfaces · f4 schema discovery. Strict: this file states only the design that is, every claim carrying a file:symbol / file:line pointer, no why. The why the native provider and not a MemTable lives in warehouse-reasoning.md. How to run queries is sql-guide.md. The tables it reads are catalogued in warehouse-design.md (f2) and warehouse-schema.md.

f0 — one read path: DataFusion over the Iceberg tables (the whole)

nornir warehouse query "SELECT …" executes arbitrary read-only SQL over the warehouse with DataFusion. It is a single function — warehouse::sql::query(wh, sql) -> Vec<RecordBatch> (src/warehouse/sql.rs:97) — that:

1. builds a DataFusion SessionContext and replaces its default public schema with a warehouse-backed SchemaProvider;
2. lets the planner resolve the referenced tables against the Iceberg catalog, each wrapped in the native iceberg-datafusion provider so the scan pushes projection + filter down to Parquet;
3. drives the plan on the warehouse's own tokio runtime (wh.block_on, sql.rs:107) and returns the result batches.

No table is copied into memory: the Parquet footers' column / row-group bounds do the pruning, so SELECT … WHERE repo = 'x' reads only the matching data files. Every warehouse table is queryable by its warehouse name, and the full DataFusion SQL surface works — WHERE, JOIN across tables, GROUP BY / aggregates, ORDER BY, LIMIT.

Reuse map (what SQL uses): it depends only on the existing warehouse — it opens IcebergWarehouse read-only (super::open_read_only) and reads the same RedbCatalog + Iceberg tables every other function writes. It adds DataFusion + iceberg-datafusion as the engine, in the same arrow-58 / iceberg-arrow58 dependency universe already used by iceberg, so the Table/Catalog types unify with zero glue. It is used by the warehouse query CLI verb (src/bin/nornir.rs:12434) and the warehouse_query MCP tool (src/bin/nornir-mcp.rs:1515).

f1 — the lazy warehouse schema (WarehouseSchema)

struct WarehouseSchema (sql.rs:57) is a DataFusion SchemaProvider (impl at sql.rs:64) backed by the warehouse's Iceberg catalog. It holds three things: the Arc<RedbCatalog>, the warehouse NamespaceIdent, and the already-known table_names list (fetched once at construction from wh.table_names(), sql.rs:101).

The laziness is the robustness property: a query loads only the tables it references (never all ~50), so it never pays to open the whole warehouse, and a single torn / blank table's metadata cannot break an unrelated query. Bare table names (FROM bench_runs) resolve because the warehouse schema is registered as DataFusion's public schema under the datafusion catalog (sql.rs:112–:117).

f2 — the native Iceberg table provider

Each referenced table is wrapped in IcebergStaticTableProvider (iceberg_datafusion::IcebergStaticTableProvider::try_new_from_table, sql.rs:83) — a read-only, catalog-backed DataFusion TableProvider that plans an Iceberg table scan with projection + filter pushdown all the way to the Parquet files. The provider is constructed inside WarehouseSchema::table from the catalog.load_table(&ident) result (sql.rs:78), so the same iceberg::Table handle the rest of the warehouse uses becomes a DataFusion source directly. This module never mutates — there is no append/insert path.

f3 — open, run, format (the surfaces)

Read-only, lock-tolerant open. Callers open the warehouse via IcebergWarehouse::open_read_only (iceberg.rs:319): a normal exclusive open, or — when a live nornir-server holds the exclusive catalog.redb lock — a consistent copied-aside snapshot of the catalog (open_snapshot), so queries see the catalog as of "now" without touching the writer's lock. Table data still resolves to the live warehouse (absolute file:// paths). See warehouse-design.md f1 for the single-writer layer.

Formatting (sql.rs):

Surfaces:

f4 — schema discovery (warehouse tables)

warehouse query needs table + column names to write a SELECT; f4 is the discovery companion that surfaces them from the SAME catalog, without reading a single Parquet file:

ColumnInfo (sql.rs, serde::Serialize) is the row shape: name (as it appears in SQL), data_type (Arrow logical type via DataType: Display — Utf8, Int64, Timestamp(µs, …)), and nullable. Discovery is cheap because it never scans data — list_tables reads only the catalog's name index, and describe_table loads one table's metadata (its current schema), not its files.

Surfaces:

• CLI nornir warehouse tables [<table>] [--json] (WarehouseTablesArgs, nornir.rs; run_warehouse_tables, nornir.rs) — no argument lists tables, a <table> describes its columns. Its own read-only catalog opener (same lock-tolerant open_read_only path as query).
• MCP warehouse_tables tool (nornir-mcp.rs, sibling of warehouse_query) — no args → the JSON table-name array; {"table": …} → that table's [{name,data_type,nullable}]. Runs embedded only (errors if NORNIR_SERVER is set), so a small/local agent can DISCOVER the schema (valid FROM targets + each table's columns) before it writes a warehouse_query SELECT — one round-trip, no guessing table/column names.

Verified by src/warehouse/sql.rs::tests over a seeded temp warehouse: select_where_over_seeded_warehouse (sql.rs:228), join_across_two_warehouse_tables (sql.rs:242), aggregate_group_by_over_call_edges (sql.rs:260), json_output_is_valid_json_array (sql.rs:288), and for f4: list_tables_surfaces_the_seeded_tables_sorted, describe_table_reports_columns_and_types, describe_unknown_table_errors_with_the_known_list. The MCP warehouse_tables tool is exercised + coverage-gated by tests/support/mcp_harness.rs (mcp_every_tool_exercised_and_asserted): {"table":"bench_runs"} must return the seeded table's repo + machine columns, and an unknown table must error — a RED-when-broken matrix row (the completeness gate fails if the tool is ever left un-exercised).