nordisk/skade

skade-flight over an existing lake — any Iceberg catalog, read-only

Design (chosen path only). Power BI / ODBC / ADBC reach an EXISTING Iceberg lake — written by Spark, catalogued in Nessie, data on MinIO — through skade-flight, read-only, pure Rust. How the catalog is opened lives in skade: catalog-connectors-design.md.

f0 — the whole idea

skade-flight's SQL layer was already catalog-agnostic: IcebergCatalogProvider takes an Arc<dyn iceberg::Catalog>. Only the Engine held a skade Warehouse. So the engine now holds the catalog:

skade::connect::{SkadeLocal | SkadeObjectStore | NessieV2 | IcebergRest}
        │ CatalogConnector::open
        ▼
Arc<dyn Catalog> ─ Engine::with_catalog ─ Flight SQL ─ ODBC / ADBC ─ Power BI

Engine::new / with_options / with_engine_options(Warehouse, …) are unchanged; they delegate with the warehouse's RedbCatalog (the warehouse is held, so its redb lock lives as long as the engine). Engine::warehouse() keeps its signature and panics (documented) on a with_catalog engine; try_warehouse() is the non-panicking form, catalog() the catalog.

Decisions

• Read-only is the engine's. Spark stays the lake's writer; skade-flight never commits. Session views are refused too under read_only — they are writes from the client's point of view.
• Nested namespaces are flat schemas. Every namespace, at any depth, is ONE DataFusion schema named by its elements joined with .: skade."a.b".orders resolves, and GetDbSchemas / GetTables — Power BI's navigator — list a.b (FlatNamespaceCatalog, skade-flight/src/namespaces.rs). The Iceberg keys are untouched; the schema name maps back to the elements (namespace_of), which is also how GetPrimaryKeys loads a nested table. The walk is parent → children from the top, so a catalog must list parents (skade's Warehouse now creates a table's ancestor namespaces). A namespace with an element that itself contains . would flatten ambiguously, so it is not served and a log line names it. The stock IcebergCatalogProvider listed only top-level namespaces and made each element its own schema — replaced.

Proven by

• skade-flight/tests/with_catalog.rs — SELECT through with_catalog in a non-default namespace, every write path refused under read_only, the warehouse constructors still delegate.
• skade-flight/tests/nested_namespaces.rs — one-, two- and three-level namespaces listed by GetDbSchemas / GetTables and SELECTed through the Flight SQL door; namespaces.rs unit tests — flattening, mapping back, and a dotted element skipped by name.