.nornir/flight-nessie-resume-plan.md

skade-flight over Nessie + MinIO for Power BI — resume plan

Resume here. State as of 2026-09-15, skade main = c079e70. What landed, what is left, and the exact commands to pick it up. The design is flight-rest-catalog-design.md and catalog-connectors-design.md.

Goal

Power BI Desktop reads an EXISTING Iceberg 1.10 lake — written by Spark 3.x, catalogued in Nessie 0.105.3 (ROCKSDB, authentication disabled), data on a mutual-TLS MinIO at s3a:// — through skade-flight, read-only, pure Rust.

Done (on `main`, `83f0b50..c079e70`)

commit	what
`c606eb9`	engine serves any `iceberg::Catalog`, own namespace, read-only when told
`d08b2a4`	`skade::connect::CatalogConnector` — `SkadeLocal`, `SkadeObjectStore`, `NessieV2`; `skade::tls::ClientTls`; `Engine::with_connector`
`3533424`	nested namespaces list as one schema `a.b`
`a46e85d`	`IcebergRest` connector (feature `rest-catalog`)
`3c9ef32`	`skade-flight` binary: `connector_from_args`; `--nessie-uri` / `--catalog-uri` / `--warehouse-uri` exclusive; tokens env-only; TLS per leg (s3, nessie, catalog, powerbi) → plain \| tls \| mtls printed at startup; default listen `0.0.0.0:10009` + exposure warning
`e5b9804`	`tests/nessie_live.rs`; bench Nessie pinned 0.105.3, `nessie-plain` on :19121 (no `nessie.catalog.*`, auth disabled)
`626d6b4`	fixes on A's connectors: S3 settings under `nessie`/`rest-catalog`; TLS S3 via the aegir store
`c079e70`	merge of `83f0b50` (Flight SQL door from nornir-flight)

Verified: 21 flag + 29 lib + 7 flight_sql_roundtrip tests green on c079e70. Live, Nessie 0.105.3 + RustFS: arm 1 (API v2, no catalog config, s3a:// table registered by a v2 commit) and arm 2 (Iceberg REST) each list the table, return 5 rows (count/sum 5/15), refuse INSERT. The live arms ran on 626d6b4; 83f0b50 touched only auth, covered by flight_sql_roundtrip.

Findings: Nessie's /iceberg/v1/config returns the prefix under defaults (main%7Cwarehouse, pattern {ref}|{warehouse}), not overrides. Arm 1 needs no server-side catalog config; arm 2 needs the nessie.catalog.* warehouse + S3 block.

Left — in order

#	item	why	next step	owner
1	Power BI Desktop on Windows, end to end	never run; Dremio documents its Flight SQL ODBC driver as Dremio-only	install the 64-bit "Arrow Flight SQL ODBC" MSI; DSN below; Get Data → ODBC; check navigator, load, filtered refresh	needs a Windows box
2	korp still dials 50055	korp misses skade-flight's new default	`korp/src/backend.rs` → 10009	main agent
3	the real lake	only containers so far	`skade-flight --nessie-uri http://<nessie>:19120/api/v2 --s3-endpoint https://<minio> --s3-bucket <b> --s3-ca … --s3-client-cert … --s3-client-key …` against one Spark-written `s3a://` table	—
4	mutual TLS	no live mTLS arm (RustFS has no cheap client-auth); the ODBC driver lists no client-cert property	verify against the real mTLS MinIO (item 3); plan Power BI → skade-flight as TLS + Basic, not mTLS	—
5	Power BI service refresh	the generic ODBC connector is Import-only and needs an on-premises gateway with the driver	decide gateway vs Desktop-only	Rickard

Minor: bench-containers cannot be built from a worktree (lockfile collision through bench's nornir path dep); nessie_live.rs #[path]-includes bench/src/containers.rs instead.

Power BI facts (checked 2026-09-15)

Microsoft's ODBC→ADBC transition covers only connectors with an embedded driver (Databricks, Snowflake, Dremio, BigQuery, Impala, Spark); it "doesn't change behavior for the ODBC connector when you use a separately installed ODBC driver" — our route.
ODBC connector: Import only; attributes after dsn= in a connection string are ignored — put them in the DSN.
No generic ADBC / Flight SQL connector for arbitrary endpoints. Apache Arrow ships no ODBC driver of its own (arrow#30622 closed, not planned).

DSN (64-bit ODBC Data Source Administrator, driver "Arrow Flight SQL ODBC"):

Host=<skade-flight host>   Port=10009
UID=<--basic user>         PWD=<--basic password>
useEncryption=false        (true with --tls-cert/--tls-key)
trustedCerts=<ca.pem>      (if the CA is not in the Windows store)

Resume commands

cd ~/git/skade/skade-flight
# flag tests (all 21)
cargo test --features nessie,rest-catalog,object-store --bin skade-flight
# live arms — starts RustFS + both Nessies over the podman socket
cargo test --features nessie,rest-catalog --test nessie_live -- --ignored --nocapture --test-threads=1
# serve the production-shaped lake
SKADE_S3_ACCESS_KEY_ID=… SKADE_S3_SECRET_ACCESS_KEY=… SKADE_FLIGHT_TOKEN=… \
  cargo run --features nessie,rest-catalog -- \
  --nessie-uri http://<nessie>:19120/api/v2 --s3-bucket <bucket> --s3-endpoint https://<minio>

Sources: ODBC→ADBC transition, Power Query ODBC connector, Arrow Flight SQL ODBC.