nordisk/skade

Warehouse capability-gap audit — "is the warehouse enough?"

Rickard: "is the warehouse enough, falkordb, more data structures?"

Scope. This audits what the nordisk warehouse can answer today across three repos — skade/skade-katalog (the embedded Iceberg data plane), the nornir warehouse tables, and korp's FalkorDB graph tab — and lists every data-structure / index gap, evaluating each: needed? reuse-first (extend skade / consume an existing engine) vs. new? It is deliberately long and opinionated so the eternal loop has a ranked backlog to draw the next round's work from.

Round-1 verdict up front: the warehouse is a strong OLAP core (columnar + ACID + time-travel) but is a one-trick access-path — everything is a min/max-skipped, bloom-probed linear/columnar scan. It has no secondary index families beyond bloom filters. The single highest-value missing structure (a spatial index) is implemented this round; the rest are ranked below.

1. What exists today (the baseline)

1a. skade-katalog — the catalog plane

• Embedded Iceberg Catalog on redb (single-file, ACID, MVCC). Pure Rust, in-process, no JVM / no REST hop.
• Cross-table atomic release (atomic_release_raw) — all-or-nothing multi-table commits in one redb txn.
• Time-travel index: a lock-free Ragnar STree64 static search tree (snapshot_id → (table_key, metadata_location)), ArcSwapOption-swapped, rebuilt by a background compactor every ≥1024 commits. This is the one first-class secondary index in the stack — and it indexes catalog snapshots, not table data.
• Cache tiers: L1 pointer mirror, L0 metadata cache, L1.5 table-handle cache.
• Change broadcast (CommitEvent) feeding the CDC stream.

1b. skade — the data plane

• Columnar storage: Arrow 58 → Parquet, Iceberg format V3 (deletion vectors, variant, native geometry/geography types, ns timestamps, row lineage).
• File-level data skipping: Iceberg min/max column stats + partition pruning (identity partitions) + SortOrder.
• Row-group skipping: per-column bloom filters (bloom_columns) for point lookups (lookup, read_filtered with ScanFilter::{eq,is_in,range}).
• Windowed / streaming reads: ParquetWindows, rowgroup_windows, read_row_groups — bounded-memory, all-core (gatling) ingest; the OSM / GeoParquet reuse seam.
• MOR: positional + equality deletes, changelog collapse.
• CDC streaming (ChangeStream, ConsumerCursor) + lineage (OpenLineage sink).
• SQL: DataFusion (Warehouse::sql, all 22 TPC-H).

1c. nornir warehouse tables

• Uses skade as the embedded warehouse for its own tables (bench results, docs sections, coverage). Same access paths as 1b — columnar scan + skip. No bespoke index structures of its own.

1d. korp — the FalkorDB graph tab

• src/graph.rs connects to FalkorDB via graphar_flight::FalkorExecutor (Cypher → Arrow rows), used only for the UI Graph view and lineage/infra visualisation. FalkorDB is a sidecar the korp UI talks to directly — it is NOT wired behind the warehouse abstraction, so no other consumer (nornir, holger, skade SQL) can issue a graph query, and graph data is not part of any ACID commit / time-travel timeline.
• korp/src/warehouse/mod.rs does define a Warehouse + Streaming trait seam (default impl SkadeWarehouse) — but it captures only table-listing + AS-OF snapshot read + single-batch ingest. It is the natural home for a second backend, but today it is skade-only and graph-blind.

Access-path matrix (today):

2. Is columnar + ACID + time-travel enough?

For the OLAP core: yes. For a warehouse that also wants to serve OSM/geoint, RAG, observability/timeseries, lineage-graph queries, and text search — no. Those are different access-path problems, and Iceberg's design answer is "pluggable table formats + secondary indexes," which skade has only begun (bloom). The value is not in replacing the columnar core but in adding first-class secondary structures behind one warehouse trait so a consumer picks the access path per query, over the same ACID/time-travelled data.

The rest of this doc is the ranked gap list. Each entry: need, reuse-first verdict, shape of the work.

3. GAP: Spatial index (R-tree / geohash / Z-order) — ★ HIGHEST VALUE — DONE this round

• Need: very high. OSM/GeoParquet ingest (ParquetWindows) and korp's geo view already exist, and geoint is a named use case, but a bounding-box or radius query meant a full table scan — no spatial acceleration existed anywhere in the constellation. This is the biggest capability-for-effort win and it is 100% pure-Rust (no service).
• Reuse-first: nothing to extend — there is no spatial structure in nordisk (skade, znippy, gatling, korp all lack one). A geohash is a prefix code, so a sorted Vec<(geohash, id)> is a spatial index reusing binary search — no R-tree crate, no geo dep. New code is justified (it's the gap itself).
• Delivered: skade::spatial::GeoIndex — geohash encode/decode, query_bbox, query_radius (haversine), nearest (kNN), and from_batches(lat_col, lon_col, id_col) — built from the Arrow batches the warehouse already returns (read_all/read_columns), so it plugs into the existing read seam without touching the fragile Iceberg write path. Exact (candidates verified against real coords — no false positives). 12 tests + a light in-proc timing guard. See .nornir/spatial-index-design.md.
• Next iterations (not this round): Z-order/Hilbert curve option for better locality than geohash on skewed data; persist the index as an Iceberg side-table (_geo_<col>) so it time-travels; wire it into read_filtered as a ScanFilter::Bbox file-pruner; drive it from korp's geo view.

4. GAP: First-class GRAPH store (FalkorDB / Cypher) as a warehouse backend — ★ 2nd

• Need: high. korp already runs FalkorDB, and lineage + infra-graph + case-graph are inherently graph-shaped. Today the graph lives outside the warehouse trait: it is a korp-UI sidecar, so it is not ACID with the tables, not time-travelled, and unreachable from nornir/skade SQL. Adjacency / reachability / shortest-path over lineage is a real query the columnar core answers only with recursive-CTE pain.
• Reuse-first: strongly reuse, do NOT hand-roll a graph engine. FalkorDB + graphar_flight::FalkorExecutor already exist and return Arrow. The work is not a new engine — it is a thin GraphBackend trait behind the same warehouse seam (korp/src/warehouse/mod.rs is the home; promote it to skade or a shared crate). Two impls: (a) FalkorGraph wrapping the existing executor; (b) an embedded fallback graph over the redb catalog so a graph query works with no sidecar (adjacency lists keyed in redb, reusing skade's ACID txn — a lineage edge is already a row). Booting the FalkorDB testcontainer must route through knut/jera (per prior-wave worklist), not a bespoke launcher.
• Shape: trait GraphBackend { fn cypher(&self, q) -> RecordBatch; fn neighbors(id); fn shortest_path(a,b); }, a from_edges(RecordBatch{src,dst,rel}) builder mirroring spatial::from_batches, and a lineage-graph adapter so lineage_events → graph for free. Second- highest value; deferred only because it needs the trait-promotion decision + a sidecar-vs-embedded call first.

5. GAP: Time-series structures (downsample / gap-fill / ASOF join / retention)

• Need: high. bench results, Loki metrics, CDC event streams, lineage timestamps, and any observability data are time-series. The warehouse can WHERE ts BETWEEN … but has no ASOF join (align two irregular series), no server-side downsample/rollup, no gap-fill, and no time-partitioned retention/TTL. Iceberg has hidden time partitioning we don't use (only identity partitions today).
• Reuse-first: extend skade, mostly. (a) Add Transform::{Year,Month,Day,Hour} partition transforms (Iceberg supports them; create_partitioned_table only wires Identity) → time-pruning for free. (b) ASOF join + time_bucket downsampling as DataFusion UDFs/UDAFs over the existing SQL surface — no new store. (c) Retention = an Iceberg expire-snapshots + partition-drop job. Only a rollup materialized view (see §9) needs new machinery.
• Shape: small, high-leverage — most is "use Iceberg features we skip + add DataFusion functions." Rank: do after graph, low risk.

6. GAP: Full-text SEARCH index (inverted index / FST / BM25)

• Need: medium-high. korp cases, docs sections (nornir ships a docs warehouse), lineage free-text, and log lines all want MATCH/contains ranked search. Today text search is LIKE '%…%' = full scan.
• Reuse-first: reuse tantivy (pure-Rust Lucene-alike, BM25, FST term dictionary) as a pluggable backend — do NOT hand-roll an inverted index. Wire it as an Iceberg side-table index (_fts_<col>) built from the same batches (from_batches pattern), queried through a SearchBackend trait on the warehouse seam. An FST-only lightweight variant (fst crate) is the no-dep-explosion middle ground if tantivy is too heavy for the embedded story.
• Shape: medium. Rank: after time-series. The from_batches seam this round establishes is the template.

7. GAP: Vector index for RAG / embeddings (HNSW / IVF, ANN)

• Need: medium (rising). If any nordisk tool grows RAG / semantic search over cases/docs, the warehouse needs approximate-nearest-neighbour over embedding columns. Iceberg V3 can store the vectors (fixed-size list / variant); there is no ANN index, so it's a brute-force cosine scan today.
• Reuse-first: reuse an HNSW crate (hnsw_rs / instant-distance, pure Rust) behind a VectorBackend trait — do NOT write HNSW. Note the structural overlap with §3: spatial::nearest is exact kNN in 2-D; a vector index is approximate kNN in N-D. A shared NearestNeighbor trait could unify them, with geohash for low-D exact and HNSW for high-D approximate. Brute-force cosine is a fine interim exact fallback (reuse the spatial candidate-verify pattern).
• Shape: medium-large (HNSW build/persist is the cost). Rank: defer until a concrete RAG consumer exists — don't build speculative index infra.

8. GAP: Bitmap indexes (roaring) for low-cardinality columns & set ops

• Need: medium. Low-cardinality dimensions (repo, status, region, category, label) are everywhere (korp cases, nornir testmatrix cells, holger decisions). Bloom answers "maybe present"; a roaring bitmap per distinct value answers exact membership + fast AND/OR/NOT across predicates + fast COUNT DISTINCT / faceting — which the facet-tree UIs (facett, korp facets) recompute by scan today.
• Reuse-first: reuse roaring crate behind the same side-table index seam; do NOT hand-roll compressed bitmaps. Strong synergy with the facet UIs and with DataFusion (bitmap → selection vector). One roaring::RoaringBitmap per (column, value) built from the read batches.
• Shape: small-medium. Rank: after FTS; cheap and broadly useful for the facet surfaces.

9. GAP: Materialized views / rollups (incremental maintenance)

• Need: medium. Dashboards (nornir viz, korp) and rollups (daily bench aggregates, per-repo coverage) re-run the same aggregation every load. A materialized view — a table maintained incrementally off the CDC stream skade already emits — would cut that to a lookup.

• Reuse-first: extend skade — this is nearly free given existing parts. A matview = a normal Iceberg table + a ChangeStream consumer that applies deltas

  • a ConsumerCursor for exactly-once resume. All three exist. The new bit is a small view definition + apply loop (and the honest hard part: incremental aggregate maintenance vs. periodic full refresh — start with periodic refresh off the cursor, which is trivial and correct).

• Shape: small (periodic-refresh) → medium (true incremental). Rank: after bitmap; leverages CDC + lineage that already ship.

10. GAP: Richer secondary / covering indexes (exact, not just bloom)

• Need: medium. Bloom is probabilistic (false positives → still reads the row group). For high-selectivity equality on a non-sort column there is no exact secondary index mapping value → file/row, and no covering index (answer from the index without touching data files).
• Reuse-first: extend skade — a sorted (value → file, row-group) side-table (same builder seam as §3/§6/§8) reusing redb for persistence. This is the generalisation the spatial/FTS/bitmap indexes are all special cases of; a future round should factor a shared SecondaryIndex trait + side-table registry in the catalog so all index families share build/persist/time-travel plumbing instead of each re-inventing it.
• Shape: medium (the unifying abstraction). Rank: the consolidation step once 2–3 concrete index families exist to generalise from.

11. Cross-cutting: the missing abstraction, not just the structures

The real gap under all of §3–§10 is that skade has no pluggable SecondaryIndex / warehouse-backend trait that lets an index family (spatial, graph, FTS, vector, bitmap) share: build from batches, persist as an Iceberg side-table (so it time-travels with the data), prune in read_filtered, and register on the warehouse seam (korp's Warehouse/Streaming traits are the seed — promote them into skade/a shared crate). This round ships the first concrete member (spatial) with the from_batches shape deliberately chosen to become that template. The recommended arc:

1. spatial (done) — establishes the from_batches build seam. ← this round
2. graph backend trait — promote korp's seam, wrap FalkorExecutor + embedded redb-graph fallback.
3. time-series — Iceberg time-transforms + DataFusion ASOF/downsample UDFs.
4. bitmap (roaring) — cheapest broad win; wires the facet UIs.
5. full-text (tantivy/fst) — reuses the build seam.
6. materialized views — off the existing CDC stream + cursor.
7. vector/ANN (hnsw) — only when a RAG consumer is real.
8. SecondaryIndex trait + side-table registry — factor the shared plumbing out of 1–7; add ScanFilter::Bbox / exact secondary pruning to read_filtered.

Answer to the question: the columnar+ACID+time-travel core is enough as a core. FalkorDB should be promoted from a korp-UI sidecar to a warehouse backend behind a trait (not left where it is). And yes — the warehouse needs more data structures: spatial (shipped), then graph, time-series, bitmap, full-text, matviews, and (later) vector — all as pluggable, time-travelling side-table indexes behind one seam, reusing existing crates (FalkorDB, tantivy, roaring, hnsw) rather than hand-rolled twins.