nordisk/skade

"My table is in MinIO and Doris can't see it" — the idiot guide

You wrote a big table with skade. The Parquet is sitting in MinIO. Doris says there are no tables. Here is what is going on and what to type.

The mental model: two separate things

A skade warehouse is two things, and people mix them up:

1. The catalog — one small file, something.redb, on a local disk. It is a phone book: "the table main.personer currently lives at this address".
2. The data — Parquet files, Avro manifests and JSON metadata. These can live anywhere: a local directory, or a bucket in MinIO.

skade-katalog serve is a little server that reads the phone book out loud over HTTP, in the language every query engine speaks (Iceberg-REST). Doris calls it, gets the address, and then goes and reads the Parquet from MinIO by itself.

So three things have to be true:

• serve must open the actual phone book you wrote into;
• serve must be able to reach the addresses in it (if they say s3://…, serve needs S3 credentials, because it has to read the metadata JSON);
• Doris must be able to reach the same bucket (it has its own credentials).

Failure 1: "namespaces is empty"

$ curl localhost:8181/v1/namespaces
{"namespaces":[]}

This almost always means serve opened the wrong file. redb does not complain about a missing database — it quietly creates a new, empty one. An empty phone book has no names in it.

Check the file before you start serve:

ls -l /path/to/your.redb        # should exist, and be megabytes, not zero

The classic way to get this wrong is the writer's argument order:

// RIGHT — the local redb file comes FIRST
Warehouse::open_with_store("/var/lib/skade/wh.redb", "s3://red/skade-warehouse", store)

// WRONG — silently writes the redb to ./s3:/red/skade-warehouse
Warehouse::open_with_store("s3://red/skade-warehouse", "/var/lib/skade/wh.redb", store)

If you did the wrong one, your redb is in a weird directory called s3: under wherever you ran the program. Look for it there.

Failure 2: "the tables are listed but I can't open one"

$ curl localhost:8181/v1/namespaces/main/tables/personer
{"error":{"message":"Failed to read file s3://red/…/00001-….metadata.json:
                     No such file or directory"}}

The phone book is fine. serve just can't reach the address: it went looking on its own hard disk for something that lives in MinIO. Give it the bucket:

export MINIO_ACCESS_KEY=…
export MINIO_SECRET_KEY=…

skade-katalog serve \
  --db            /var/lib/skade/wh.redb \
  --warehouse-uri s3://red/skade-warehouse \
  --s3-bucket     red \
  --s3-endpoint   https://172.16.12.9:9000 \
  --rest          0.0.0.0:8181

Two catches:

• the binary must have been built with the feature: cargo build --release --features object-store (a plain build tells you so).
• --warehouse-uri must be exactly the URI the writer used. Not a similar one. It is how the addresses were written down.

Newer builds will not even start in that state — if you give a s3://… --warehouse-uri and forget --s3-bucket, serve says so and exits instead of coming up and listing tables it cannot open. That is on purpose: a server that half-works is much harder to debug than one that refuses.

Failure 3: "403 InvalidAccessKeyId"

Your keys aren't in the environment, or they're the wrong keys. serve never takes credentials on the command line on purpose — anyone running ps would see them. Export them and restart.

If instead you see

Failed to parse json string, source: expected value at line 1 column 1

that is the same problem on an old build. S3 answered "no you may not", in XML, and skade handed that XML to the JSON parser as if it were your table's metadata. Current builds say s3 get <key>: HTTP 403: …AccessDenied… instead. Upgrade before you spend an afternoon on it.

Am I sure it works? Ask the machine

Rather than curl three URLs and squint, run the shipped check. It writes a tiny table into your bucket, serves it, reads it back over REST and tidies up:

export MINIO_ACCESS_KEY=… MINIO_SECRET_KEY=…
export SKADE_LIVE_S3_ENDPOINT=https://172.16.12.9:9000 SKADE_LIVE_S3_BUCKET=red
cargo run --release --features object-store --example live_object_store

The last line is live_object_store: PASS or it is not.

Then, in Doris

CREATE CATALOG skade PROPERTIES (
  'type'                 = 'iceberg',
  'iceberg.catalog.type' = 'rest',
  'uri'                  = 'http://<host-running-serve>:8181',
  'warehouse'            = 's3://red/skade-warehouse',
  's3.endpoint'          = 'https://172.16.12.9:9000',
  's3.access_key'        = '…',
  's3.secret_key'        = '…',
  'use_path_style'       = 'true'
);

SWITCH skade;
SHOW DATABASES;          -- expect: main
SELECT count(*) FROM main.personer;

Doris needs the S3 keys as well, because it reads the Parquet itself. serve only hands out addresses.

Two things people expect that aren't true

• iceberg.catalog.type = hadoop will not work on a skade warehouse. The hadoop catalog finds tables by looking for v3.metadata.json plus a version-hint.text file. skade writes 00001-<uuid>.metadata.json and keeps the pointer in the redb catalog — which is the whole point of having a catalog. Doris will see the namespace directory and no tables. Use rest.
• Only one process at a time may own the redb. That is deliberate: serve is the single writer, and everything else talks to serve. Stop the writer before starting serve.