nordisk/skade

Serving a MinIO/S3 warehouse over Iceberg-REST — guide

skade-katalog serve fronts a RedbCatalog over Iceberg-REST and/or gRPC. By default it assumes the Iceberg blobs live on the local filesystem. Build it with --features object-store and it can front a warehouse whose blobs live in S3/MinIO — i.e. the warehouse skade::Warehouse::open_with_store writes.

cargo build --release --features object-store   # in skade/serve

The two things that must match the writer

A warehouse is a redb file plus a place the blobs live. serve needs both, and both must be what the writer used:

Mind the argument order. Warehouse::open_with_store takes (db_path, warehouse_uri, store) — the local redb file first. Passing them the other way round is silent: the redb lands in a directory literally named s3:/<bucket>/… under the writer's working directory, and the blobs go to a prefix named after the redb path. serve --db then opens a different, empty redb and reports {"namespaces":[]}.

Running it

export MINIO_ACCESS_KEY=…        # or SKADE_S3_ACCESS_KEY_ID / AWS_ACCESS_KEY_ID
export MINIO_SECRET_KEY=…        # or SKADE_S3_SECRET_ACCESS_KEY / AWS_SECRET_ACCESS_KEY

skade-katalog serve \
  --db          /var/lib/skade/warehouse.redb \
  --warehouse-uri s3://red/skade-warehouse \
  --s3-bucket   red \
  --s3-endpoint https://minio.example:9000 \
  --rest        0.0.0.0:8181

Credentials are taken from the environment only — never a flag, so they stay out of ps, shell history and systemd unit files. Path-style addressing is the default (MinIO needs it); pass --s3-virtual-hosted for real AWS-style hosts. --warehouse ROOT is not needed in this shape; it stays required when you serve a plain local file:// warehouse, exactly as before.

Check it:

curl -s localhost:8181/v1/namespaces
curl -s localhost:8181/v1/namespaces/main/tables
curl -s localhost:8181/v1/namespaces/main/tables/personer | head -c 400

loadTable returning a metadata document with an s3://… metadata-location means the whole path is live.

Or let the shipped check do all three and tell you PASS/FAIL. It writes a small table into a throwaway prefix, serves it, loads it back over REST and deletes what it wrote:

export MINIO_ACCESS_KEY=… MINIO_SECRET_KEY=…
export SKADE_LIVE_S3_ENDPOINT=https://minio.example:9000 SKADE_LIVE_S3_BUCKET=red
cargo run --release --features object-store --example live_object_store

It skips with one line (exit 0) when those variables are unset, so it is safe to wire into any pipeline. SKADE_LIVE_S3_KEEP=1 leaves the objects for you to inspect.

serve refuses to start rather than half-work

The failure this feature exists to remove is a catalog that lists every table and loads none. So the two pairings that produce it are checked before the socket binds:

• --warehouse-uri s3://… without --s3-bucket — the catalog would come up on the local filesystem while handing out s3:// addresses;
• --warehouse-uri s3://… on a binary built without --features object-store — nothing in it can reach that scheme.

--warehouse-uri file://… is fine in both shapes: relocating a local root is not the same as needing a store.

Pointing Apache Doris at it

CREATE CATALOG skade PROPERTIES (
  'type'                = 'iceberg',
  'iceberg.catalog.type'= 'rest',
  'uri'                 = 'http://<serve-host>:8181',
  'warehouse'           = 's3://red/skade-warehouse',
  's3.endpoint'         = 'https://minio.example:9000',
  's3.region'           = 'us-east-1',
  's3.access_key'       = '…',
  's3.secret_key'       = '…',
  'use_path_style'      = 'true'
);

Doris fetches the table pointer and metadata from serve and then reads the Parquet directly from MinIO with its own credentials — which is why the S3 properties appear on both sides.

Troubleshooting