Reading a whole table fast — what Table::read does for you
Class:
guide· Function:read. How the full-scan read behaves and when it takes which path. You call one method; skade picks the cheapest correct decode. Seeread-scan-decode-design.mdfor the shape.
Just call read()
let batches = table.read().await?; // full current-snapshot scan → ArrowYou never choose a decode strategy — read_all does:
| Table shape | Path | Cost |
| append-only, single schema, no deletes, local warehouse | stream each file from its fd, all-core (gatling) | fastest, low memory |
| append-only, single schema, no deletes, remote FileIO (S3) | read whole bytes via FileIO, all-core (gatling) | fast; holds file bytes in memory |
| schema evolution / any delete file | Iceberg engine to_arrow (merge-on-read) |
correctness-complete |
| no current snapshot | zero batches | free |
All paths return the same rows — the only difference is how much I/O and memory it takes to get them.
What changed (and what didn't)
For a local warehouse (anything opened with Warehouse::open / skade::open) a full scan no longer reads every data file's whole bytes into memory before decoding — it streams each file straight from disk. On a big table this drops the peak memory from the whole dataset to the batches you get back, and speeds the scan (no serial pre-read in front of the decode).
Nothing about the result changes: same batches, same order, same schema. A remote/S3 warehouse behaves exactly as before (whole-bytes read through FileIO — the only thing an object store can serve).
Want fewer rows / only some columns?
read() materializes the whole snapshot. If that's more than you need:
read_columns(&["a", "b"])— project columns (less I/O + decode).read_filtered(&filter, &cols)— push a predicate down (file/row-group pruning).read_limited(max_rows)— stop after N rows (cancels the rest of the scan).lookup(&key, as_of)— a single row by key (bloom-pruned raw-Parquet fast path).
Observing it
skade/read emits a functional_status marker on each full scan (gated #[cfg(feature = "testmatrix")]). In tests, the LOCAL_STREAMED_READS / BYTES_SLURP_READS counters (white-box, #[cfg(test)]) tell you which decode strategy a scan took — used to prove a local warehouse takes the streamed path. The skade.read_local_stream bench arm A/Bs the two strategies (rows/sec + the whole-dataset byte footprint the slurp holds that streaming avoids).