Databricks / Apache Spark UI parity — a facett build map
Status: RESEARCH + DESIGN (no code). This doc catalogues everything in the open-source Apache Spark Web UI and the Databricks Spark/SQL observability surfaces, then maps each view onto facett's component family — what already covers it, what needs an extension, and what is net-new. The goal is "we want that class of UI in facett": a self-hosted, egui-native, headless-testable rebuild of the Spark/Databricks monitoring experience, fed by engine-agnostic view-models (the same source-agnostic seam as
facett-pipelineandfacett-grid).facett owns only the picture. A host (knut / korp / nornir viz) maps the Spark REST API / event log / Databricks
system.query.historyonto facett view-models; facett never learns what a "stage" is. This mirrors the existingPipelineAdaptersplit (engine → adapter → normalized model → view).Companion docs:
components.md(the crate list this cites),facett-grid.md,facett-pipeline.md,facett-graphview.md,facett-syschart.md,roadmap.md.
0. Executive read
The Spark/Databricks UI is, structurally, five visual primitives repeated across ~12 tabs:
- Big sortable metric tables (Jobs, Stages, Tasks, Executors, Storage, Environment, Query History, JDBC/Connect sessions) — facett already nails this with
facett-grid(1M-row virtualized, multi-sort, pinned cols, Σ footer, in-cell bars/sparklines/chips, conditional-format heat). - Operator/RDD DAGs (job DAG, stage DAG, SQL query plan, Databricks Query Profile graph) — partly covered by
facett-pipeline::PipelineView+facett-graph::DepGraphView; the gap is per-node metric bodies + cost-heat + AQE initial/final + collapse. - Timeline / Gantt swimlanes (Jobs event timeline, per-stage task timeline with stacked phase segments) — net-new; nothing in facett draws a horizontal time-axis Gantt today (
facett-plotis x/y only). - Percentile / distribution rows (Summary Metrics for Tasks: Min/25/Median/ 75/Max; skew detection) — partly covered by
facett-stat::StatGridViewsparklines; needs a box/percentile cell. - Time-series line+histogram pairs (Structured Streaming stats, Databricks compute metrics) —
facett-plotcovers the line/ bar half; the paired histogram + stacked "operation duration" band is an extension.
Plus two profiling artifacts (thread dump → flame graph, plan text tree) that are minor/long-tail.
The three highest-ROI builds (see §3) are: a facett-timeline Gantt (net-new, unlocks Jobs + Task timelines + Lakeflow run Gantt), a plan-DAG extension to facett-pipeline/DepGraphView for metric-bearing, cost-heated, collapsible nodes (the SQL plan / Query Profile graph), and wiring the existing facett-grid to the Stage/Task/Executor tables (near-zero-cost, huge coverage).
1. The full Spark / Databricks UI inventory
Legend for visual form: table · gantt (horizontal time-axis swimlane) · dag (node/edge operator graph) · plan-tree (collapsible text) · percentile (Min/25/Median/75/Max distribution row) · line+hist (timeline chart + paired histogram) · bar (progress / segmented bar) · flame (stacked-sample flame graph) · kv (key/value table).
1A. Open-source Apache Spark Web UI
| View | Data it shows | Visual form | Why useful |
| Jobs — landing | header (User, Uptime, Scheduling Mode, Active/Completed/Failed counts); jobs table: Job Id, Description, Submitted, Duration, Stages succeeded/total, Tasks progress | table + bar |
fastest triage — spot the failed/slow job, the stalled task bar |
| Jobs — event timeline | executor add/remove markers + each job as a bar on a shared wall-clock axis | gantt |
correlate executor churn (dynamic allocation, lost executors) with job timing |
| Job detail | header (status, stage counts active/pending/completed/skipped/failed, SQL link); stages table (Stage Id, Desc, Submitted, Duration, Tasks bar, Input, Output, Shuffle R/W) | table + bar |
which stage dominates; skipped = cache/shuffle reuse |
| Job detail — RDD DAG | RDD/DataFrame vertices, op edges, boxed by stage (boundaries = shuffles); green = cached, grey = skipped | dag |
how the computation split into stages, where shuffles force boundaries |
| Job detail — event timeline | executor track + each stage as a bar | gantt |
stage serialization vs overlap, gaps between stages |
| Stages — landing | per-status stage tables (same cols as job detail); Fair Scheduler Pools table (name, min share, weight, active stages, running tasks, mode) | table |
cross-job stage overview; pool fairness |
| Stage detail — header | Total Time Across Tasks; Locality Summary (PROCESS/NODE/RACK/ANY); Input/Output/Shuffle R/W size+records; associated Job Ids | kv |
stage-level totals + locality quality |
| Stage detail — operator DAG | physical operators in the stage (BatchScan, WholeStageCodegen[id], Exchange, HashAggregate, Sort, ShuffledRowRDD) | dag |
ties the stage back to exact operators |
| Stage detail — Summary Metrics for Tasks | per-metric distribution at Min/25th/Median/75th/Max: Duration, GC, Scheduler Delay, (De)serialization, Getting Result, Peak Exec Memory, Input, Output, Shuffle Read (+ fetch wait, remote), Shuffle Write (+ time), Spill (mem), Spill (disk) | percentile |
the #1 skew/straggler tool — Median≪Max ⇒ skew; high spill ⇒ low memory |
| Stage detail — Aggregated Metrics by Executor | per-executor rollup: Exec Id, Address, Task Time, Total/Failed/Killed/Succeeded Tasks, excluded flag, Input/Output/Shuffle R/W, Spill mem/disk | table |
skew (one task) vs bad node (one executor slow across all tasks) |
| Stage detail — Accumulators | named accumulators + aggregated values | table |
user counters |
| Stage detail — Tasks table | per-task: Index, ID, Attempt, Status, Locality, Exec Id, Host, Launch Time, Duration, Scheduler Delay, Deser Time, GC, Result Ser, Getting Result, Peak Exec Mem, Accumulators, Input, Output, Shuffle Read fetch-wait/size/remote, Shuffle Write time/size, Spill mem/disk, Errors | table |
row-level forensics — the exact straggler, its host, its error |
| Stage detail — task event timeline | one bar per task, segmented by phase: Scheduler Delay, Task Deser, Shuffle Read, Executor Computing, Shuffle Write, Result Ser, Getting Result | gantt (stacked) |
where time actually goes — lots of non-compute ⇒ over-partitioning/GC/scheduling |
| Storage — landing | cached RDD/DF: Name/Id, Storage Level, Cached Partitions, Fraction Cached, Size in Memory, Size on Disk | table |
confirm full materialization; spotted spill |
| Storage — detail | per-partition: Block Name, Storage Level, Size mem/disk, hosting executor(s); data-distribution-on-cluster table | table |
uneven partition distribution, eviction |
| Environment | Runtime Info (Java/Scala), Spark Properties, Hadoop Properties, System Properties, Classpath Entries, Resource Profiles | kv |
ground truth for "what config was actually in effect" |
| Executors | per-executor + driver: Exec Id, Address, Status, RDD Blocks, Storage Mem (on/off heap), Disk Used, Cores, Active/Failed/Complete/Total Tasks, Task Time (GC Time), Input, Shuffle R/W, Logs; per-row Thread Dump / Heap Histogram / Flame Graph | table (+ flame) |
cluster health/balance; dead nodes, GC-bound execs, hot threads |
| SQL/DataFrame — query list | ID, Description, Submitted, Duration, Job IDs, status | table |
bridge logical query → physical jobs |
| SQL/DataFrame — query plan DAG | operator boxes grouped in blue WholeStageCodegen[id] boxes; per-node metrics: output rows, peak memory, spill, data size, scan/metadata time, shuffle bytes/records/time, fetch wait, sort time, hash-probe, build-side size, commit time, Python UDF bytes | dag (metric-bearing) |
where rows explode, where memory peaks/spills, which operator dominates |
| SQL/DataFrame — plan text | Parsed → Analyzed → Optimized logical → Physical plan (codegen *(id) markers, AdaptiveSparkPlan/isFinalPlan) |
plan-tree |
confirm Catalyst/AQE choices (pushdown, join type, coalescing) |
| Structured Streaming — query list | Name, Status, ID, Run ID, Submitted, Duration, Avg Input/sec, Avg Process/sec, Latest Batch, Error | table |
which stream is failing / falling behind |
| Structured Streaming — statistics | Input Rate, Process Rate, Input Rows, Batch Duration (each line+hist); Operation Duration stacked (addBatch/getBatch/latestOffset/queryPlanning/walCommit); state metrics (total/updated rows, state memory, dropped-by-watermark, watermark gap) | line+hist + gantt(stacked) |
input>process ⇒ backlog; batch-dur→trigger ⇒ falling behind; state growth ⇒ OOM risk |
| Spark Connect | sessions table (User, session id, Start/Finish, Duration, executed requests); request/SQL stats linking to jobs/SQL | table |
attribute jobs back to a remote client session |
| JDBC/ODBC (Thrift) | sessions (User, IP, Session id, Start/Finish, Duration, Total Execute); SQL stats (User, Job id, Group id, Start/Finish/Close, Exec/Duration time, Statement, State, Detail) | table |
BI admin: who ran what; compile vs exec vs fetch |
| (legacy) DStream Streaming | per-batch Scheduling Delay, Processing Time, Total Delay; batches table | line+hist + table |
superseded by Structured Streaming tab |
1B. Databricks-specific additions / changes
| View | Data it shows | Visual form | Why useful |
| Embedded Spark UI (compute page) | the full Spark tab set, proxied through control plane; live on running clusters, history-server replay (~last 1k jobs) on terminated; executor loss reason (decommission); Master worker-decommission surface | (all of 1A) | same Spark UI without driver network access; gated by workspace ACLs |
| Query Profile — graph | operator DAG, leaves→results; each node shows switchable Time / Memory peak / Rows; highlight most-expensive nodes; zoom/search/filter; preview card | dag (metric-bearing, cost-heat) |
visual hotspot finder for a single query |
| Query Profile — tree | condensed hierarchy ordered slowest-first | plan-tree |
slowest operators at a glance |
| Query Profile — overview | status; user/compute/runtime; UUID; SQL; wall-clock "where time went" (waiting-for-compute / in-queue / scheduling / optimizing+pruning / executing / result-fetch); aggregated task time; rows returned/read; bytes read (+from cache); files/partitions read; files/bytes pruned; bytes spilled to disk; % task time in Photon; Top operators ranked by cost; Performance Insights (beta) | bar (segmented) + table |
one-screen diagnosis: spill, skew, pruning effectiveness, Photon coverage |
| Query Profile — per-node metrics | Time spent, Memory peak, Rows, Cumulative time, Tasks total time, Source/Sink rows; scan/IO (files read/pruned, sizes, bytes read/cache, metadata/scan time, partitions); Bytes spilled; # tasks; verbose CPU/cloud metrics | dag node body |
attribute cost to the relational operator |
| Query History | list (query text, status, start, duration, end, warehouse, user, rows, statement id); filters (user/date/compute/duration/status/type/tag); detail panel | table + detail |
find slow/expensive/failed queries across a warehouse |
| Query History — phase breakdown | segmented bar: Waiting-for-compute / Waiting-at-capacity (queue) / Scheduling / Compilation+pruning / Execution / Result-fetch (*_duration_ms) |
bar (stacked) |
is slowness provisioning, queuing, planning, execution, or fetch? |
| Photon indicators | Photon* operator nodes + adapter boundary; % task time in Photon; coloring (Spark UI: orange=Photon/blue=Spark; Query Profile: purple=Photon/grey=Spark) |
dag coloring + metric |
is the query vectorized; where did it fall back to JVM |
| AQE adaptive plan | AdaptiveSparkPlan root w/ isFinalPlan; initial vs current/final both shown; DAG mutates live; runtime stats isRuntime; AQEShuffleRead/CustomShuffleReader (Coalesced / skewed partition counts); dynamic SMJ→BHJ visible as plan diff |
dag (versioned) |
proof AQE re-planned; quantifies coalescing/skew-split |
| Lakeflow Jobs run views | Matrix (runs×tasks status cells, bar-height=duration), Graph (task DAG, click=metrics+query-history links), Timeline (Gantt) (task bars + deps + perf insights), List (sortable) | table/dag/gantt |
multi-task workflow monitoring; click-through to Query Profile |
| Compute Metrics (replaces Ganglia) | Hardware: CPU util+active nodes (stacked by mode), container memory, JVM heap, network rx/tx, free FS, mem+swap. Spark: server-load distribution heatmap/tiles, task metrics, shuffle bytes, task duration. GPU: per-GPU util/mem/fb/enc/dec | line/bar/heatmap |
I/O-bound (iowait), memory pressure/swap, skew (uneven tiles), GPU saturation |
| Thread dump → flame graph | per-executor/driver JVM thread snapshot rendered as async-profiler stacked samples (frame width ∝ time) | flame |
hot CPU paths / hung tasks without an external profiler |
| Streaming (notebook live + UI) | live line charts per micro-batch + Raw StreamingQueryProgress JSON; numInputRows, input/processed rows/sec, durationMs phases, stateOperators (rows/memory/rocksdb), per-source lag (Kafka offsets-behind, Kinesis ms-behind, Auto Loader files-outstanding) |
line+hist + kv |
backlog, batch-dur trend, state growth, per-source lag |
2. facett mapping — what covers it, the gap, what to build
Per Spark/Databricks view: the facett crate that covers it today, the gap, and what to build. Crates cited are real (see components.md and the workspace Cargo.toml members).
| Spark/DBX view | facett component today | Gap | What to build |
| Jobs / Stages list, Tasks table, Aggregated-by-Executor, Executors, Storage, Environment, Accumulators, Query History list, JDBC/Connect sessions | ✅ facett-grid Grid (virtualized, multi-sort, pinned cols, Σ footer, in-cell progress bar for Tasks-done, sparkline, chips for status/locality, color-scale heat for duration/spill); facett-table for tiny lists |
none structural — pure wiring. Need a host adapter Spark REST/JSON → RecordBatch. Minor: a "Tasks succeeded/total" dual-segment progress cell (grid has single-value bar) |
Wire it + a tiny condfmt "stacked progress" cell variant. ~zero new component. |
| Environment (kv), Storage detail, streaming Raw JSON | ✅ facett-grid/facett-table; facett-card CardView for the kv detail; facett-docview for big text |
none | wire kv into a CardView or 2-col grid |
| Job DAG, Stage operator DAG | 🟡 facett-pipeline PipelineView (L→R layered DAG, status colour, animated flow edges, job-progress bar, throughput edge labels) + facett-graph::DepGraphView (toposort columns, boxes/arrows, sublabel, click→drill-down) |
DAGs are stage-boxed (nested grouping), nodes need green=cached / grey=skipped semantics, and the job DAG groups RDDs inside stage boxes | Extend PipelineView/DepGraphView: node grouping boxes (WholeStageCodegen / stage clusters) + per-node cached/skipped state colour. Most of the layout already exists. |
| SQL query plan DAG, Query Profile graph, Photon coloured plan, AQE adaptive plan | 🟡 DepGraphView gives boxes+arrows+drill-down; PipelineView gives themed status colour |
The big gap: (a) per-node metric body (multi-line: rows / peak mem / spill / time), (b) cost-heat node fill (Time/Memory/Rows switch like Query Profile), (c) collapsible WholeStageCodegen/Photon groups, (d) initial-vs-final / isFinalPlan toggle showing two plan versions, (e) Photon orange/purple palette per node |
NET-NEW facett-plandag (or a heavy DepGraphView mode): metric-bearing, cost-heated, collapsible, versioned operator-DAG. See §4. |
| Plan text tree (parsed/analyzed/optimized/physical), Query Profile tree view | 🟡 facett-jobview (hierarchical tree with ASCII branch guides, fold/unfold, depth) is the closest shape; facett-docview for static plan text |
jobview rows are job entries, not generic indented metric rows; needs slowest-first ordering + per-node metric column | Extend facett-jobview into a generic indented metric tree (depth + guides + a value/heat column) — reuses its guide renderer + fold tween. |
| Jobs event timeline (executors + jobs/stages bars), Job-detail timeline | ❌ nothing — facett-plot is x/y line/bar only, no horizontal time-axis Gantt |
whole primitive missing: wall-clock axis, stacked swimlanes (executor lifecycle / job / stage / task bars), zoom/pan, colour-by-status | NET-NEW facett-timeline — a swimlane Gantt. See §4. (Tab label already exists in facett-tabbar: "🧵 Timeline".) |
| Per-stage task event timeline (stacked phase segments per task) | ❌ none | same Gantt primitive, but each bar is internally segmented into phases (scheduler delay / deser / shuffle read / compute / shuffle write / ser / getting-result) with a phase legend | facett-timeline with segmented bars + legend (one component covers both timeline kinds). |
| Lakeflow run Gantt + Matrix + Graph | Matrix → 🟡 facett-matrix (repo×aspect coloured cells — same shape as runs×tasks!); Graph → DepGraphView/PipelineView; Timeline → facett-timeline (new); List → facett-grid |
matrix bar-height-by-duration; timeline = the new component | reuse facett-matrix for the run matrix (cell colour + a height/intensity tweak); rest covered above |
| Summary Metrics for Tasks (Min/25/Median/75/Max percentiles) | 🟡 facett-stat StatGridView (label + value + inline bar + sparkline + verdict) |
no box/percentile cell; can't show 5-number distribution or the Median≪Max skew signal | Extend facett-stat with a percentile/box-plot cell (5-number horizontal box). Small, high value (skew detection). |
| Structured Streaming statistics (Input/Process Rate, Input Rows, Batch Duration line+hist) | 🟡 facett-plot PlotView (line/bars via egui_plot) |
missing the paired histogram beside each timeline, and the stacked "Operation Duration" band | Extend facett-plot: a line+hist paired layout + a stacked-area kind (egui_plot supports both). |
| Compute Metrics (CPU/mem/heap/network/FS line charts) | ✅ facett-plot line/area; ✅ facett-syschart for node badges |
server-load heatmap/tiles by node | facett-plot covers the lines; the per-node load heatmap → facett-matrix (node×time coloured cells) or a facett-grid heat column |
| Streaming query list, Connect/JDBC sessions, state-operator metrics | ✅ facett-grid |
none | wire |
| Thread dump → flame graph | ❌ none | flame graph (stacked samples, width∝time) is its own primitive | NET-NEW facett-flame (lower priority) — recursive stacked rects, zoom-on-click. Could share the facett-timeline painter (both are nested horizontal rects). |
| Executors thread dump (raw), driver/executor logs | ✅ facett-log (ring log, kind-coloured) + facett-console (monospace terminal, ANSI, scrollback) |
none | wire stderr/stdout/thread-dump text in |
| The whole multi-tab Spark UI shell | ✅ facett-tabbar (tab bar; already ships a "Timeline / DepGraph / Release" demo) + facett-core::FacetDeck (ui tabbed + wall_ui multi-pane wall, palette/zoom/clipboard routed) |
none | compose: one FacetDeck per Spark tab, exactly like facett-warehousedeck composes panes from existing widgets |
Coverage tally: of the ~25 distinct views, ~12 are already covered by facett-grid/table/card/log/console/tabbar/deck (pure wiring), ~7 need extensions to existing crates (facett-pipeline/DepGraphView, facett-stat, facett-plot, facett-jobview, facett-matrix), and ~3 are net-new (facett-timeline, facett-plandag, facett-flame).
3. Prioritized build roadmap (highest ROI first)
Ordered by (views unlocked) × (how Spark-defining it is) ÷ (build cost).
P0 — Wire the tables onto facett-grid (days, no new component)
The cheapest, biggest single win. A host adapter SparkRestApi/EventLog → Arrow RecordBatch plus Grid::from_record_batch(...) immediately delivers: Jobs, Stages, Tasks, Aggregated-by-Executor, Executors, Storage, Environment, Query History list, Connect/JDBC sessions — ~half the entire UI. facett-grid already has the in-cell progress bar (task progress), sparkline, status chips (locality/status), and colour-scale heat (duration/spill) these tables want. Only addition: a dual-segment "succeeded/total" progress cell (Spark's task bar shows succeeded+running+failed). Deliverable: a knut-side SparkGridAdapter
- a column-spec table. No facett source change beyond an optional condfmt variant.
P1 — facett-timeline (net-new): the Gantt swimlane
The single most Spark-defining visual that facett lacks. One component covers three views: the Jobs event timeline (executor lifecycle + job/stage bars), the per-stage task timeline (segmented phase bars), and the Lakeflow run Gantt. Wall-clock x-axis, stacked named lanes, segmented bars (per-phase colours from the active Theme), drag-pan + scroll-zoom (reuse facett-core::scroll_engine::SmoothScroll), click-select a bar → detail, hover tooltip. Facet + state_json like everything else. The "🧵 Timeline" tab already exists in facett-tabbar. See §4.
P2 — facett-plandag (net-new) / heavy DepGraphView mode: metric-bearing operator DAG
The SQL query plan / Databricks Query Profile graph — the other signature Databricks surface. Builds on the existing DepGraphView toposort + PipelineView theming, adding: (a) per-node metric body (rows / peak mem / spill / time), (b) cost-heat fill with a Time/Memory/Rows switch (Query Profile's core interaction), (c) collapsible WholeStageCodegen/Photon groups, (d) Photon orange/purple node palette, (e) initial-vs-final isFinalPlan toggle (two plan versions). High value, higher cost — schedule after the Gantt. See §4.
P3 — Targeted extensions (small, each unlocks one signature view)
facett-statpercentile/box cell → Summary Metrics for Tasks + skew detection (Median≪Max). Smallest change, classic Spark tuning view.facett-plotline+hist pair + stacked-area → Structured Streaming statistics + Operation Duration band + compute-metrics charts.facett-jobviewgeneric indented metric-tree mode → plan text tree / Query Profile tree view (reuses its branch-guide + fold tween).facett-matrixheight/intensity tweak → Lakeflow run matrix + compute server-load heatmap (node×time).
P4 — facett-flame (net-new, long-tail): the thread-dump flame graph
Lowest frequency, but the only true flame surface (Executors → Thread Dump → Flame Graph). Recursive stacked rects, width∝samples, zoom-on-click. Cheap if it shares the facett-timeline nested-rect painter (a flame graph is a depth-stacked Gantt without a time axis) — which is a reason to build P1 first.
P5 — Compose the shell
A SparkUiDeck à la facett-warehousedeck: one facett-tabbar across Spark tabs, each tab a FacetDeck of the widgets above, in wall_ui (multi-pane) or tabbed mode. Zero new components — pure composition of P0–P4, exactly the warehousedeck pattern.
4. The 2–3 net-new facett components this implies
NEW #1 — facett-timeline (a swimlane Gantt) — P1, build first
Why new: facett has no horizontal time-axis view. facett-plot is x/y line/bar; PipelineView/DepGraphView are left-to-right topological (not temporal) DAGs; facett-jobview is a vertical list. A Gantt — bars positioned and sized by wall-clock time across stacked lanes — is a genuinely new primitive.
Model (source-agnostic, mirrors the PipelineAdapter seam):
TimelineModel { lanes: Vec<Lane>, t0, t1 } // wall-clock window
Lane { label, rows: Vec<Bar> } // e.g. "executor-3", "stage 7"
Bar { start, end, status, label, // status → Theme colour
segments: Vec<Segment> } // optional intra-bar phases
Segment { kind, fraction } // scheduler-delay/compute/shuffle…Covers: Jobs event timeline, Job-detail timeline, per-stage task timeline (segmented bars = the phase legend), Lakeflow run Gantt. Reuses: Theme colours, effects::glow_rect for the selected bar, scroll_engine::SmoothScroll for pan/zoom, the Facet+state_json+FacetCaps contract. Caps: selectable
- themeable + resizable + scalable (zoom = time-axis zoom). Headless-testable: inject bars → assert laid-out bar rects +
state_json. A flame graph (NEW #3) is the same nested-rect painter without a time axis, so build this first.
NEW #2 — facett-plandag (metric-bearing, cost-heated, collapsible operator DAG) — P2
Why new (vs extending): the SQL plan / Query Profile graph wants four things no current graph facet has at once: a multi-line metric body inside each node, cost-heat fill with a Time/Memory/Rows switch, collapsible operator groups (WholeStageCodegen / Photon), and a two-version isFinalPlan (AQE) toggle. DepGraphView gives the toposort columns + boxes + drill-down; PipelineView gives themed status. This composes both (the established "compose facett_graph, paint our own" pattern from facett-pipeline.md) and adds the metric/heat/collapse/version layer. Could ship as a heavy DepGraphView mode instead of a crate — decide at build time; the capabilities are what matter. Covers: SQL query plan DAG, Databricks Query Profile graph, Photon plan, AQE adaptive plan. For huge plans, it is the natural first consumer of facett-graphview's vello render engine + its stubbed LOD/culling/clustering (collapse dense subgraphs into super-nodes).
NEW #2b — the constraint overlay (bottleneck badge · downstream emphasis · critical path)
Design (what shipped). The operator DAG is a DAG, so it has the same constraint structure a funnel Plan-DAG has — a PlanDagView now surfaces where a plan is constrained, additively, over the metric/heat/collapse/version layer. A pure analysis on the resolved graph (facett-plandag/src/model.rs):
Resolved::bottlenecks() -> Vec<Bottleneck{ id, fan_in, fan_out, downstream }>— ranks every node by its transitive-descendant cone (downstream= all work stalled behind it) →fan_out→fan_in→ id, sobottlenecks().first()is the single biggest constraint.Resolved::top_bottleneck()— the highest-ranked node with an incoming edge (skips the pure-source root whose cone is trivially the whole plan — the generic analog of the funnel "mother idea" a renderer filters out). This is the "clear this next" node the view badges.Resolved::critical_path()— the longest producer→consumer chain (structural longest path, deterministic id tie-break).
This mirrors funnel_core::PlanDag::{bottlenecks, critical_path}() — not a funnel-core dependency: facett-plandag renders a host-fed generic PlanModel (≠ a funnel Funnel) and is pinned to the buildable dependency-island of facett-core/facett-graph (the same reason it inlines facett-pipeline's status colours — see its Cargo.toml). The nornir-viz layer maps a funnel PlanDag → this PlanModel; the ranking rule (reach → fan_out → fan_in → id) is byte-identical so the two agree by construction.
The picture (facett-plandag/src/lib.rs, render), all gated on state.show_constraints (subtle + on by default; a header ⚑ constraints toggle / Msg::SetConstraints turns it off — see the SVG below):
- a warm ⚑ badge pins
top_bottleneckon the node's top edge + a stronger border, with its own AccessKit node (get_by_label("⚑ bottleneck · <title>")) carrying the downstream reach as the numeric value; - every node gets an outer accent ring whose weight/alpha grows with its
downstreamreach (how much work is stalled behind it); - the critical-path chain rides the accent colour with a soft weight beneath the arrow.
Guide (host / robot). The overlay reads the SAME resolve() the painter and state_json read, so the pixels ARE the data: state_json gains bottlenecks (the full ranking), top_bottleneck (the badged id), critical_path (the chain) and show_constraints; top_bottleneck + show_constraints also join portable_state/load_state. A robot never touches egui — it asserts state_json["top_bottleneck"], or (render-proof) get_by_label the badge in the AccessKit tree. facett-plandag/tests/render.rs does exactly that on a wide-vs-deep DAG: the Deep node (fan_out 1, downstream 4) is badged over the Wide node (fan_out 3, downstream 3) — RED-when-broken if the reach ranking ever flips to fan-out-primary — plus overlay-off drops the badge.
For-idiots. Which box do I fix first? The one with the little ⚑ flag — it has the most work piled up behind it. The brighter a box's outer ring, the more the plan is waiting on it; the blue arrows trace the longest road through the plan (its schedule length). Flip the ⚑ constraints switch off if you just want the raw metrics.
NEW #3 — facett-flame (flame graph) — P4, optional / long-tail
Why new: depth-stacked sample rectangles (width∝time) with zoom-on-click — no facett component does this. Cheap if it reuses facett-timeline's nested-rect painter (a flame graph ≈ a depth-stacked Gantt with no time axis). Covers: Executors → Thread Dump → Flame Graph (the only true flame surface). Lowest priority — build only after the timeline painter exists to share.
Net-new count: 2 essential (facett-timeline, facett-plandag) + 1 optional (facett-flame). Everything else is facett-grid wiring or a bounded extension to facett-stat / facett-plot / facett-jobview / facett-matrix / facett-pipeline+DepGraphView, all composed under the existing facett-tabbar + FacetDeck shell.
Sources
Apache Spark Web UI: spark.apache.org/docs/latest/web-ui.html; apache/spark core/.../ui/jobs/StagePage.scala (verbatim task columns + the TIMELINE_LEGEND phase labels); The Internals of Spark Core (japila). Databricks: docs.databricks.com Query Profile, Query History (+ system.query.history schema), Photon, AQE (optimizations/aqe), cluster / compute metrics, Spark-UI debugging guide, Lakeflow job monitoring; the Photon SIGMOD 2022 paper (operator class names); Databricks engineering blogs (Structured Streaming UI in Spark 3.0, DBSQL Query Profile, Faster SQL with AQE). Caveats carried from research: Photon operator class names are from the SIGMOD paper not the docs; two Photon palettes (Spark UI orange/blue vs Query Profile purple/grey); AQE node naming drift (CustomShuffleReader in DBX docs vs AQEShuffleRead in OSS); Query-History column set partly inferred from filters + detail panel; ~1k-job replay limit is community-sourced, not an SLA.