facett-core/src/render/gputurn.rs
//! **THE GPU TURN — one wgpu device bring-up at a time, in this process and on this box.**
//!
//! This is the one writer for a rule the workspace learned three separate times, and
//! re-learned on 2026-08-31 because it was only ever enforced in one crate:
//!
//! | when | where | what happened |
//! |---|---|---|
//! | 2026-08-27 | `facett-map` kittest files | two `Harness::render()` bring-ups on parallel `#[test]` threads → `signal: 11`, 2/5 and 3/5 |
//! | 2026-08-27 | `facett-map` `line_gpu` | seven raw `request_device` calls, one per test → SIGSEGV 1/8 |
//! | 2026-08-31 | `facett-map` × `facett-geomap` | the same crash **across processes**, which the per-binary `Mutex` could not see |
//! | 2026-08-31 | `facett-core` lib tests | **20 concurrent `render::gpu::*` device bring-ups deadlocked the binary** — 13 min, GPU at 0 %, every thread in `futex_do_wait`, 632 MiB of VRAM held. Reproduced in BOTH profiles in one day; the run had to be killed by hand both times |
//! | 2026-08-31 | `facett-map3d` `ssao_proof` | `signal: 11` on a 2-test file, in a whole-workspace release run |
//! | 2026-09-03 | `facett-graph3d` `neon` LIB tests | `cargo test --workspace --lib` died in this crate. `facett_graph3d neon:: --test-threads=2` → SIGSEGV **17 of 30** runs, `--test-threads=1` → **0 of 15**. Two `Harness::render()` bodies, five kittest devices, no turn — and the ONLY `.render()` in any facett `--lib` test, so it was the whole workspace's verdict |
//!
//! The 2026-09-03 row is the same lesson a third time, in the one place nobody had looked:
//! a `#[cfg(test)]` module inside a **library source**. `facett-map`'s guard
//! (`tests/one_gpu_turn_is_the_only_one.rs`) scans `facett-map/tests/` — it cannot see
//! another crate's `src/`. A workspace sweep of all 395 `*/src/*.rs` found exactly five
//! files that open a device from test code: `facett-graph3d/src/neon.rs` (the crash) and
//! four feature-gated hand-rolled `compute_device()` copies in
//! `facett-graphview/src/gpu_layout.rs` and `facett-map3d-flight/src/{cdlod,extrude_gpu,scatter_gpu}.rs`.
//! All five take the turn as of 2026-09-03.
//!
//! `facett-map` fixed its half in `tests/common/one_gpu.rs` and wrote a guard to keep the
//! lock single. But the lock lived in a **test file of one crate**, so `facett-core`'s own
//! eight hand-rolled device probes and `facett-map3d`'s device tests could not reach it —
//! and both then produced exactly the failures the lock exists to prevent. A rule enforced
//! in one crate out of four is not enforced; it is documented. So the turn moves HERE,
//! where every facett crate's tests can take it, and `facett-map`'s file becomes a
//! re-export rather than a second copy (LAW 5 — two mutexes over one device serialise
//! nothing, and the second one reads as coverage).
//!
//! # Two halves, taken in this order and dropped in the reverse one
//!
//! 1. **In-process** — a `Mutex`. Thread against thread: the 2026-08-27 SIGSEGVs and the
//! 2026-08-31 `facett-core` deadlock.
//! 2. **Across processes** — an `flock(2)`, **machine-wide**. `cargo test` runs a crate's
//! test binaries one after another, so the process-against-process collision needs two
//! cargo invocations at once — two lanes on one box — and those have different target
//! dirs. A turn scoped to the build tree would be green in every experiment while
//! covering only the case that cannot happen. The resource is one RTX 4090; the lock is
//! scoped to the resource. `flock` and not a lock file because the kernel releases it
//! when the fd closes **or the process dies**, so a crashed test leaves nothing behind.
//!
//! **A wedged holder must not wedge the suite.** The cross-process wait is `LOCK_NB`
//! against a deadline ([`WAIT_LIMIT`]), never a blocking `LOCK_EX`: a GPU test that hangs —
//! and they do, see the table — would otherwise stop every other test binary behind it. On
//! timeout the turn is taken WITHOUT the cross-process half and says so on stderr. That is
//! the behaviour this module replaces: degraded, loudly, never hung.
//!
//! # Use
//!
//! ```ignore
//! #[test]
//! fn something_that_opens_a_device() {
//! let _gpu = facett_core::render::gputurn::hold();
//! …
//! }
//! ```
//!
//! Hold it across the bring-up at least. It is **not re-entrant** (a plain `Mutex`): a
//! caller that already holds it must not take it again, which is why the crate-level
//! device helpers take it *inside* themselves rather than asking every call site to.
//!
//! `$FACETT_GPU_LOCK` overrides the lock file's path — point two runs at different files
//! to take the cross-process serialisation off on purpose (reproducing the crash is a
//! legitimate thing to want).
/// How long a caller waits for the CROSS-PROCESS half before giving up on it and running
/// anyway. Generous next to a device bring-up (milliseconds) and a pixel census (a second
/// or two), short next to the wedges this must not join.
pub const WAIT_LIMIT: std::time::Duration = std::time::Duration::from_secs(180);
static ONE_GPU_AT_A_TIME: std::sync::Mutex<()> = std::sync::Mutex::new(());
/// The turn. Holding this value holds the lane; dropping it releases both halves, the
/// cross-process one first (fields drop in declaration order).
pub struct Turn {
/// `Some` while this process owns the file lock; `None` when the deadline passed and
/// the caller was let through without it (or on a platform with no `flock`).
_across_processes: Option<std::fs::File>,
_in_process: std::sync::MutexGuard<'static, ()>,
}
/// **Take the turn** for the rest of the caller's scope.
///
/// Poison is taken rather than propagated: a test that panicked while holding this has
/// already failed and reported its own reason, and turning that into a second failure in
/// every sibling test would hide it.
#[must_use]
pub fn hold() -> Turn {
let in_process = ONE_GPU_AT_A_TIME.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
Turn { _across_processes: flock_turn(), _in_process: in_process }
}
/// The one lock file for this machine's GPU turn. `$FACETT_GPU_LOCK` wins, then
/// `$XDG_RUNTIME_DIR`, then the temp dir — a zero-byte runtime lock belongs in a runtime
/// directory, not in a build tree.
#[must_use]
pub fn lock_path() -> std::path::PathBuf {
if let Some(p) = std::env::var_os("FACETT_GPU_LOCK") {
return std::path::PathBuf::from(p);
}
std::env::var_os("XDG_RUNTIME_DIR")
.map(std::path::PathBuf::from)
.filter(|d| d.is_dir())
.unwrap_or_else(std::env::temp_dir)
.join("facett-one-gpu.lock")
}
#[cfg(unix)]
fn flock_turn() -> Option<std::fs::File> {
use std::io::Write as _;
let path = lock_path();
let file =
std::fs::OpenOptions::new().create(true).truncate(false).write(true).open(&path).ok()?;
let fd = std::os::fd::AsRawFd::as_raw_fd(&file);
let deadline = std::time::Instant::now() + WAIT_LIMIT;
loop {
// LOCK_EX | LOCK_NB: never block, so a wedged holder cannot wedge this run.
if unsafe { libc::flock(fd, libc::LOCK_EX | libc::LOCK_NB) } == 0 {
return Some(file);
}
if std::time::Instant::now() >= deadline {
// Loud, and on stderr where a SIGSEGV's own output would be: the run is now
// back to the racy behaviour this module exists to remove, and a later
// `signal: 11` should be read next to this line.
let _ = writeln!(
std::io::stderr(),
"[one-gpu] waited {}s for {} and gave up — this bring-up is UNSERIALISED \
against other processes. Something is holding the GPU turn far longer \
than a render should.",
WAIT_LIMIT.as_secs(),
path.display()
);
return None;
}
std::thread::sleep(std::time::Duration::from_millis(25));
}
}
/// No `flock` off unix: the in-process half still holds, and that is the half every
/// measured crash in the table above needed.
#[cfg(not(unix))]
fn flock_turn() -> Option<std::fs::File> {
None
}
#[cfg(feature = "wgpu")]
pub use probe::{open, OnSoftware, Probe};
/// **THE headless device bring-up** — the one writer the eight probes now share.
///
/// Before this module, `facett-core` opened a wgpu device in **eight** hand-rolled
/// places: five `headless_device()`/`compute_device()` copies in `render::gpu::*` tests
/// and three library entry points ([`crate::render::l1::render_overlay`],
/// [`crate::render::gpu::offscreen_render`],
/// [`crate::render::gpu::render_graph_offscreen`]). Every copy repeated the same
/// twenty lines and every copy got two things wrong:
///
/// * **no turn** — see this module's table; and
/// * **`request_adapter(PowerPreference::default())`**, which is `LowPower` and lets
/// wgpu's internal fallback ordering hand back `llvmpipe` when the real card cannot
/// give a device. MEASURED on oden 2026-08-31: with the 4090 saturated by concurrent
/// lanes, a suite selected `llvmpipe (LLVM 21.1.8, Vulkan, Cpu)`. It did not error. It
/// rendered **different pixels** and then passed or failed for reasons unrelated to the
/// code under test.
///
/// So a probe here does three things no copy did: it takes the turn and hands it back to
/// the caller (so it covers the whole test body, not just the bring-up), it goes through
/// [`request_best_adapter`](crate::render::gpu::request_best_adapter) — facett's own
/// policy, which ranks software **last** — and it **names the adapter it got on stderr,
/// every time**, then applies an [`OnSoftware`] policy if that name is a software
/// rasteriser. libtest CAPTURES stderr, so that receipt surfaces under `--nocapture` or
/// in the replayed output of a FAILING test — which is when it is needed: a red whose
/// output carries no `[<who>] adapter — …` line opened no device, and a red that names
/// llvmpipe is not a red about the code. In a captured green run nobody reads stderr at
/// all, which is exactly why the [`OnSoftware`] policy does the work and the print does
/// not. MEASURED: a whole-workspace `cargo test` shows zero of these lines and is none
/// the worse for it — the 3 309 passes were still all on the 4090, because a software
/// adapter could not have got that far.
#[cfg(feature = "wgpu")]
pub mod probe {
/// What a probe does when the adapter it actually got is a **software rasteriser**.
///
/// Never "carry on quietly" — that is the behaviour being removed. Both arms are
/// overridden by `$FACETT_ALLOW_SOFTWARE_GPU=1` for a host that genuinely has no card.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum OnSoftware {
/// **Panic.** For a probe whose whole output is a pixel measurement: a pass on
/// llvmpipe is as meaningless as a fail, so the run must say so and stop.
Refuse,
/// **Return `None`.** For the three library entry points, which already return
/// `Option` and whose every caller already reads `None` as "no GPU here, skip".
/// llvmpipe *is* "no GPU" for the purpose of a pixel proof — and on a genuinely
/// CPU-only host this is also the correct production answer (L1 degrades to L0
/// rather than panicking inside an app).
Skip,
}
/// A live headless device **and the turn that makes it exclusive**.
///
/// Field order is drop order: `device`/`queue`/`adapter` go first, `_turn` last, so
/// the lane is still held while wgpu tears the device down.
pub struct Probe {
pub device: wgpu::Device,
pub queue: wgpu::Queue,
pub adapter: wgpu::Adapter,
/// The adapter, named — the same string that was printed to stderr. Kept so a
/// test can put the adapter into its OWN failure message.
pub named: String,
_turn: super::Turn,
}
/// Everything a [`Probe`] is except the device and queue — kept alive so the lane
/// stays held. What [`Probe::split`] hands back.
pub struct ProbeGuard {
pub adapter: wgpu::Adapter,
pub named: String,
_turn: super::Turn,
}
impl Probe {
/// `true` when this device is a software rasteriser that was let through by
/// `$FACETT_ALLOW_SOFTWARE_GPU`.
#[must_use]
pub fn is_software(&self) -> bool {
crate::render::adapter::is_software_rasteriser(&crate::render::gpu::facts_of(
&self.adapter.get_info(),
))
}
/// Split into `(device, queue, guard)` for the call sites that already bind that
/// shape. **The guard must be bound**, not `let _ = …`: dropping it releases the
/// lane, and a `_`-pattern drops it immediately.
///
/// `Probe` itself is the stronger form — its field order keeps the lane held
/// through wgpu's device teardown, whereas a `let (device, queue, _gpu) = …`
/// drops the locals in reverse and so releases the lane a moment earlier. Every
/// crash in this module's table was a *bring-up* race, which both forms cover.
#[must_use]
pub fn split(self) -> (wgpu::Device, wgpu::Queue, ProbeGuard) {
(
self.device,
self.queue,
ProbeGuard { adapter: self.adapter, named: self.named, _turn: self._turn },
)
}
}
/// `$FACETT_ALLOW_SOFTWARE_GPU` — the one opt-out, read the same way by both arms.
fn software_allowed() -> bool {
matches!(
std::env::var("FACETT_ALLOW_SOFTWARE_GPU").ok().as_deref(),
Some("1" | "true" | "on" | "yes")
)
}
/// **Take the turn and open a headless device**, or `None` when this host has no
/// usable adapter (CPU-only CI — the caller self-skips rather than failing falsely).
///
/// `who` is the probe's name: it labels the device, prefixes the adapter line on
/// stderr and names the failure. `limits` receives the chosen adapter and returns the
/// limits to ask for — or `None` to decline this adapter entirely, which is how
/// `particles` keeps its compute-shader / storage-buffer preconditions.
///
/// # Panics
///
/// With [`OnSoftware::Refuse`], if the adapter is a software rasteriser and
/// `$FACETT_ALLOW_SOFTWARE_GPU` is unset.
pub fn open(
who: &str,
on_software: OnSoftware,
limits: impl FnOnce(&wgpu::Adapter) -> Option<wgpu::Limits>,
) -> Option<Probe> {
// THE GPU TURN, taken FIRST — before the adapter request, because the bring-up
// is itself the thing that raced. It travels out in `Probe` so it covers the
// caller's whole body: MEASURED 2026-08-31, serialising only the bring-up still
// left nine `label_collide` devices alive at once and the binary deadlocked for
// 13 minutes (20 threads in `futex_do_wait`, GPU at 0 %, 632 MiB held). The same
// 74 `render::gpu::*` tests pass in 5.2 s serialised. Not re-entrant: a caller
// holding a `Probe` must not also call `hold()`.
let turn = super::hold();
let instance = wgpu::Instance::default();
// The POLICY, not `request_adapter(&Default::default())`: it enumerates and ranks
// software last, so a busy 4090 is a wait, not a silent demotion to llvmpipe.
let adapter = crate::render::gpu::request_best_adapter(
&instance,
crate::render::gpu::preferred_backends(),
)?;
let info = adapter.get_info();
let facts = crate::render::gpu::facts_of(&info);
let named = facts.describe();
// ALWAYS, and on stderr where a SIGSEGV's own output would be. This line is the
// receipt: no line, no device.
eprintln!("[{who}] adapter — {named}");
if crate::render::adapter::is_software_rasteriser(&facts) && !software_allowed() {
match on_software {
OnSoftware::Refuse => {
// One writer for the message, shared with the sites that get their
// device from egui_kittest and can only inspect it afterwards.
crate::render::gpu::refuse_software_adapter(who, &info);
}
OnSoftware::Skip => {
eprintln!(
"[{who}] SKIPPING — {named} is a SOFTWARE rasteriser. Whatever \
this rendered would be about llvmpipe, not about the GPU lane. \
Set FACETT_ALLOW_SOFTWARE_GPU=1 if this host really has no card."
);
return None;
}
}
}
let required_limits = limits(&adapter)?;
let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor {
label: Some(who),
required_features: wgpu::Features::empty(),
required_limits,
memory_hints: wgpu::MemoryHints::default(),
experimental_features: wgpu::ExperimentalFeatures::disabled(),
trace: wgpu::Trace::Off,
}))
.ok()?;
Some(Probe { device, queue, adapter, named, _turn: turn })
}
/// The common ask: `downlevel_defaults()`, no preconditions. Six of the eight.
#[must_use]
pub fn downlevel(_: &wgpu::Adapter) -> Option<wgpu::Limits> {
Some(wgpu::Limits::downlevel_defaults())
}
}
#[cfg(test)]
mod tests {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
/// Run the overlap experiment once with a given "take the turn" step, and report
/// whether a second thread got inside while the first was still there.
///
/// Extracted so the SAME experiment can be run against a **no-op** guard. That is the
/// red arm: if the no-op run does not overlap, the experiment is blind and proves
/// nothing about the real one — which is the failure mode that let six twinned
/// mutexes read as coverage for four days (LAW 2).
/// `G: 'static` only — deliberately NOT `Send`: [`super::Turn`] holds a `MutexGuard`
/// and so is `!Send`. Each thread makes and drops its own; nothing crosses.
fn overlap_seen<G: 'static>(take: fn() -> G) -> bool {
let inside = Arc::new(AtomicBool::new(false));
let overlapped = Arc::new(AtomicBool::new(false));
let first = take();
inside.store(true, Ordering::SeqCst);
let (i2, o2) = (inside.clone(), overlapped.clone());
let t = std::thread::spawn(move || {
let _second = take();
if i2.load(Ordering::SeqCst) {
o2.store(true, Ordering::SeqCst);
}
});
// Give the other thread every chance to barge in.
std::thread::sleep(std::time::Duration::from_millis(150));
inside.store(false, Ordering::SeqCst);
drop(first);
t.join().expect("the second taker finished");
overlapped.load(Ordering::SeqCst)
}
/// **The turn really excludes, and the experiment that says so can fail.**
///
/// Two arms over one body:
/// * RED — the same two threads with `()` in place of the guard **must** overlap. If
/// they do not, the timing is wrong and the green arm below is vacuous.
/// * GREEN — with [`super::hold`], they must not.
#[test]
fn a_second_taker_waits_for_the_first_to_let_go() {
assert!(
overlap_seen(|| ()),
"RED ARM DID NOT FIRE: with NO guard at all the second thread still failed to \
get inside, so this experiment cannot detect a lock that excludes nothing. \
Whatever the green arm below reports is meaningless until this fires."
);
assert!(
!overlap_seen(super::hold),
"two holders were inside the turn at once — the lock excludes nothing, which \
is exactly the shape six twinned mutexes had on 2026-08-27"
);
}
/// **The CROSS-PROCESS half really excludes** — the half that no in-process `Mutex`
/// can provide, and the one the 2026-08-31 `facett-map` × `facett-geomap` crash needed.
///
/// `flock(2)` associates a lock with the *open file description*, not with the
/// process, so a second `open()` of the same path conflicts with the first even from
/// the same thread (`flock(2)`: "these file descriptors are treated independently …
/// an attempt to lock the file using one of these file descriptors may be denied by a
/// lock that the calling process has already placed via another"). That gives a real
/// red/green with no second process and no GPU:
///
/// * RED — with no turn held, a fresh `LOCK_EX | LOCK_NB` **succeeds**;
/// * GREEN — while the turn is held, the identical call **fails**.
///
/// Both arms are required. The red one is what catches a `lock_path()` that has
/// drifted, an `flock` that silently never ran, and the case where `hold()` was
/// quietly reduced to its mutex.
#[cfg(unix)]
#[test]
fn the_cross_process_half_really_excludes() {
fn a_stranger_can_take_it(path: &std::path::Path) -> bool {
let f = std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(path)
.expect("the lock path is writable");
let fd = std::os::fd::AsRawFd::as_raw_fd(&f);
unsafe { libc::flock(fd, libc::LOCK_EX | libc::LOCK_NB) == 0 }
// `f` drops here, releasing whatever it took.
}
// RED ARM, on a private path so it never fights the box for the real turn: the
// identical call SUCCEEDS when nothing holds the file. No env var is touched —
// `$FACETT_GPU_LOCK` is process-global and sibling tests run on other threads.
let mine =
std::env::temp_dir().join(format!("facett-turn-selftest-{}.lock", std::process::id()));
assert!(
a_stranger_can_take_it(&mine),
"RED ARM DID NOT FIRE: a fresh flock on an unheld file {} was refused, so the \
green arm below cannot tell an flock that works from one that never ran",
mine.display()
);
let _ = std::fs::remove_file(&mine);
// GREEN ARM, on the REAL path. Holding the turn also holds this binary's
// in-process mutex, so no sibling test can be inside `hold()` while we look.
let turn = super::hold();
assert!(
turn._across_processes.is_some(),
"hold() did not take the cross-process half at all ({} — unwritable? or \
another process has held the box's turn for over {}s), so every run is \
serialised only against its own threads",
super::lock_path().display(),
super::WAIT_LIMIT.as_secs()
);
assert!(
!a_stranger_can_take_it(&super::lock_path()),
"a second open() of {} took the lock while the turn was held: the \
cross-process half excludes nothing, and two `cargo test` invocations on \
this box will bring up devices concurrently again",
super::lock_path().display()
);
drop(turn);
}
/// **The device probes cannot overlap** — the property the eight `render::gpu`
/// bring-ups actually need, asserted on the real entry point rather than on the
/// mutex underneath it.
///
/// Two threads both call [`probe::open`]; a counter records the highest number ever
/// inside at once. It must be 1. The red arm for the mechanism is
/// `a_second_taker_waits_for_the_first_to_let_go` above — this one is about the
/// wiring: a probe that forgot to take the turn, or that dropped it before returning,
/// shows up here and nowhere else.
///
/// Self-skips (never a false fail) when the host has no adapter.
#[cfg(feature = "wgpu")]
#[test]
fn two_device_probes_never_hold_the_card_at_once() {
use std::sync::atomic::AtomicUsize;
static LIVE: AtomicUsize = AtomicUsize::new(0);
static PEAK: AtomicUsize = AtomicUsize::new(0);
fn one_probe() -> bool {
let Some(p) = super::probe::open(
"gputurn-overlap-probe",
super::probe::OnSoftware::Skip,
super::probe::downlevel,
) else {
return false;
};
let now = LIVE.fetch_add(1, Ordering::SeqCst) + 1;
PEAK.fetch_max(now, Ordering::SeqCst);
// Long enough that an unserialised sibling would certainly be inside too.
std::thread::sleep(std::time::Duration::from_millis(120));
LIVE.fetch_sub(1, Ordering::SeqCst);
drop(p);
true
}
let t = std::thread::spawn(one_probe);
let a = one_probe();
let b = t.join().expect("the second probe finished");
if !a || !b {
eprintln!("[gputurn] no usable GPU adapter — skipping the probe overlap proof");
return;
}
assert_eq!(
PEAK.load(Ordering::SeqCst),
1,
"two probes held a device at the same time — `probe::open` is not taking the \
turn, or the `Turn` is being dropped before the `Probe` reaches the caller"
);
}
/// The lock file is a real, openable path — a `lock_path` that pointed at an
/// unwritable place would silently drop the cross-process half on every run.
#[test]
fn the_lock_path_is_writable_so_the_cross_process_half_is_not_silently_off() {
let _turn = super::hold();
let p = super::lock_path();
assert!(p.is_file(), "taking the turn did not create {} — flock never ran", p.display());
}
}