[jgoyvaerts]

When one client stalls or crashes the GPU, the other clients' work times out or is dropped by the firmware, and the driver treated that as their own failure: the compositor got ETIMEDOUT/EIO, recorded a sticky device loss and aborted. Plus a power hold that removes a ~1 ms wake-up from every frame. Six commits, all in the G17 backend:

• Bounded cross-VM grace for render timeouts. Compute already classifies a scheduler timeout (g17::timeout): work behind another VM's older unfinished kick is renewed up to MAX_RENEWALS times, and the oldest unfinished work is never excused. Renders now get the same rule: render and compute publications share one ordering counter, each render pair reports its oldest unfinished published group as a timeout head (also while it is quarantined, since its work holds the GPU until the firmware drains it, as compute keeps its head), and a render timeout behind another VM's older unfinished work is deferred within the same renewal budget. The culprit, being oldest, still fails at its first timeout.
• Spare innocent renders once the grace runs out. As for compute, a render still blocked behind another VM after MAX_RENEWALS is classified as spared, so its timeout fails only the fence instead of the VM.
• Keep a VM usable behind its spared render quarantine. A publication that times out behind a quarantine holding only spared work fails its own fence with EBUSY (not a device-loss error), so the VM recovers once the pair drains. Any guilty failure makes the quarantine guilty as before (KUnit test).
• Retire abandoned spared renders without failing clients. A spared front group that the existing proof shows abandoned (retired, never executed, stamps covering, no recovery pending, job list empty) is retired on its own with ECANCELED even with successors queued; spared work completed with ECANCELED no longer reports device loss.
• Replay every render group the firmware dropped unexecuted. After another VM's recovery the firmware can retire an innocent pair's queued groups without running them. The pair now parks all of them, in order, once every one carries the strict abandonment witness, and publishes them again front first: only the whole set, so nothing replayed runs after a live successor; a failed replay fails the parked successors with it. Work a recovery blamed is never replayed (including when the guilty queue ID cannot be resolved to retained work, unless an earlier recovery already spared it), and the per-command replay limit is unchanged. The decision is a pure function (g17/replay.rs) with KUnit tests.
• Hold the GPU powered for idle_off_delay_ms after work. On the M4 Pro (25G83) the GPU cores power off 2 ms after the last job, the firmware reads that delay only at boot, and every wake-up then costs about 1.1 ms before a job runs, which a 120 Hz desktop pays on every frame. A root-writable sysfs attribute idle_off_delay_ms (2-1000, default 2 = no hold, so nothing changes unless it is set) holds the GPU powered with the idle policy control the driver already uses around registration (DeviceControl 0x0a), and powers it off again after that many ms without submissions or work in flight.

Hardware tested on: Apple MacBook Pro (16-inch, M4 Pro, 2024), J616s / T6040, macOS 26.6.2 GPU firmware (25G83), booted by our own loader.

How tested: these commits have run in our daily kernels since 2026-10-19 to 10-21 (now 7.3.0-m4.12, with #408's T6040 support underneath), driving the desktop every day. Each commit message describes the failure observed on that tree before it (a stalled test client taking the compositor down; a quarantined innocent pair failing with EIO one second later; dropped groups failed instead of replayed); after review, four changes were built but have not run yet: the render head kept while quarantined and taken as the oldest unfinished group, a blamed group an earlier recovery spared staying spared, the KUnit form of the tests, and the knob's lower bound. The KUnit tests (replay decision, idle hold, spared/guilty quarantine) were built, not run: no QEMU here, and the driver is in use. The latency numbers above (1.2-1.4 ms for a tiny render after a 2 ms gap, 0.19 ms back to back) were measured on the M4 Pro.

Not tested / limits: not run on the A18 Pro (T8140, 26A428): the logic is backend-generic, but the firmware behaviours it reacts to (retiring an innocent pair's groups unexecuted after a recovery; the 2 ms idle power-off) were observed on 25G83 only. The default idle_off_delay_ms keeps the firmware's behaviour, so the power hold changes nothing until userspace sets it. No dedicated before/after recovery test was run on exactly this set. Region C in this tree has 2 at +0x9b8 (unk_9b8) and 40 at +0x9bc (idle_power_off_delay); which field sets the measured 2 ms was not established, so the power-hold commit names neither. A remaining case (an unpublished dependent packet inheriting ECANCELED is still reported as device loss) is not addressed here.

Base: integration/aurora (1aa2857509d); applies without #408. Built at every commit with W=1 (CONFIG_DRM_ASAHI=m, CONFIG_DRM_ASAHI_G16=y, KUnit enabled): no new warnings (the tip's own 8 fetch_update deprecations and, with KUnit, the unused MM static remain); checkpatch --strict clean apart from the generic MAINTAINERS note for the new g17/replay.rs (drm/asahi has no MAINTAINERS entry).

🤖 Generated with Claude Code

[fbalazsy]

Ran the KUnit tests of this PR at 645b4e55c under QEMU 11.1.2 (kunit.py run --arch arm64, -M virt -cpu max, TCG; GCC 16.1.1, rustc 1.99.0).

Result: Ran 100 tests: passed: 100. All eight new cases pass: asahi_g17_render_replay 5/5, asahi_g17_idle_hold 2/2, and a_guilty_failure_makes_a_spared_quarantine_guilty in asahi_g17_fault_admission (3/3). No oops or WARN; only the existing fetch_update deprecations.

kunitconfig

CONFIG_KUNIT=y
CONFIG_RUST=y
CONFIG_ARCH_APPLE=y
CONFIG_ARM64_16K_PAGES=y
CONFIG_PM=y
CONFIG_OF=y
CONFIG_DRM=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_THERMAL=y
CONFIG_THERMAL_OF=y
CONFIG_APPLE_PMP=y
CONFIG_DRM_ASAHI=y
CONFIG_DRM_ASAHI_G16=y

[RyanTheTide]

Thanks. The goal is right, and most of the machinery is careful. Every retire and replay waits for the existing abandonment witness, parked packets are skipped by timed_out() and cancel_render(), the spared bit is cleared everywhere quarantined is, and the sysfs knob is bounded and does nothing at its default. We're holding the series anyway. It changes timeout and recovery behaviour with default settings on the A18 Pro (T8140, 26A428), which is our daily machine, and two of the paths below can make a GPU hang there worse rather than better. References are to integration/aurora 1aa2857 with this series applied.

Major

1. A blamed packet can still be replayed. In runtime/render/settle.rs, settle_blamed_render() (about line 131) leaves the firmware-blamed queue's oldest packet spared when an earlier recovery spared it. fail_render_packet() then calls replay_abandoned(grace = false), and in queue/render.rs (about line 1255) that packet counts as innocent because !completion.guilty(). A group aborted by a recovery carries the same witness as an innocent group that was dropped, so the work the firmware blamed gets replayed, up to MAX_REPLAYS, and each replay can set off another recovery. That contradicts "work a recovery blamed is never replayed". Keeping the non-sticky error class is fine, but please add a per-packet "no replay" bit, set it in settle_blamed_render(), and check it in innocent.
2. A newer culprit can be spared (queue/render.rs about line 1003, runtime/render.rs 967-1000, runtime/compute_control.rs 34). classify() assumes the culprit is the oldest unfinished work. If an innocent frame F was published before the culprit C and stalls behind it, F times out first and fails as its own fault, so the compositor dies just as before. F's quarantined pair then keeps its timeout head, C defers MAX_RENEWALS times (about 16 s), and C ends up classified as Spared: non-sticky, and not torn down after 2 s. Before this series, renders had no grace, so C failed after about 2 s. Every compositor has a render head, so on the Neo this covers most render and compute hangs. A quarantined pair that never drains does the same to every later timeout. Suggestion: a timeout whose only older blockers are pairs or queues that already failed by their own timeout gets Own once its grace runs out, never Spare. Please also log the blocker's VM and publication in the watchdog lines.

Minor

• queue/render.rs about line 1488: the timeout and settlement paths replay after a single witness sample (grace = false). A sample can straddle a firmware update. The cost is re-running a group that already executed (double blending), not a use-after-free. Taking a second snapshot after an acquire fence would close it.
• queue/render.rs about line 1519: the fallback loop from commit 2 uses abandoned(...)? for successors. A torn snapshot's Err then goes up through timeout_render to stop_after_error()/lose_device_quietly(), so every client loses the device. Commit 6 maps the same error to "no proof"; please do the same here.
• If the front is exhausted, the replay fails the front alone and leaves its successors active, so they replay later and consume the failed front's output. The commit says the successors fail with it. Please change either the code or the message.
• Idle hold: the Assert sends KeepPowered once, unwaited, from the event pass, while acquire() re-sends it after a wake. On 26A428 it may not stick, and nothing releases the hold at system suspend. It only matters when the knob is set. The PR says 0x0a is "already used around registration", which holds for the T8140 (mmio_pair_registration); with #408's 25G83 variant registration doesn't send it, so there the hold would be its first runtime use.
• render_timeout_decision runs no final retirement scan before classifying (compute calls scan_compute first), so a finished but unprocessed head can defer another client.
• Nits: timeout_head has two doc first lines, and the "Render watchdog" warning lacks the publication= field the compute one has. The commit messages say the unexecuted retirement after a recovery was only seen on 25G83, but b9fd334 in our tree documents it on the Neo too.

Testing we'd like to see before merging. The idle hold (commit 5) is safe as it is and could go on its own. For the rest, once Major 1 and the ? fallback are fixed and Major 2 is either fixed or made an explicit choice, we'll run this on the Neo next to vkcube, glmark2 and gnome-shell:

1. A bounded stall: a piglit shader_runner fragment loop of about 6 s, then one of about 25 s, 10 times each so that both publication orders occur.
2. The ray-query deqp-vk subset that produced b9fd334.

It passes if only the culprit gets GPU work of a client failed, the innocent clients keep rendering, there's no decision=Spare for the culprit's owner, and no replaying line follows firmware blamed queue for the same slot. We'll run the same steps on the current daily kernel first as the baseline. fbalazsy's QEMU KUnit run (100/100) covers the new suites.

[RyanTheTide]

We've gone ahead and fixed the points from our review ourselves, as seven follow-up commits from us on top of your series (ryan/g17-innocent-clients-413); your commits stay as they are.

• A packet a recovery blamed now carries a bit that never clears, and the replay refuses it, whatever error class it keeps.
• A timeout whose only older blockers already failed by their own timeout defers for the grace and then gets Own, never Spare, for renders and compute alike. The watchdog lines name the timed-out work's VM and publication and the blocker's. One consequence: an innocent client stuck behind a culprit that has already failed and never drains fails as Own after the grace, as it did before this series.
• A torn sample in the timeout fallback counts as no proof instead of stopping the GPU for every client.
• When the front has hit the replay limit, its dropped successors fail with it.
• Finished renders are retired before a render timeout is attributed, like compute already does.
• The doc nits.

Tested with our universal config on the Neo (A18 Pro), together with #408 and its follow-ups (7.3.0-aurora-73u): no regressions in our smoke run (GPU, display, Wi-Fi, BT, Touch ID, audio, trackpad, glmark2 within noise), and a 30 s s2idle came back clean. For the fault behaviour we ran dEQP-VK.ray_query.stress next to vkcube and gnome-shell, five rounds. In three of them the culprit's geometry shader case timed out: only that client got "GPU work of a client failed (ETIMEDOUT)", vkcube and gnome-shell kept rendering, and there was no Spare decision, no replay and no WARN. The baseline kernel (#394 on 1aa2857) showed the same with one timeout in two rounds. KUnit: 82/82 asahi cases on the combined tree, and 104/104 with fbalazsy's config on the series alone. We didn't run a deliberate fragment-shader stall, so the 6 s and 25 s cases from our review are still open as a follow-up.

Merging this, then our follow-ups.