Repository navigation
AVD: safety and recovery (park DMA on retire, rest instead of retire, strict DART flushing for media/ML engines) - #758
Merged
Conversation
When the recovery budget runs out, or a reset fails, the decoder is retired: its jobs fail without touching the hardware until the module is reloaded. The frame that triggered it is still finished with an error, so its buffers go back to userspace while the engine that just timed out may still be running, which is the DMA-after-free pattern behind the J616s hangs. Retire through the same first steps as a recovery: disable the interrupt, stop the CM3 and attach the empty IOMMU domain, then leave it attached. A still-running engine can then only fault at the DART. A reset that fails after DMA was blocked keeps it blocked as well. Detach the empty domain at remove so a reload starts from the DMA domain. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
… storm A third recovery within a minute retired the decoder until the module was reloaded. That is too harsh for a desktop: on an M2 the H.264 conformance suite retired it within three seconds, because a few vectors use features the hardware rejects with a VP error, and every later stream failed until a reload. A user with a bad file would lose hardware decode until a reboot. Recoveries are now quiesced (DMA blocked at the DART before the reset) and have held up under forced errors, so the budget only needs to bound their rate. When it runs out, park the decoder (interrupt off, CM3 stopped, DMA blocked) and fail new frames without touching it until the budget window ends. The next frame after that resets it with DMA still blocked, restores the DMA domain and boots the firmware. Only a reset, restore or boot that fails retires the decoder. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The video decoder, video encoder, neural engine and camera ISP are run
by their own coprocessor firmware, which can still have DMA in flight
after the host driver gives up on a job: a frame timeout, a reset, a
stream torn down mid-frame. With deferred flushing a freed buffer's
IOVAs keep translating to its pages until the flush queue drains, while
the pages are already back in the allocator, so such a late write
corrupts unrelated memory without any fault. The M4 Pro hard hangs
under AVD stress all followed decoder resets and left no panic and an
empty pstore, which fits this.
Give these clients a strict DMA domain, chosen from the client's device
tree compatible ("apple,<soc>-avd", "-ave", "-ane", "-isp" and
"apple,isp"), so a late write faults at the DART instead. They map
their buffers once per stream or model, so the per-unmap invalidate
stays off the hot path. Per-stream T8140/T8142 DARTs were already
strict through apple_dart_capable().
Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…ueue The first frame watchdog of every boot logged "workqueue: work func avd_watchdog_func enqueued on deprecated workqueue" because it was queued with schedule_delayed_work() and mod_delayed_work(system_wq). The watchdog has no CPU affinity to keep, so queue it on system_dfl_wq. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
RyanTheTide
added a commit
that referenced
this pull request
Nov 1, 2026
Brings in #758 follow-ups and #773 (PMC performance floor with the AVD perf-domain consumer). The one conflict is probe: keep the rewrite's avd_pm_enable() and firmware probe boot, and settle the attached domain links to suspended before it, as #773 does for the stock driver. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #725, for decoder hang safety. One series, four commits:
CONFIG_IOMMU_DEFAULT_DMA_LAZY=y, so a freed buffer's IOVAs keep translating for up to 10 ms after its pages return to the allocator. A late coprocessor write in that window corrupts unrelated memory and never faults. Clients whose DT compatible isapple,<soc>-avd,-ave,-ane,-isporapple,ispnow get a strict DMA domain, chosen inapple_dart_def_domain_type(). DT-driven, no knob.Why
The J616s hard hangs on 2026-11-01 (analysis:
reports/engines-20261031/avd/HANG-ANALYSIS.md, 11-01 addendum) all followed decoder resets or stream teardown with DMA in flight, and none left a panic or a pstore record. On the M2 with #725, the DMA-after-error was observed directly. After a VP9H8 errorandrecover: stop firmware, the DART logged ten translation faults on the empty domain within 2 ms. The engine keeps writing after it reports an error. Commits 1 and 3 close the two remaining ways such writes could reach memory: the retire path, and the lazy-flush window after any free.Tests
apple_dart_test_strict_engines, which covers every engine compatible in our DTs plus look-alikes that must not match.apple-dart.obuilds W=1 clean with the fleet config.tools/asahi/run-host-tests.pyon this branch: 158/158.-hwaccel v4l2requestframemd5 identical to software for H.264, HEVC and VP9 at 1080p and 4K./sys/kernel/iommu_groups/N/typewith the driver unbound, and compared on the same boot.The J616s (T6041) is where commit 3 changes AVE and ISP from lazy to strict. Its AVE lazy-vs-strict receipt comes with the AVE performance PR's J616s run.
🤖 Generated with Claude Code