[RyanTheTide]

Small safety series for the shipping AVD driver, independent of the frame-queue rewrite (#713). Each commit stands alone and is reviewable on its own.

Why

During last night's decode benchmarks the J616s hung hard three times and the Neo once. pstore was empty every time, and on the third J616s hang the hardware watchdog did not recover the machine. Every hang came after decoder timeouts and resets; runs without a reset did not hang.

The stock driver was involved directly:

• 03:09: the J616s hung under the stock module during 4-stream H.264 runs, after five "Frame processing timed out!" and reset cycles in five minutes.
• 01:49: the Neo hung under the stock module after 40 "H0 error" / "Frame processing timed out!" pairs. At the end it was resetting once per millisecond.

Logged evidence:

• 4-stream stock runs failed bit-exactness at frames 83 and 249 and logged "ignoring stale completion". Frames were finishing before the hardware had.
• 02:54:46, J616s, stock: a burst of DART "write PTE invalid" faults at just-released IOVAs, two seconds before a timeout. The decoder was still writing a buffer the driver had already returned. Had the IOVA been reused first, the write would have corrupted other memory silently.
• Neo: every "H0 error" was the HEVC VP rejecting slices described in FFmpeg's layout, which the driver did not understand.
• J616s and Neo: UBSAN index -1 into reference_frame_ts on the first AV1 decode.

What it changes

1. AV1: bound ref_frame_idx before indexing reference_frame_ts. A -1 entry now takes the existing missing-reference path.
2. Do not pop an empty mailbox: check MBOX1_STATUS bit 17 first, and give a late message 10 us.
3. Finish a job only on the PP DONE of its own PP: a surplus PP DONE of a multi-segment picture no longer finishes the next job early.
4. Quiesce before reset, and stop resetting:
   • before the pmgr reset: disable the interrupt, stop the CM3, attach the empty IOMMU domain so in-flight DMA faults at the DART, and wait 5 ms; afterwards, restore and check the domain, then boot;
   • at most two recoveries per minute, at least 20 ms apart;
   • a failed or third recovery retires the decoder until the module is reloaded.
5. Stop a stream after a frame fails in hardware: its jobs fail without touching the hardware until streaming restarts, and the request controls are completed so the request fds signal.
6. Accept FFmpeg's HEVC slice layout: a complete, unambiguous absolute layout is rewritten to the NAL-relative one the builder expects. GStreamer input is untouched.

Tests

• Built commit by commit as an external module against the 7.3.0-aurora-fleet12e build tree (Fedora asahi base + asahi.config): ARCH=arm64 LLVM=1 W=1 KCFLAGS=-Werror, clean at every commit.
• checkpatch is clean on every commit.
• Rebuilt for 7.3.0-aurora-tt3 (aurora-wip 9ed109d, the fleet13 kernel on the M1 Pro, M2 and Neo) the same way, with no warnings. The AVD directory at this head equals tt3 plus these six commits.
• Hardware, M2 (T8112), tt3, 2026-11-01, as a module swap on one boot. Logs and raw results are in reports/engines-20261101/avd/m2-725/.

Results on the M2

The bench is avd-bench (FFmpeg V4L2 request, demux in memory, every displayed frame MD5-checked against software decode before and after the timed passes). Each row is the best of 3 passes in aggregate fps. Stock is the tt3 in-tree module on the same boot.

Clip	stock N=1	#725 N=1 (Q1/Q4/Q16)	#725 N=2	#725 N=4 (Q1/Q4/Q16)
H.264 1080p	480/504/503	480/503/504	503	503/503/500
H.264 4K	185/190/190	185/190/190	190	190/190/189
VP9 1080p	348/369/369	348/369/369	368	369/369/369
VP9 4K	120/123/123	120/123/123	123	123/123/123
HEVC 1080p	not run (stock resets every frame on FFmpeg input)	776/847/847	847	845/847/847
HEVC 4K	not run	321/336/336	337	337/337/337

• Every row is bit-exact, with no kernel line in any row: no timeout, stale completion, DART fault or UBSAN. HEVC through FFmpeg decodes for the first time, and so does N=4. On the J616s, stock failed H.264 N=4 rows about 40 % of the time.
• Stock was not run at N=4 or on FFmpeg HEVC, because both took machines down last night.
• ffmpeg 8.1 with the V4L2 request hwaccel (-hwaccel v4l2request) gives framemd5 identical to software decode for H.264, HEVC and VP9 at 1080p and 4K. A 4K HEVC clip decodes 120 frames in 0.39 s at 32 % of one CPU.
• Mixed load: three clean streams decoded while a fourth stream was corrupted. All three stayed bit-exact (H.264 3x169 fps, HEVC 3x290, VP9 3x124).
• Teardown: 40 decodes were SIGKILLed at random points with frames in flight. No kernel line was logged, and a clean decode afterwards was bit-exact at 503 fps.
• Forced hardware errors: a bit-flipped H.264 stream sent through ffmpeg makes the VP raise its error. Each one gave exactly H4 error, recover: stop firmware, block DMA, reset, restore DMA, boot firmware, done, then stream stopped, and the recovery took 5 ms. The third error within the minute logged decoder retired after repeated resets. A clean decode then failed at once without touching the hardware, and after a module reload it decoded bit-exact at 503 fps. No DART faults appeared in any of this.
• The M1 Pro (T6000) run follows. This leaves draft now that the M2 run has passed, so it lands in the next fleet kernel.

rmmod apple_avd && insmod apple-avd.ko
# avd-bench bit-exact sweep (1 to 4 streams), GStreamer rows, then:
dmesg | grep -iE 'recover:|retired|stream stopped|timed out|stale|translation fault|UBSAN'

The root-cause write-up (ranked hypotheses for the hangs) lives with the AVD lane's reports; the same fixes are in the rewrite branch (#713).

🤖 Generated with Claude Code

[RyanTheTide]

Post-merge evidence from the same M2 run (tt3). A bit-flipped VP9 stream decoded through ffmpeg -hwaccel v4l2request raised H8 error. After recover: stop firmware and recover: block DMA, the DART logged ten translation faults on the empty domain within 2 ms (stream 0, IOVAs 0xff5ebc100/0xff5ebc180). Only then did reset, restore DMA, boot firmware and done follow. So the engine keeps doing DMA after it reports an error and after the CM3 is stopped. Before this series those writes went through the live domain into buffers the error path returns to userspace. The 5 ms drain covered it: the last fault came 2.6 ms before the reset.