Repository navigation
media: apple: avd: retain zero-copy H.264 pictures across frame operations - #723
Closed
RyanTheTide wants to merge 34 commits into
Closed
RyanTheTide wants to merge 34 commits into
RyanTheTide wants to merge 34 commits into
Conversation
Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Keep the legacy open firmware serialized while moving program storage and completion ownership into tagged frame slots. Leave Borage activation dependent on the register and completion facts absent from the clean specification. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Add deterministic host tests against production helpers and KUnit coverage of slot reuse, ordered retirement, watchdog wrap, inbox retry, command builders and completion parsing. Track the unproved Borage fields explicitly and preserve session generations over firmware reboot. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Remove contexts from the scheduling list before releasing their m2m state, reject already-scheduled jobs during teardown, and keep Apple fp pointers absent on the open backend. Preserve tagged slots over inbox-full retries and avoid periodic watchdog wakeups when only submitted frames remain. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Keep the CODECS trailer intersection with the SoC capability table when selecting the open backend. Bound firmware copies by the mapped code resource. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Only the H.264 compatibility builder explicitly completes held slice requests during preparation. Continue submitting legacy HEVC requests that retain their capture buffer. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…ation Prevent queued contexts from observing the same free capacity. Run preparation on a context work item because m2m try_schedule can invoke device_run synchronously while the queue mutex is held. Drain that work before freeing the context. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Keep power through slot retirement and context drains, restore firmware before retrying reserved frames, and defer DMA releases while the DART is inactive. Re-enable a retained Apple MCPU image before mailbox link setup, without uploading it again. Pause and drain the queue for system sleep and final teardown. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Request the mailbox IRQ without automatic enable, open it only after successful firmware boot, and synchronize and close it after backend suspend. Close the host IRQ on teardown even if the final PM transition fails. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Map only DT-declared PIODMA and SRAM resources and replay the shifted SRAM base before every open CM3 boot. Keep reset, empty-domain attach/detach and firmware reboot ordered; refuse to reboot if the DMA domain cannot be restored. Keep host IRQs closed until old slots are invalidated. Add synthetic-RAM boot replay and recovery-gating KUnit coverage. Queued PIODMA use still requires the DT to describe the DMA stream as well as decode. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Replay PIODMA setup in the common boot dispatcher so queued open transports inherit it even with their own mailbox boot implementation. Keep firmware generation handling and initial-only backend fallback in the same dispatcher, and test the real open boot path through it. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Brings in the frame-queue core (avd/rewrite-core) and its runtime and system PM, PIODMA replay and DART recovery (avd/power-idle). Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Each frame in VP to PP flight owns one hardware instruction FIFO, named by EXEC bits 7:4. The hardware has 15 of them on v3 and later tier 1 parts and 7 on T8103 and T8140, so the 16-slot queue must not keep more frames in flight than the variant has FIFOs. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
T6040 decodes HEVC with the v5 header. Add HEVC to its capabilities but keep it hidden unless the loaded open firmware lists h265 in its CODECS trailer, so older firmware without the trailer behaves as before. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
When the VP instruction queue is at least half full the open transport slept a fixed 500-650 us (100-150 us on parts with v5 pacing) before the next segment. Poll the queue level every 10 us instead and continue as soon as it drops below half, keeping the old sleep as the upper bound. The submission order and queue thresholds do not change. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The open transport writes every instruction word to the VP FIFO register from the CPU, sleeps while the FIFO is half full, and kicks the PP from the VP-done interrupt thread, one frame at a time. avd-fw queue ABI 2 moves all of that into the CM3. The host streams each frame program into a word ring in CM3 SRAM, posts a 64-byte descriptor and rings mailbox 0; the firmware feeds the VP FIFO, writes the PP kick (plus one continuation kick per extra segment for HEVC, VP9 and AV1) when the VP is done, and posts a tagged completion on mailbox 1. No PIODMA is used, so the second DART stream that the current device trees lack is not needed. The transport is chosen after every firmware boot from the image info block: ABI 2 with the SRAM ring, and an SRAM aperture matching the node. Anything else keeps the legacy transport. Each frame in flight gets its own instruction FIFO (the slot index), and the queue keeps at most the FIFO count in flight. Frames of one context stay serialized by the queue core, so different streams overlap; within a stream the gain is the removed CPU word writes, pacing sleeps and interrupt round trip. T6040 now loads avd-fw-v5-t2, whose SRAM aperture matches the node, and falls back to v5-t1 when that image is not installed. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
CM3 SRAM keeps its contents across a firmware reload. After an ABI 2 image ran, a later boot of an older image would still show the queue info magic and the driver would pick the queued transport for firmware that does not speak it. Clear the magic before every open-firmware boot so only an image that publishes it again selects queue mode. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
With the firmware chaining VP to PP, the VP of picture N+1 can run while the PP of picture N is still busy, but the legacy H.264 builder kept one set of scratch buffers, one VP to PP instruction FIFO and one slice table per context, and the queue core held the m2m scheduler until a picture retired. Give H.264 two banks of everything the hardware touches while decoding a picture (instruction FIFO, pipe state, above, loop-filter and MV row buffers) and of the slice copies. A new picture takes a free bank and records it in its CAPTURE buffer; retirement frees that bank. Pictures whose last slice never arrived restart in their own bank. The legacy transport keeps one bank, so nothing changes there. The queue core now lets a legacy codec context hold as many pictures as it has banks, carries each frame's FIFO address to the transport, and only keeps the m2m scheduler for slices that hold their CAPTURE buffer, so the next picture is prepared while the previous one decodes. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
… the queue image HEVC gets the same two-bank treatment as H.264 on the queued transport: the VP to PP instruction FIFO, pipe state and all row/column scratch exist once per bank, a picture records its bank in its CAPTURE buffer, and a new done hook frees it at retirement. The VP of picture N+1 then overlaps the PP of picture N. The legacy transport keeps one bank. The per-buffer bank field is now shared by both codecs. Codecs a variant holds back until a newer firmware (fw_caps_required, T6040 HEVC) are now also enabled when the loaded image runs the queue ABI and lists the codec in its info-block codec mask, not only through a CODECS trailer. Falling back to the legacy transport restores the trailer-derived set. Both the transport choice and the codec list come from what the image reports at boot, and clients enumerate formats and size their contexts before a first decode would power the decoder up, so boot the firmware once at probe and let it autosuspend. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
avd_frame gained a DMA address for its instruction FIFO, and the host build of the queue test has no dma_addr_t. Define it in the host shim. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…eue_mutex The threaded interrupt handler takes queue_mutex before it touches the decoder. Queue reset runs from the queue worker with queue_mutex held, and its recovery called disable_irq(), which waits for that handler thread. When an interrupt arrived around a frame timeout, the handler blocked on queue_mutex and the reset blocked on the handler: the worker and the IRQ thread deadlocked with the mutex held. Runtime suspend had a similar cycle with a submission waiting for it under queue_mutex. Close hardware access with a spinlock instead: the handler runs the backend's interrupt code only under irq_lock, with hw_on set and the interrupt generation it sampled before taking queue_mutex unchanged. Turning the IRQ off clears hw_on and bumps the generation under that lock, so an access in progress finishes first and a handler woken by an older firmware instance leaves the hardware alone, and then uses disable_irq_nosync(). Device removal, which holds no queue_mutex, still synchronizes with the handler. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The old driver kicked the PP for every VP DONE message of the open firmware. The legacy shim in the frame queue dropped a VP DONE that came in while no frame was in flight. That happens when a surplus PP DONE of a multi-kick picture completed the next frame early: the next frame's VP DONE then found no active frame, its PP was never started, and it timed out. Kick PP for the last submitting context in that case, as before. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Reading MBOX1_RETRIEVE when the mailbox is empty returns the previous message again. On a spurious or early interrupt that is a repeated PP DONE, which completes the next frame before its PP has run. The next PP then never reports done, the frame times out and the decoder is reset. The legacy shim and the queued transport both popped unconditionally. Check the empty flag in MBOX1_STATUS (bit 17) first, give a message that is not readable yet up to 10 us, and otherwise acknowledge and ignore the interrupt. Both transports share the helper. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
A frame timeout on the J616s was followed by the machine going down with nothing in the journal, so nothing showed how far the recovery got. Log every recovery step (stop firmware, reset, DART flush, firmware boot) before it runs, with the DART, runtime PM and IRQ state at the start, refuse and warn once if recovery is ever entered with the decoder powered off, and name the codec, slot state, frames in flight and transport in the timeout message. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
… a loop The Neo hung after a burst of "H0 error" (VP0, the HEVC VP, raising its error interrupt) with "Frame processing timed out!" for every frame: each error reset the decoder (pmgr reset, DART empty-domain flush, firmware reload), the next frame of the same broken stream failed at once, and the resets ran back to back, up to one per millisecond, until the machine stopped. The frame queue inherited the same loop. Bound it at three levels: - A stream that fails three frames in a row in hardware is stopped: its frames complete with an error, without touching the decoder, until streaming is restarted. - Decoder recoveries are at least 20 ms apart. - More than eight recoveries within 10 s put the decoder in a 10 s cool-down in which new frames fail at once. A faulted decoder (failed recovery) no longer blocks the m2m scheduler, which left clients waiting forever: its frames fail at submission, the request controls are completed so the request fd signals, and the next clean power-up clears the fault. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The HEVC builder computes each slice's coded data address and size from GStreamer's layout: slices tile the OUTPUT buffer as whole NALs, bit_size covers the whole NAL and data_byte_offset is the slice data's offset in it. FFmpeg's V4L2 request backend gives data_byte_offset from the start of the buffer and a bit_size of the slice data only. Every slice then got a wrong address and a size short by its header, the VP raised its error interrupt on every frame, and the decoder was reset once per frame. This is the "H0 error" storm the Neo hit in an FFmpeg-based benchmark. When the slices describe a complete, unambiguous absolute layout, rewrite the run's private copy into the equivalent NAL-relative description. The GStreamer layout and anything ambiguous are left untouched. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
avd-inst.h builds every instruction word with FIELD_PREP() but relied on an indirect include. The arm64 build gets it through other headers; the UML KUnit build of the avd tests does not and fails. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The recovery trail queried the runtime PM state before the existing inactive-DART check, which the PM KUnit test exercises with no device. Check the DART first and log that case on its own. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…e context
job_ready() admitted at most one context at a time (queued_ctx) and only
while a slot was free. Contexts that queued work while another was
admitted were rejected, and were retried only from the queue worker,
which walks the context list in order: the first context won every time
and later ones could wait for seconds. With four streams the starved
stream's OUTPUT buffer was not returned within FFmpeg's two-second poll
("Failed waiting on OUTPUT buffer") and the run failed, with no decoder
timeout or error.
Drop the reservation and the capacity condition from job_ready() so every
context with work enters the m2m job queue, which runs jobs one at a time
in arrival order. A job that finds no free slot or no transport capacity
is kept as m2m's running job and restarted by the queue worker once a
frame retires; draining its context finishes it.
Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Collect H.264 slice controls and OUTPUT DMA descriptors until the last request of a picture. Retain those OUTPUT buffers through picture retirement, including the legacy transport, and release partial pictures on errors and streamoff. Give each live picture its own scratch and instruction FIFO. Build native queued-open programs with the existing instruction encoding, while keeping old open firmware on the legacy callbacks. Extend the public V3 parameter projection to independent slice records and keep unresolved Borage fields explicitly unsupported instead of inventing their encodings. Exercise the production prepare/retire paths and source ownership helpers alongside the frozen Linux builder and parameter-field host tests. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Use the builder's effective shifted address in the V3 parameter projection, including the motion-vector tail, while keeping open instruction words identical to the frozen builder. Make the host shim consume reference-buffer arguments and use the actual V4L2 format type so the lifecycle tests compile with warnings treated as errors. Clarify the public V3 layout's provenance. The 96 builder cases, V3 bounds, actual prepare/retire lifecycle, source ownership and queue scheduler host tests pass. The complete AVD directory builds as an arm64 external module with W=1 KCFLAGS=-Werror. Assisted-by: LLM Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Contributor
Author
|
Closing per the one-PR-per-series rule. The branch stays and gets integrated into the AVD rewrite, which lands as one PR once it works on hardware. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
H.264 pictures accumulate slice controls and OUTPUT DMA descriptors until the final request, then build one complete instruction program. The core retains intermediate OUTPUT buffers until picture completion and releases incomplete pictures on error or streamoff. Each live picture owns separate scratch and an instruction FIFO; reference motion vectors stay with CAPTURE.
Based on the integration branch in #713. Native H.264 programs use the queued open backend, while older open firmware keeps the legacy callbacks. The shared queue header is identical to the integration copy.
The Apple portion remains incomplete: the public Viola/V3 projection has bounded per-slice records, but it does not establish T6000's later LilyD ABI. Borage still lacks proved parser-field mappings, packed-address rules at command +0x3c/+0x44, and complete work-count bounds. Those paths return
-EOPNOTSUPP; this draft does not enable Apple decoding. T8112/T8140 retain explicit ledger TODO entries.Validation on the build host:
W=1 KCFLAGS=-Werror, including the existing KUnit objects.No target decoding, performance result, or encoded-stream comparison with eiln's decoder is claimed. The build and host checks do not establish the unresolved Apple ABI.
This integration draft uses a new branch so the previously published #702 history can remain intact without a force push or merge.