Skip to content

media: apple: ave: V4L2 stateful encoder for the J700 AVE, and enable it in the J700 DT - #615

Merged
RyanTheTide merged 20 commits into
aurora-wipfrom
ave/v4l2-encoder
Oct 30, 2026
Merged

RyanTheTide merged 20 commits into
aurora-wipfrom
ave/v4l2-encoder

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Draft. A V4L2 stateful encoder driver for the AVE video encoder in the MacBook Neo (J700, T8140). The PR also enables the encoder in the J700 DT and builds it as a module in asahi.config.

H.264 (Baseline, Main, High) and HEVC (Main) from NV12, at sizes from 320x240 up to 2560x1600, including the 2408x1506 panel. It works with GStreamer's v4l2h264enc/v4l2h265enc and ffmpeg's h264_v4l2m2m/hevc_v4l2m2m. No firmware files are needed.

Builds on Acelogic's mailbox and ring transport from 4612e5f, which stays as the IPC layer. Thanks, Ace.

Field layouts and constants are hardware facts from our research traces; no copied code.

How it works

  • Firmware. iBoot preloads the encoder firmware and the Aurora Stage 2 loader reserves it. The driver checks the TEXT and DATA checksums Stage 2 records under /chosen. It keeps a copy of DATA and restores it before every firmware boot.
  • Firmware build gate. The driver builds every command itself, so it only drives the firmware build those layouts were checked on: AppleAVE2FW 9003.78.0, TEXT checksum 69c54454. On any other build the device binds, but starting a stream fails with ENOTSUPP and a message naming the checksum.
  • Generated commands. Start and Process are built in a zeroed buffer from the header, the constant words and the computed fields. The constant words are named and commented tables in ave-image.c. The computed fields cover sizes, cropping, profile, level, rate control, GOP and B-frame structure, frame rate, buffer addresses and per-frame values. macOS also builds these commands in code rather than storing them. With identical settings and addresses masked, all four generated commands match macOS byte for byte.
  • Sessions. Each stream opens and starts a session. Up to five frames are inside the firmware at once (three without B-frames). The firmware reorders B-frames itself, so CAPTURE buffers come back in bitstream order with the timestamp of the frame they code. Two streams can run at once on one firmware boot.
  • Power management. The firmware boots for the first stream. It is drained, stopped and halted after the last one, as macOS does, and the next stream starts it from reset. Runtime PM powers the encoder down two seconds later.
  • System sleep. Open streams are drained and closed and the firmware is halted before the power goes. On resume each stream reopens on its next frame, which starts with an IDR and fresh parameter sets, so clients see no error.
  • CBR filler. CBR streams are padded with filler-data NAL units once more than 0.5 s of bit rate has gone unused, which is the rule seen in macOS streams. If the firmware output carries no filler, the driver appends it.
  • Stride fix-up. The encoder reads NV12 with the 64-byte aligned stride that S_FMT reports. ffmpeg fills buffers with its own line size, which differs when the width is not a multiple of 64. The driver detects that from the payload size and spreads the lines out in place for driver-allocated buffers, or copies them for imported ones.
  • Buffer sizing. Working buffers are sized from the picture size and allocated largest first in the encoder's 256 MiB DMA window. Two groups were measured on hardware to need more room than their size fields say: the reconstruction buffers (1.5x, they hold chroma after luma) and the +0x3a0 group (1.25x). The OUTPUT and CAPTURE queues get a fixed share of the window.
  • Bounded waits. Every wait for the firmware has a limit: 5 s per command or frame, 10 s for Start. A firmware that stops answering fails the stream with EIO and keeps its power and buffers until reboot. Every error path logs its reason.
  • debugfs. apple-ave/last_start_{h264,hevc} and last_process0_{h264,hevc} hold the last generated commands, for comparison with reference captures.

Testing

Hardware results for ec7c3a7 are in #615 (comment). In summary, run on the Neo with no session-image files installed:

  • the generated commands match macOS byte for byte, Start and Process, H.264 and HEVC;
  • 1920x1080, 1280x720, 1366x768, 2408x1506, 1000x562 and 640x360 with ffmpeg and GStreamer, both codecs: every run 300/300 frames and decode-clean, including the stride fix-up at 2408, 1366 and 1000 wide;
  • CBR padding, VBR, GOP 15 and 1, two concurrent streams;
  • system sleep mid-encode through rtc1: 1500/1500 frames for H.264 and HEVC;
  • idle power-off of every venc0 domain, v4l2-compliance 48/48, no DART faults;
  • a 10-minute live run, 18000/18000 frames, under 50 C;
  • quality on par with x264/x265 medium.

The head commit after that run, f494c84, only removes bring-up diagnostics.

Limits and open items

  • Not offered: 4:2:2, 4:4:4 and 10-bit (Main10, High 10, P010 input); slices; VUI and colour signalling; constant-quality rate control; sizes above 2560x1600.
  • The firmware build gate above. A new firmware build needs its command layout checked before it can encode.
  • One encoder unit. The SoC has a second (VENC1), which macOS did not use for two streams either.
  • The interrupt line that signals the rings is not known, so the driver polls with sleeping waits.

Build

External module against the 7.3.0-aurora-fleet10nv tree and the ave objects in the full tree, both with W=1: no warnings. checkpatch --strict is clean on the driver and its documentation.

🤖 Generated with Claude Code

RyanTheTide and others added 12 commits October 30, 2026 16:17
The AVE driver was a root-only debugfs transport that replays captured
firmware commands at a fixed 320x240, and it could not be built into a
kernel with suspend or kexec. Rework it into a normal V4L2 stateful
encoder driver on top of Acelogic's mailbox and ring transport from
4612e5f ("media: apple: add an opt-in J700 AVE encoding
transport"), which stays as the IPC layer with its start-up handshake.

- Probe binds by DT only: no enable or snapshot parameters, no SoC
  checks beyond the compatible. The five encoder power domains are
  attached as a list and follow the device's runtime PM.
- The firmware is booted on first use instead of through debugfs. The
  driver keeps a copy of the firmware DATA segment taken while it still
  matches the loader's checksum and puts it back before every boot, so
  the coprocessor can lose power and start again from the same state.
- Commands are built in the driver. Config, Open, Complete, Stop, Close
  and Halt are complete. Session start and per-frame Process are left
  returning -EOPNOTSUPP until their layouts are described.
- Firmware replies are checked for their status word.
- A V4L2 M2M encoder (NV12 in, H.264 or HEVC out) per
  dev-encoder.rst, with drain, EOS and bitrate, GOP, profile, level and
  QP controls.
- System sleep marks the firmware lost when the coprocessor lost power;
  the next session boots it again. Runtime suspend keeps power while the
  firmware runs until its stop sequence is in.

Rename the symbol to VIDEO_APPLE_AVE and drop the !SUSPEND,
!HIBERNATION and !KEXEC_CORE dependencies.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aef49ec ("arm64: dts: apple: t8140-j700: leave the video encoder
disabled") turned the AVE encoder, its two DARTs and the six venc0 power
domains off because no driver bound apple,t8140-ave and the domains sat
in sync_state forever. The apple-ave driver now binds the node, so
enable them again.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The J700 video encoder (AVE) gets a V4L2 stateful encoder driver in this
series. Build it as a module in the universal config, next to the decoder
(AVD) and ISP, so the device-tree enable does not leave the encoder and its
power domains waiting on a driver that is never built.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fill in the parts of the J700 encoder driver that were left returning
-EOPNOTSUPP, so a V4L2 client gets H.264 or HEVC out of NV12 frames.

Session start builds the Start command from the encoding parameters:
picture and coded size, cropping, profile, level (raised to what size,
frame rate and bit rate need), VBR with a one-second data rate cap or CBR,
QP range, GOP length, the B-frame structure and frame rate. The driver
allocates the encoder working buffers with sizes derived from the picture
size and lists them in the command.

Frames go out as Process commands over a ring of five slots (three
without B-frames). The firmware reorders B-frames itself and answers in
bitstream order, so CAPTURE buffers come back in that order with the
timestamp of the frame they code. The first frame is sent alone and
followed by a Complete, as macOS does. Process carries the input planes,
frame number, presentation time, the current bit rate (it may change
while streaming) and the forced key frame request. HEVC output is
assembled from the parameter sets, the slice header and the slice data
the firmware leaves in three buffers. For H.264 the length is not known
yet, so the coded buffer is read as Annex B up to where the data ends.
Odd output is reported once per session with a dump under dynamic debug.

Commands no longer wait for their answer one at a time. Each command
buffer is tracked until the firmware acknowledges it, answers are matched
by command id, and frame answers by session and slot. A short poller
collects frame answers while frames are outstanding, since the firmware
interrupt is not wired up.

Drain sends four Completes, then Priority, Stop and Close, and the
firmware is halted after every session. Halt is never answered; after a
short wait the coprocessor counts as stopped, runtime suspend may drop the
power domains, and the next session restarts the firmware through the
boot path from the saved DATA copy. A firmware that stops answering keeps
its power and its buffers, and is not used again until reboot.

The constant words of Start and Process are not built into the driver.
It loads them per firmware build from
apple/ave-t8140-<text crc>-{h264,hevc}-{start,process}.bin and
overwrites every session and frame field. Without those files STREAMON
fails with ENOENT.

Field layouts are facts from our private research (macOS traces); no
copied code.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k frames

The first hardware run encoded frames, and showed three problems.

H.264 output was guessed from the coded buffer. The firmware reports the
coded length at metadata +0x120 (repeated at +0x12c) for H.264 as for
HEVC, and the coded buffer holds only the slice NAL units. Use that
length for both codecs, and take the parameter sets from the metadata
buffer: at +0xb2a4 where HEVC has them, otherwise from the first SPS
(H.264) or VPS (HEVC) start code found past the header words.

Frames that the firmware stops answering left the client waiting
forever: the poller had no time limit. When frames are outstanding that
the firmware must answer (any frame without B-frames, four with them) and
nothing comes back for 5 s, the firmware is marked stuck and the session
fails, which completes every buffer with an error.

The run also showed DART write faults right after Start. To find their
source, the driver now logs its buffer map under dynamic debug and warns
about any word in the Start or Process images that looks like an encoder
DART address but is not one of the session's buffers, which would be a
stale address left in a session image. The output dump now also covers
the metadata around the parameter sets and every nonzero metadata line.

Field layouts are facts from our private research (macOS traces) and our
own hardware runs; no copied code.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The second 1080p H.264 run (P frames only) still had DART write faults
right after the first frame was encoded, from 0x7800 to at least 0xa000
bytes past the end of the first +0x390 buffer, and the firmware stopped
answering after that frame. The size the driver puts in Start for that
group is the one macOS reports, so the firmware writes beyond the size it
is given and macOS must map more than that.

Keep the reported sizes, but map the +0x390 group at one and a half times
its size and every other working buffer with an eighth (at least 64 KiB)
more. Buffers now carry their mapped size separately, and the debug
buffer map shows both.

The stale address check no longer reports pointers into the firmware
image itself (the Start image points at the TEXT segment, which sits at
the same address as on macOS).

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…afe buffers

The third hardware run encoded H.264 cleanly twice in a row, with a power
cycle between the sessions. HEVC then hung its client for 20 minutes and
broke every later session.

ffmpeg's v4l2m2m encoder sends a frame and waits for its coded output
before sending the next. With B-frames the firmware keeps frames until
later ones arrive, so both sides waited, and the stuck-frame timeout only
counted four or more held frames. Now, when frames are held and no input
came for 50 ms, the driver sends a Complete, which makes the firmware
code what it holds (macOS sends one after the first frame of every
session). If those frames do not come back within 5 s, the session fails
as before.

The next session started before runtime suspend had powered the encoder
down, and restarting the halted coprocessor while it kept power got no
answer. A session now runtime suspends the encoder first when the
firmware is halted, and a restart is refused while the coprocessor still
runs, instead of trying it.

A failed firmware boot marked the firmware broken for good and kept the
buffers of the session that triggered it, so later sessions could not
allocate. A failed boot now leaves the firmware halted, to be retried
after a power cycle. Only buffers the firmware was given in a Start are
kept when it stops answering.

Field layouts are facts from our private research (macOS traces) and our
own hardware runs; no copied code.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v4l2-compliance failed the format tests: a stateful encoder must answer
VIDIOC_ENUM_FRAMESIZES for every format it enumerates, and the driver
only did for the coded ones. Report the same range for NV12.

CREATE_BUFS with the queue maximum failed because 32 buffers of each kind
do not fit the encoder's 256 MiB DART window next to a session. Allow 16
OUTPUT and 8 CAPTURE buffers.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit set max_num_buffers on both queues, which this
kernel's vb2_core_queue_init() refuses below VB2_MAX_FRAME, so every open
of the device failed. Leave the queue maximum alone and limit the count
in queue_setup instead: 16 OUTPUT and 8 CAPTURE buffers, so a full set
still fits the encoder's 256 MiB DART window.

Fixes: 593f94d ("media: apple: ave: report raw frame sizes and cap the buffer counts")
Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… retry

Fixes from the fourth hardware round.

Rate control: with only BITRATE set (ffmpeg's case) the one-second
data-rate limit equalled the average rate, and the firmware then coded
far below the requested rate. macOS always sets the limit to twice the
average. BITRATE_PEAK now defaults to 0, which means exactly that; a
client value is still used as given (and never below the average).

Memory: H.264 at 2408x1506 ran out of room in the encoder's 256 MiB DART
window. The DMA layer maps every buffer at a power-of-two alignment, so
the size-proportional margins pushed buffers into the next size class.
Margins are now 256 KiB on the +0x390 group and 64 KiB elsewhere, and the
OUTPUT and CAPTURE queues get a fixed share of the window (64 and 32 MiB,
counted at the mapped size, at least 6 OUTPUT and 2 CAPTURE buffers).

HEVC with B_FRAMES set to 0, as ffmpeg sets it: the firmware still uses
its B-frame structure, so each frame waited for the 50 ms no-input flush.
Send a Complete right after every frame in that case instead.

Restart: right after a session the coprocessor can still be powered
when the next one starts, because the DARTs suspend a little later.
Instead of failing with EAGAIN, cycle the power up to ten times, 100 ms
apart.

Formats: a huge OUTPUT bytesperline wrapped the size computation to 0
(v4l2-compliance). Clamp it to twice the maximum width.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The +0x390 group holds a 4:2:0 picture and its size field only covers
the luma part. macOS leaves 1.5 times that size behind each one (0x318000
behind the 0x20d000 buffer at 1080p), and with only 256 KiB of margin the
firmware wrote chroma far into unmapped space (from +0x252000 on, with
hundreds of thousands of faults suppressed) and stopped answering mid
stream. Map 1.5 times the size plus the usual 64 KiB.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Encoded streams look right but drift from the source picture over a GOP,
and the firmware is known to write past the sizes it is given. A write
that runs into a neighbouring mapped buffer raises no DART fault, so it
would go unnoticed.

Keep a kernel mapping of the working buffers and, when a session ends
normally, report for each one how far past its size the firmware wrote,
with a warning when it reached the end of the mapping. This costs a scan
of the zeroed margins once per session.

At 2408x1506, ffmpeg's v4l2m2m encoder produced a sheared picture: it
copies NV12 with its own frame line size and ignores the bytesperline
from S_FMT, which differs when the width is not a multiple of 64. The
encoder needs the 64-byte aligned stride, so the driver now keeps the
client's bytesused on OUTPUT buffers, recognises a frame laid out with a
smaller stride from it, and fails the stream with a message naming both
strides instead of encoding garbage.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Hardware results on the MacBook Neo (J700). This was a one-shot kernel 7.3.0-aurora-fleet10nv (fleet10 46b7a93f plus #626), with the module built out of tree from this branch. v8 is 44596dc and v9 is b36da98. The live DT comes from Stage 2 aura 1.0.0b1 (8fec5a8f), which has the encoder node enabled.

The session images for Start and Process are loaded from apple/ave-t8140-69c54454-{h264,hevc}-{start,process}.bin. For this test they were private files made from our research captures and installed only on the test machine. They are not part of this PR, and whether they can ship is still open.

What works (v8/v9)

Test Result
H.264 1080p30, ffmpeg h264_v4l2m2m -b:v 8M -g 30 -bf 0, 300 frames 300/300, Main L4.0, decodes clean, about 78-95 fps, back-to-back sessions OK
H.264 720p, ffmpeg 300/300, about 150 fps
H.264 2408x1506 (panel native), GStreamer v4l2h264enc 300/300, decodes clean, picture correct, about 67 fps
HEVC 1080p, ffmpeg hevc_v4l2m2m 300/300, 0 decode errors (checked with ffmpeg on another machine), about 76 fps
HEVC 2408x1506, GStreamer v4l2h265enc 300/300, 0 decode errors
GStreamer v4l2h264enc / v4l2h265enc 1080p 300/300 each, about 110 fps
VBR on hard content (noise, 8 Mb/s) 15.8 MB / 300 frames, within the 2x peak
Idle power-off 4 s after the last session, runtime PM is suspended and venc0_sys/dma/pipe4/pipe5/me0/me1 are all off; the next session cold-starts the firmware
v4l2-compliance 48/48 (v8)
DART faults none (v8/v9)

Open

  • ffmpeg at widths that are not a multiple of 64 (such as 2408): ffmpeg's v4l2m2m copies NV12 with its own linesize and ignores bytesperline, so v9 refuses those frames with a clear error. GStreamer is fine. A bounce copy would fix ffmpeg at the cost of DART window space.
  • CBR on easy content does not pad up to the target (164 KB for 300 frames of pattern=ball at 8 Mb/s). Still being checked.
  • The reference buffers are written past their reported size: ref_a by 0x106604 at 1080p and 0x1c7e06 at native, ref_b by 0x6a6a and 0xc77d. v9 measures this and v10 will size from the measurements.
  • System suspend with an open encoder is not tested. The J700 has no RTC wake source (rtcwake: /dev/rtc0 not enabled for wakeup events), so s2idle can't be woken unattended.
  • The fanless Neo reached 97 C during a back-to-back batch, so long runs need pacing.

Field layouts are facts from our private research (macOS traces); no copied code.

Logs: dmesg per run and outputs are kept with the night report (neo-night-20261030/ave/).

🤖 Generated with Claude Code

RyanTheTide and others added 2 commits October 30, 2026 21:26
The overrun report from the previous commit gave the same numbers in
every session. At 1080p the firmware wrote 0x313604 bytes into the
0x20d000 +0x390 buffer and 0x26a6a into the 0x20000 +0x3a0 one; at
2408x1506 0x557e06 into 0x390000 and 0x4c77d into 0x40000. No other
buffer was written past its size.

So the +0x390 group is a 4:2:0 picture whose size field is the luma part:
map 1.5 times it. Map the +0x3a0 group at 1.25 times its size (it is
written up to 1.2 times, and macOS spaces it that way). Drop the margins
on every other buffer, which saves DART window space.

The session debug line now also shows the rate-control mode, bit rate,
data-rate limit, GOP length and frame rate the firmware gets.

Measurements are from our own hardware runs; no copied code.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A GStreamer run stopped at frame 240 with only "encoding failed: -5". A
job refused to touch a session whose firmware it no longer owned, but a
drain closes the session and halts the firmware first and only then hands
out the frames still waiting for CAPTURE buffers. When those buffers came
in later, the next job saw the closed session and failed the stream
without a word. Only a session that is still open must own the firmware.

Every other path that ends a stream with -EIO now logs its reason too: a
command or wait on a firmware that is no longer running, a firmware that
stopped answering in an earlier session, a failed Halt, a session ended
by system sleep, frames outstanding without a firmware, and a frame the
firmware answered with an error status.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

v11 (7962e09) on the Neo, same setup as my earlier comment. I ran the whole batch twice in a row: 1080p noise and testsrc via ffmpeg, GStreamer CBR and VBR on pattern=ball at 1080p, and GStreamer H.264 and H.265 at 2408x1506. All runs gave 300/300 frames and decoded clean (HEVC checked on another machine). v4l2-compliance passed 48/48, with 0 DART faults.

The v10 GStreamer failure (it stopped at 240/300 with a bare -EIO) did not come back. The ref_a/ref_b overrun report now stays inside the measured mappings.

Still open: ffmpeg at widths that aren't a multiple of 64 is refused with a stride error, CBR does not pad on easy content, and suspend with an open encoder can't be tested here because there is no wake source.

🤖 Generated with Claude Code

RyanTheTide and others added 5 commits October 30, 2026 21:54
Make the encoder usable as a general V4L2 device:

- Two streams at once. One firmware boot serves both, as macOS was seen
  doing: each session has its own command token counter and Process
  buffers, replies are routed by session token, header word +0x2c follows
  the set of open sessions (0x140 on Open and Start and 0 elsewhere while
  another session is open), and the firmware is halted only when the last
  session closes.

- System sleep. Open sessions are drained, stopped and closed and the
  firmware halted before the power goes. On resume each session is opened
  and started again with the same parameters on its next frame, which
  then starts with an IDR and fresh parameter sets. Clients see no error.

- Input with a different line size. ffmpeg's v4l2m2m encoder fills NV12
  buffers with its own line size, which differs from the 64-byte aligned
  stride the encoder needs when the width is not a multiple of 64. Such
  frames, recognised from the payload size, are now copied to a per-slot
  buffer in the encoder's layout instead of being refused.

- Session images are named after the machine instead of the firmware
  checksum (apple/ave-j700-{h264,hevc}-{start,process}.bin), since they
  are extracted from the user's own macOS installation. Errors name the
  missing or mis-sized file.

- Polling backs off to sleeps of up to a millisecond once a wait goes on,
  so waiting for frames costs next to no CPU time. Which interrupt line
  signals the rings is still not known.

- At the end of an H.264 session the debug log reports how much of the
  per-frame buffer (Process +0x58c8) the firmware used, to size it later.

Also clean up the remaining checkpatch --strict notes.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe how to use the encoder, where the session images come from (the
user's own macOS installation, through the Aurora firmware extractor; no
Apple data in the tree), how sessions, power and system sleep behave, and
the limits. Update the Kconfig help to match, and add a MAINTAINERS entry.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS does not store the Start and Process commands anywhere: its driver
builds them from the session settings and its own defaults. Apart from
the fields this driver already computes, every captured command holds the
same small set of constant words. They are now named tables in
ave-image.c, grouped and commented, and each command is built in a zeroed
buffer from the header, those words and the computed fields. The four
session image files and request_firmware() are gone.

The constants were checked on one firmware build only (AppleAVE2FW
9003.78.0, TEXT checksum 69c54454), so streams refuse to start on any
other build, with a message naming the checksum.

With these facts in place:

- HEVC can run without B-frames, with the low-delay P reference structure
  macOS uses when frame reordering is off (Start +0xe120 table, the
  reorder and reference counts, Process +0xcf8). The per-frame Complete
  that stood in for it is gone.

- H.264 marks a client's B-frame opt-out in Start byte +0x2585, and HEVC
  sets +0x2588, as macOS does.

- CBR pads with filler data once more than 0.5 s of bit rate has gone
  unused, the rule seen in macOS streams. If the firmware's own output
  carries no filler, the driver appends a filler-data NAL unit to the
  access unit; the debug log says which side pads.

debugfs apple-ave/last_start_{h264,hevc} and last_process0_{h264,hevc}
hold the last Start and the first Process of the last session, for
comparison with reference captures.

Field values are facts from our private research (macOS traces); no
copied code.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At 2408x1506 every H.264 setup and HEVC with B-frames ran out of the
encoder's 256 MiB DART window. Three changes make room:

- The H.264 per-frame buffer of Process +0x58c8 was sized after the
  macOS slot spacing (0x364000 at 1080p, scaled with the picture). The
  firmware never wrote a byte of it in any bit-rate session on hardware,
  so it is now one page.

- Session buffers are mapped largest first. The DMA layer aligns every
  mapping to a power of two, and the smaller buffers now fill the gaps
  the large ones leave. The debug log reports how much a session maps,
  and an allocation failure says how far it got.

- ffmpeg's lines are spread out in place when the OUTPUT buffer is one
  the driver allocated (MMAP), which is always large enough for the
  encoder's layout. That needs no extra DMA space; imported buffers still
  get a per-slot copy. The OUTPUT and CAPTURE queues get 48 and 24 MiB.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s buffers

A byte comparison of the generated commands against macOS for the same
settings (1080p30 Main, VBR 4 Mb/s, GOP 30, B-frames), with addresses,
tokens and timestamps masked, found both Process commands identical and
five Start words different:

- H.264 +0x3020: byte +0x3020 is 1. The word was not in the constant
  list because its byte +0x3023 (8x8 transform) varies.
- HEVC +0x2960 and +0x592c: the bytes around the computed max-8-bit flags
  are 1 (+0x2960, +0x2961, +0x2963 and +0x592f).
- HEVC +0x270c = 2 and +0x9608 = 0x10000: the rate-control words that
  only change in constant-quality mode, at their bit-rate mode values.

GStreamer stopped after five frames at 2408x1506: the v14 queue limits
left it 6 OUTPUT and 3 CAPTURE buffers there. Sessions map much less now,
so the queues get 80 and 40 MiB of the DMA window again (10 and 5 buffers
at that size), and a trimmed or refused buffer request is logged at debug
level.

Field values are facts from our private research (macOS traces) and our
own hardware runs; no copied code.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

v15 (ec7c3a7) hardware results on the MacBook Neo (J700). Test kernel 7.3.0-aurora-fleet10nv, module built out of tree from this branch, Stage 2 with the refreshed fleet10 device tree (encoder enabled).

This round, no session-image files are installed. The driver builds every Start and Process command itself from the documented fields. Compared with the macOS commands for the same settings (1080p30 Main, VBR 4 Mb/s, GOP 30, B-frames on), with only the DART addresses, session tokens and timestamps masked, all four commands are byte for byte identical: H.264 Start, H.264 Process, HEVC Start and HEVC Process.

Matrix (300 frames each unless noted; every stream decodes with 0 errors, HEVC checked on another machine)

ffmpeg h264 ffmpeg hevc gst h264 gst h265
1920x1080 OK, 3.6 s OK, 3.8 s OK, 2.9 s OK, 2.8 s
1280x720 OK OK OK OK
1366x768 (stride fix-up) OK OK OK OK
2408x1506 (panel, stride fix-up) OK, 6.1 s OK, 6.1 s OK, 4.6 s OK, 4.7 s
1000x562 (stride fix-up) OK OK OK OK
640x360 OK OK OK OK
  • Rate control:
    • VBR 8 Mb/s on noise: 15.8 MB (within the 2x peak).
    • CBR now pads easy content: 8 Mb/s gives 9.50 MB, 12 Mb/s gives 14.25 MB / 300 frames, in both codecs.
    • CBR on snow: 14.9 MB.
  • GOP: -g 15 gives 20 key frames / 300; -g 1 gives 300/300.
  • Two streams at once: 720p H.264 + 720p HEVC, and 1080p H.264 + 720p HEVC, both 300/300 each.
  • Suspend with the encoder open: live 1080p GStreamer with rtcwake -d rtc1 -m mem -s 20 in the middle. 1500/1500 frames, decodes clean, no error to userspace, for H.264 and HEVC.
  • Idle: 4 s after the last session, runtime PM is suspended and all six venc0 domains are off.
  • v4l2-compliance: 48/48. DART faults: 0.
  • Long run: 10 minutes of live 1080p30 H.264 at 8 Mb/s: 18000/18000 frames, 0 decode errors, SoC at 47-49 C throughout.
  • Quality (SSIM vs source, testsrc2 1080p, 300 frames; software encoders on another machine, same source):
bitrate AVE H.264 x264 medium openh264 AVE HEVC x265 medium
2 Mb/s 0.9829 0.9829 0.9732 0.9831 0.9842
4 Mb/s 0.9901 0.9904 0.9837 0.9906 0.9909
8 Mb/s 0.9968 0.9974 0.9928 0.9971 0.9966
  • Speed: about 80-95 fps at 1080p and about 50 fps at 2408x1506 through ffmpeg, which also generates the source on the CPU. GStreamer does about 105 fps at 1080p. The encoder itself uses little CPU.
  • Thermal: back-to-back ffmpeg runs reached about 100 C on the fanless Neo. That comes from ffmpeg generating the source on the CPU, with the encoder runs in between. The real-time 10-minute run stayed under 50 C.

Field layouts and constants are hardware facts from our research traces; no copied code.

🤖 Generated with Claude Code

The buffer sizes are settled on hardware, so remove what was only there
to measure them: the scan of every working buffer's margin at the end of
a session, which logged at info level each time, and the usage scan of
the H.264 per-frame buffer. Working buffers no longer need a kernel
mapping. The stride fix-up note and the retried restart of a still
powered coprocessor are normal operation and move to debug level.

Assisted-by: LLM
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Smoke test of the head (f494c84, v16: diagnostics removed after the tested v15) on the Neo. GStreamer H.264 and H.265 at 1920x1080 and 2408x1506 gave 300/300 frames each, and ffmpeg H.264 at 1366x768 (stride fix-up) 300/300 with 0 decode errors. v4l2-compliance passed 48/48. There were 0 DART faults and no err/warn lines from apple-ave in dmesg. Marking ready for review.

🤖 Generated with Claude Code

@RyanTheTide
RyanTheTide marked this pull request as ready for review October 30, 2026 12:16
@RyanTheTide
RyanTheTide merged commit 95dee7a into aurora-wip Oct 30, 2026
1 check passed
@RyanTheTide
RyanTheTide deleted the ave/v4l2-encoder branch October 30, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant