Repository navigation
Evaluate FFmpeg 8/9 support and AV1 MP4 sample-description switching #14
Description
Following the discussion in #12, I would like to discuss newer FFmpeg support as a separate follow-up.
The level-only differences are already handled by #12 and your subsequent fixes. The potential additional benefit is handling other valid boundary/source configurations through multiple MP4 sample descriptions, where container and playback checks establish that this is safe, while retaining the copied video.
My earlier investigation found the multiple-description path in FFmpeg 8.0's MOV/MP4 muxer. It consumes AV_PKT_DATA_NEW_EXTRADATA and records which description applies to each chunk. The AV1 MP4 binding permits multiple av01 sample entries.
In a targeted FFmpeg 9.0.2 remux test of the earlier rav1e-boundary fixture, all 214 packet payloads were unchanged. The output switched descriptions as 2 → 3 → 2, and the configuration fields matched the corresponding samples. FFmpeg 7.1.1 wrote one description and failed that configuration check. This was a muxer experiment, not a complete SmartCut migration or player-compatibility test; the 8.x conclusion in that comparison came from source inspection.
I suggest treating this as two stages:
- Evaluate the FFmpeg version, Rust bindings, build/package changes and regression results, keeping the existing smart-rendering policy initially.
- Add AV1 MP4 description switching separately, with correct extradata propagation and tests for sample-to-description mapping, strict decoding, seeking, timestamps, copied payloads and representative players. Retain guarded fallback for unsupported cases and keep Matroska/WebM policy out of this change.
Upgrading FFmpeg alone would not enable this correctly in SmartCut; the application still needs to supply the configuration changes to the muxer.
Would you prefer retaining FFmpeg 7.1 support alongside newer builds, or raising the minimum supported version? Is there a preferred target between 8.x and 9.x, or a distribution/build constraint that should guide the choice?
I can provide the reproducer, packet hashes and sample-description checks from the earlier experiment as a starting point.
Activity
DeepRegular commentedon Nov 6, 2026
Thank you for opening this separately, and for the careful write-up. Splitting it into two stages makes sense to me, and I agree that a newer FFmpeg alone would not get us there: SmartCut would still have to hand the configuration changes to the muxer itself.
To answer your questions first:
Minimum version. I'd like to keep FFmpeg 7.1 as the minimum for now. Debian 13 ships 7.1, and that is what the development machine and CI run, so raising the floor would drop anyone building from source there. I'd revisit this once the distributions we support move to a newer series by default.
8.x or 9.x. I'd rather not pick one yet. When we do move, I'd follow whichever series the next Debian release carries at that point.
Some context on why I'm cautious about stage 1. The 7.1 dependency goes deeper than the version number in
Cargo.toml.ffmpeg-sys-nextselects its bindings by version,av1_mp4.rsdeclares parts ofbsf.hby hand because the 7.1 bindings omit it, and every suite plus the byte-for-byte Linux/Windows comparison in CI is pinned to 7.1 behaviour. Moving to 8 or 9 would mean re-verifying the whole smart-rendering path, not just the AV1 MP4 case. I'd also prefer not to support two binding generations at once (for example, enabling description switching only when built against 8+), since that doubles the CI matrix for one narrow case.On stage 2: today, when the AV1 configurations cannot share one description,
Guardrefuses and we fall back to the safe path, so no broken files are produced. The gain is avoiding a re-encode in those cases. Before committing to the migration, I'd like to understand how often that actually happens. Now that the level-only differences are handled, which real encoders or sources still produce configuration changes at a seam? Player behaviour also matters a lot here. A copied stream that some players cannot follow across a description switch would be worse for users than a re-encode, so representative playback results would weigh heavily.So I'd be glad to take you up on the offer: the reproducer, packet hashes and sample-description checks would be a good starting point, along with any examples of where this configuration comes up in practice. With that, we can decide how to prioritise this against the cost of the FFmpeg move. I'll keep the issue open for that.
SteveYu000 commentedon Nov 7, 2026
Measured AV1/MP4 benefit of a newer FFmpeg
I tested this against
e4da05de5f8b04ce67951838bbaaeb787ea38566, with actual Windows/MSVC Rust builds for FFmpeg 7.1.1 and 9.0.2. The result supports there is no need to upgrade the FFmpeg version but needing to fix one smaller issue.I compared four variants: unchanged 7.1; 9.0.2 with the same guard policy; 9.0.2 with an experimental multiple-description path; and 7.1 with the source chroma location passed to the boundary encoder.
Of 26 successful encoder configuration requests, yielding 20 distinct input hashes, four still caused a whole-selection retry in both unchanged-policy builds: libaom 8-bit and 12-bit sources with native
chroma-sample-position=1or2. These were produced by the encoder, not by rewriting the headers afterward. Several generic chroma-location options produced byte-identical default outputs; I have not counted those duplicates as additional bitstream coverage. The other tested configurations, including the earlier level-only families, did not need that retry. Two additional generation attempts failed and are excluded from those 26; a requested SVT High Tier case actually wrote Main Tier, so I am not claiming High Tier coverage.Both experimental paths removed all four retries. The smaller 7.1 change was to pass
codecpar.chroma_locationtoAVCodecContext.chroma_sample_locationwhen opening the boundary encoder. The tested SVT and rav1e boundaries then stated the source's chroma position, without changing copied packets or requiring another MP4 description.For timing, I used three 30-second clips, two selections and four variants, with one warm-up and three sequential measured runs per combination: 96 exports in total. In the affected selections, the final output went from 724 re-encoded pictures to 74, or from 680 to 130. Multiple descriptions saved 14.1–58.2% of median elapsed time against the unchanged policy on the same 9.0.2 SDK. The 7.1 chroma propagation change achieved exactly the same copied-picture counts and saved 17.2–54.5% against unchanged 7.1. The default-chroma controls kept the same copy counts throughout.
There are two important limits to that evidence. The inputs are targeted encoder configurations, not a survey of distributed recordings, so 4/26 is not an estimate of how often users hit this. One longer clip used real local recording content, but I controlled its AV1 encoding parameters; it is not an independently found AV1 recording with this problem. Also, the SDKs bundle different SVT versions, so I have kept same-SDK policy comparisons separate from the overall speed difference between 7.1 and 9.0.2.
The 104 short outputs and 24 representative long outputs passed the container, strict decode, timing and copied-payload/pixel checks. The long outputs also passed 24 libaom/dav1d comparisons, 120 software seek checks and 18 NVIDIA decode/seek checks. The 9.0.2 workspace unit run passed 594 tests with the four previously established Windows exclusions; clippy completed. This is still a targeted AV1 experiment, not verification of the whole migration.
Player results were less clean. VLC 3.0.24 with dummy video output reached EOF, but both multiple-description multi-range outputs repeatedly logged four timestamp-conversion errors when started at 11.84 seconds at normal speed. The 7.1 chroma-aligned and unchanged-policy controls had no errors in that normal-speed test. The same two multiple-description results repeated after the other jobs had stopped. I have not established whether those log errors cause visible dropped pictures, so I would not call player compatibility resolved.
The existing extraction path already supplies
NEW_EXTRADATA: merely building against 9.0.2 also produced two descriptions for some field-compatible sequences, although the guard still forced the incompatible cases to retry. Thus description count alone is not a measure of saved re-encoding.I have the native encoder reproducer, exact input and executable hashes, packet hashes, description mappings and timing records. My recommendation from this set is to pursue a properly scoped chroma-location propagation fix first, and leave migration open for cases that cannot be solved by matching the boundary encoder.
fix: pass the recording's chroma location to the seam encoder
DeepRegular commentedon Nov 8, 2026
Thank you for running this so thoroughly, and for pointing at the smaller fix. I agree with your recommendation, and it's now in 42b2a44.
The seam encoder now takes the recording's chroma location from
codecpar.chroma_location. I reproduced your libaom case here on Linux with FFmpeg 7.1.5: a 12-second clip cut at 1.3–9.7 s. Withchroma-sample-position=1or2, the old build retried and copied nothing. The new build keeps all 144 copied pictures bit-identical, and--verifypasses.One difference from what you described: I don't pass "left" on for H.264 and HEVC. For those two it is the default, and the decoders report it even when the stream doesn't signal it. Passed on unconditionally, x265 started writing an explicit
chroma_loc_info(type 0) into the seam SPS, which the copied pictures don't carry. AV1 always gets the value, since left there means position 1. With that rule, the H.264, HEVC-without-the-field and MPEG-2 outputs are byte-identical to before. An HEVC source signalling type 2 now gets seams that state 2 as well, where they used to state nothing.Your VLC results with multiple descriptions are another reason I'd rather not go there yet. So, as you suggested, I'll leave this issue open for the cases that matching the boundary encoder can't solve. If any of your Windows SDK configurations still retry with this commit, I'd be glad to hear which ones.
DeepRegular commentedon Nov 8, 2026
The chroma-location fix is now released in v0.8.42, with the Linux and Windows builds.
A few things were added after 42b2a44:
The release builds were compared against 0.8.41 on Linux and on Windows (under wine). All six regular comparison cuts (broadcast ranges, a join and a recorder .m2ts) came out identical.
As before, I'll keep the issue open for the configuration changes that matching the boundary encoder can't solve. Thanks again for the measurements: they made the right fix obvious.