← Documentation ・ ← SmartCut ・ 日本語
Everything else in this program takes one recording and writes part of it out. This page is about the three things that happen once several recordings are written into the same file: laying them end to end, making one of them fit the shape of another, and putting something between them.
They arrived together because they are the same question asked three times. A file has one answer for what its pictures are, and a list of recordings has as many answers as it has recordings.
The machinery this needed mostly existed, and the reason is worth stating before the new parts.
A cut of a single recording is a list of kept ranges laid end to end on one output timeline. Each range is anchored to the instant its own pictures start, so an error at one seam cannot accumulate into the next; the sound of each range is mapped against that anchor rather than carried forward; and every segment opens the file it reads for itself rather than sharing one demuxer. None of that was written for joining — it was written because a recording cut into five ranges has four seams in it, and each of them is the same problem a join is.
So the change is not "write a concatenator". It is: let the list of ranges come off more than one recording.
pub struct Reel<'a> {
pub src: &'a Source,
pub plans: &'a [RangePlan],
pub after: Transition,
}
pub fn join(reels: &[Reel], master: usize, output: &str, opts: &CutOptions) -> Result<()>cut is join of one reel, and goes down the same path. That is checked
rather than asserted: tests/run_join_tests.sh writes the same single-clip cut
twice and compares the bytes.
A stream is declared before the first packet is written and cannot be taken back afterwards. So the file has exactly one answer for each of these, and one reel — the master — supplies all of them:
| The pictures | Codec, size, pixel format, frame rate, scan, pixel aspect, colour |
| The timeline | What a tick is, worked out from the master's rate; see Grid |
| The tracks | How many sound tracks the file has, what each is declared as, and what the captions are |
| The tables | A broadcast's own SDT, EIT and TOT, and the PIDs everything is written on |
| The subtitles a disc draws | The master's only; see below |
The master is the caller's choice and not the first row, because the two are different questions. A list of twelve episodes and one trailer wants the episodes' shape, and the trailer may well be at the top of the list.
The pixel aspect is the one the master is shown at: its container's, where an
MP4 or Matroska file states one over its sequence headers, and the bitstream's
otherwise. An .mp4 or .mkv states it for the stream. A .ts, .m2ts, disc,
.mpg or .vob has no such place, so where the master's container and its
sequence headers disagree, every copied sequence header in the file is rewritten
to the master's shape and the seams are encoded to say the same (the Rust
core).
A join copied into one .ts does not change shape partway.
Its pictures and its sound, and its captions. Which of its streams stands in for which of the output's tracks is decided by position — its first sound track is written into the output's first, its second into the second. There is nothing else to go on: a PID means one thing in one broadcast and another in the next, a stream index is something libavformat makes up per file, and a language is missing more often than not. Two recordings of the same broadcast carry their tracks in the same order, which is the case this is for.
Captions are counted the same way, but among their own kind and format: a broadcast lists its captions and its crawl in either order, and counted together the second recording's captions were written onto the master's crawl, and an ARIB track could be handed to a TTML one.
A reel with fewer tracks than the master leaves a gap in the ones it has not got, and a reel with more has the spare ones left out. Both are said once, before anything is written.
A reel with more caption streams of a kind and format than the master has
them — a captioned recording joined onto a master with none, say — has the
spare ones left out the same way, and that is said once too: note: <reel> carries N more caption stream(s) than the master does. Up to 0.8.36 they
were left out unsaid.
The subtitles a disc draws are carried from the master only. Not a limit that saves work — a limit that keeps an answer honest. What a graphics track carries is not packets but the state of a plane: a display set puts something on screen and a later one takes it off, the cut has to replay whatever was standing when a range opens and take it down when the range ends, and a DVD's are converted against a palette read off the disc they came from. None of that composes across files. Two reels would be two planes and two palettes written onto one track.
crate::conform decides it, and only it, so that the answer a list is drawn
from and the answer the cut acts on cannot differ.
A clip matches when it agrees with the master on the codec, the frame size, the pixel format, the frame rate, the scan, the pixel aspect and the three colour fields. A sound track matches when it agrees on codec, sample rate and channel count. Anything else is a mismatch, reported by name:
note: other.ts is not the shape of one.ts, so its pictures and its sound are
written afresh to fit it -- frame size: 640x360 vs 320x240; frame rate: 29.97
vs 25; sample rate: 48000 Hz vs 44100 Hz; channels: 2 vs 1.
A sound track that differs and is still not written afresh is left out of the
note: one dropped with --drop-stream, which is not written at all, and a
reel's TrueHD, MLP or DTS in another shape under a master of the same codec,
which is carried frame for frame
(below).
A TrueHD 2.0 reel joined onto a TrueHD 5.1 master was announced as decoded and
encoded whole while its frames were copied. In the same way, where only a
clip's sound is written afresh, its pictures' differences are left out and so
is the closing word on their quality: a join of two phone clips whose sound
alone differed was told its pictures were only as good as one pass could make
them, though they were copied.
The window asks the same question. The join_fit command takes the list's
paths and which of them is the master, and answers, per clip, whether its
pictures and its sound are written afresh and what differs. The output screen's
re-encode note asks this before it asks for a plan: the other way round, a clip
being written afresh from end to end is announced as copied losslessly.
Everything compared above is read out of the container, and a transport stream is under no obligation to describe itself. Broadcast MPEG-2 states its colour in a sequence display extension that a recording begun mid-programme can be missing altogether. The field order is worked out by libavformat from the pictures it probed, and comes back unknown where those did not settle it. A stream that never stated a pixel aspect ratio has none to read.
Those are gaps in the description rather than differences in the pictures, and one recording of a programme has them where the next week's has not. Read as differences they were an hour of encoding apiece — for two recordings off one channel, which are the same pictures in the same format.
So a comparison where either side is silent is passed over. The output states what the master states, which is the only claim there was, and the silent recording contradicts nothing. The colour is three fields read one at a time, so a recording that states two of them and leaves the primaries or the transfer unspecified is the same colour as one that states all three.
The matrix is the exception: an unstated one is read, not passed over. A
player reads it by the picture's size, and so does the cut where it converts a
reel (below), and a decoder that has met a stated matrix keeps it for the
pictures after it that state none. Passed over, a 720p recording stating BT.601
(what a screen recorder set to its 601 colour space writes) and a 720p clip
stating nothing were one colour, the clip was copied, and its BT.709 samples
played through BT.601, in a .ts, .mp4 or .mkv alike. Now an unstated
matrix is compared as the size reads it (conform::read_matrix); a matrix that
is not one pictures are converted between (RGB, YCgCo) still matches an
unstated one.
What this costs is a real difference only where a recording states something it
is not — and a recording whose description disagrees with its content cannot be
told from one that says nothing, so there was never an answer there to lose.
The rule is conform::stated and conform::same_colour.
Since 0.8.11.
Beside the three colour fields, the range the samples run over is compared:
full, or studio levels 16..235. A yuvj pixel format is full range by
definition; any other is full only where color_range says pc, and unstated is
studio, which is what every broadcast is and what a decoder assumes.
Up to 0.8.10 a clip in the master's pixel format was copied whatever its range — a ten-bit full-range clip joined to a ten-bit studio master, say. Its black went out at nought and was read against the master's declaration, where black is 64: crushed blacks, clipped whites. A difference in range is now a reason to write the clip afresh.
And swscale is made to convert it on the way, which it does not do of its own
accord. It reads a range off the pixel format alone — full for yuvj, studio
for everything else — and a picture already the size and format it is going to
is copied, levels and all. So where the ranges differ the context is built by
hand with src_range and dst_range set before it is initialised, which is what
-vf scale=in_range=..:out_range=.. does. So is one where the ranges agree but
a format misstates one of them — a yuvj clip into a full-range master in a
plain format such as yuv420p10le pc. The still laid over a transition is
brought into the master's range the same way. See blend::Scaler.
Measured: full-range black (0) joined into an eight-bit studio master came out at 16, and at 64 into a ten-bit one; the other way, a studio test pattern at 40..888 spread to 0..965 in a full-range master.
The YUV matrix is converted the same way. A picture scaled from one size to
another keeps the samples of the matrix it was made with, and libswscale
converts between matrices only when it is told both. Left to itself, a
standard-definition reel (BT.601) written into a high-definition master
(BT.709) went out as it was and was shown through the other matrix: the green
of a test card a seventh darker, reds turned orange — every DVD or SD broadcast
joined to an HD recording. Now cut_conform::matrices reads the picture's
matrix and the master's, and where they differ convert_matrices hands both
(and both ranges) to the scaler, as -vf scale=in_color_matrix=..:out_color_matrix=.. does. An unstated matrix is read
by size, as mpv reads it: BT.709 at 1280 across or more than 576 lines, BT.601
below, so a 720p film cropped to 1280x536 counts as HD. A reel whose matrix
agrees with the master's is written as before.
RGB has no matrix, but the YUV on the other side of it has one. A QuickTime Animation or PNG reel (RGB) joined onto an HD master went into its YUV through libswscale's BT.601 and was shown through BT.709, a pure green a sixth darker and red at 255,25,0; an HD reel went into an RGB master through BT.601 the same way. Up to 0.8.46 both were written so. Now both sides of such a conversion are given the YUV side's matrix: the master's for an RGB reel, the reel's own for an RGB master. Where that is BT.601 nothing changes.
A palette reel (a 256-colour QuickTime Animation, a palette PNG .mov, an
8-bit .avi) has its colours looked up into RGBA first
(blend::palette_as_rgba), as a palette image does, and goes on as an RGB
reel: libswscale turns a palette into YUV with BT.601's numbers whatever matrix
it is told, and up to 0.8.46 such a reel went into an HD master with red at
255,25,0 even after the RGB fix above. Its output is now the RGB reel's to the
byte; on an SD master only the rounding moves. No reel can be written afresh
into a palette master (libswscale makes no palette), and that is refused with a
sentence of its own.
A month of broadcast: 1186 recordings off 32 channels, read with conformdiag --group. Twelve channels hold recordings that are not all one shape and
differ in nothing but the scan; three more differ in something real (a
programme broadcast at 720x480 among a channel's 1440x1080, one file that
would not open). Across the whole corpus, 1088 recordings come back
interlaced top field first, 76 progressive and 22 bottom field first — and 40
of those are the odd one out on a channel whose other recordings are all the
same.
The scan is libavformat's reading of the pictures it probed, not a field the stream states outright. A Japanese broadcast carries progressive-coded frames inside a 1080i stream constantly — most animation is coded that way — so whether the frames a probe happens to see are progressive depends on whether the recording began in the programme or in the commercial before it. The same programme two weeks running comes back progressive once and interlaced once.
That is a whole clip re-encoded for a reading rather than for a difference, and it is what somebody joining two episodes of one series actually hits.
The clincher is what the pictures themselves say. Two recordings off one channel, a day apart, one read as interlaced and the other as progressive; sixty seconds sampled from the middle of each:
| container says | progressive-coded pictures | interlaced-coded |
|---|---|---|
| interlaced, top field first | 507 | 1138 |
| progressive | 1090 | 427 |
Both recordings hold both kinds, and the container's single answer is only which way the frames a probe happened to see leaned. There is no property here to conform one clip to the other's.
So the scan does not cost the pictures anything: What::costs_pictures is
is_video less Scan, and fit_of asks that one. Every picture carries its
own flags -- a frame is coded as a frame or as two fields whatever the track
is declared as -- so a decoder reads a copied one as it reads the recording's
own. The reference tool does not re-cue for it either.
It is still compared and still reported. A run that copies such a pair says so once: "were read as different shapes -- scan: interlaced, top field first vs progressive. That is a reading rather than a difference." A difference that is seen and decided about is worth a line; one that is silently dropped is a tool nobody can check.
29.97 and 30 are not the same rate. The tolerance is a ten-thousandth, relative; what it is for is the last figure of a rate that arrived as a decimal, and it has to be well under the thousandth that separates 30000/1001 from 30. A timeline built on either of those shows the other drifting a frame every thirty-three seconds.
A variable master is compared by the rate both were authored at. A
variable recording's average is not a rate it has: two clips off one phone,
both declaring 60 and averaging 56.59 and 58.26 for whatever the light did, were
two rates by their averages, and the second was written afresh whole. Into a
variable master nothing stands on the average -- the timeline is divided
finely and every copied picture is placed by its own time -- so where the
master varies and the two declare the same rate, the clip is copied
(conform::same_authored_rate). A constant master is still compared by the
average, since a varying clip would be rounded onto its frames; and a declared
1000, which is Matroska's clock and no rate, is not taken for one.
There is no partial answer available. A clip of another size has no picture in
it the file could carry, so the planner's whole business — find the stretch
that can be copied, re-encode the partial GOPs at its ends — has nothing to
divide. plan::reencode_range hands over one segment per range, and
conform_segment decodes it and writes it into the master's shape.
Two things there are worth naming.
The pictures are placed on the master's grid, not their own. A seam places each decoded picture at its own instant, and that is right where the pictures around it came at the same rate. Here they did not: a 25 fps clip placed by its own instants into a 29.97 file lands between the frames of every second one, and what comes out is a stream whose pictures are neither one rate nor the other. So the output's own frame instants are counted off from the range's start and each is given whichever source picture was on screen at that moment. Slower in than out and a picture is written twice; faster and one is passed over. Measured: a 25 fps clip of 8.02 s joined into a 29.97 file came out 241 output frames, which is 8.02 s of the file's own rate.
That is the rate conversion, and it is the one a smart renderer can honestly offer. Nothing here invents a picture that was never taken.
Except into a master whose pictures do not come at its average
(cut_conform::own_instants, which is VideoInfo::off_the_average; see
the Rust core).
A screen recording sends thirty pictures a second while something moves and
holds one for seconds while nothing does, so it averages nine: a broadcast reel
counted off at that was written at nine a second, 207 of its 682 pictures, and
a second screen recording joined onto it lost the pictures of every stretch
where its screen moved. The output's timeline there is divided finely, so each
picture is written once at its own instant, as a seam's re-encode places them,
and is up until the next one; a picture held for longer than a second is
written again each second, since an MP4 states each step in 32 bits of a
timescale held only to the master's own length. Two pictures closer than the
timeline can tell apart keep the first. The encoder is told the reel's own
rate rather than the master's average, since it spends its bit rate over as
many pictures as it is told come in a second: a broadcast reel at its own
thirty into such a master came out at 3.98 Mbit/s against the master's 1.01.
Not for MPEG-1 or MPEG-2, which write the rate into every sequence header, and
not through a transition, whose progress is counted in the master's frames on
both sides. A variable master whose average is its rate (a phone's dropping a
picture now and then) is counted off as before, and so is an interlaced one
declaring its fields.
Key pictures come as often as the master's. A seam is written with one key
picture, at its head, because it is shorter than the GOPs either side of it. A
reel written whole was written the same way, up to 600 pictures to a key: a
phone clip joined onto a camera's 720p master went into a .ts as one IDR and
199 pictures after it, a seek into the reel decoded nothing, and an hour of a
DVD conformed onto a DVD master had a key picture every 20 seconds. A reel
written whole now takes the master's own spacing, the median of the gaps
between its entry points (cut::master_gop), so it can be entered where the
master can.
The scaler is rebuilt whenever the decoder starts handing over a different shape. Not paranoia: a broadcast recording can change frame size mid-file where the station switched feeds, and a scaler built for the first shape silently refuses the second.
One answer per track for the whole output, not one per reel. A track is declared once, as one thing, and what a stream says about itself has to describe every frame on it — so a track written from the master's own packets for one stretch and from an encoder for the next would be a track whose two halves are described by one header that fits one of them.
What crosses a reel boundary is the encoder; what changes is the decoder in
front of it. audio::Reencoder::retune swaps the decoder, the rate its samples
arrive at, the resampler between that rate and the output's, and the last
timestamp seen — and keeps the encoder, the count of samples fed, and the
samples already decoded and not yet framed. A second encoder started at the
join would number its frames from nought and write the rest of the file on top
of what is already there.
Not where the master's sound is TrueHD, MLP or DTS. That sound is carried
through frame for frame, never re-encoded (see the sound), so the
track is the master's own frames and a reel whose sound is another codec has
nowhere to go. Up to 0.8.27 the cut went ahead
regardless: an AAC recording joined onto a TrueHD one came out with its 100
seconds silent and an exit code of 0, and AAC or E-AC-3 joined onto a DTS master
went into the DTS track as bytes no decoder of it reads. That join is now refused
before anything is written (cut::lossless_master_refusal), naming the reel,
what differs, and what does work: taking that reel as the master (--master N,
or the master clip in the window), or naming a codec for the whole track
(--audio-codec). Where the reel's own sound is TrueHD, MLP or DTS the first is
left out, since swapping them meets the same wall. --sound-only refuses the
same join in the same words. A reel of the master's own codec in another shape
-- 2.0 TrueHD joined onto 5.1 -- is carried as it always was: its frames say
what they are, and a decoder follows them. --sound-only now carries it the
same way; up to 0.8.27 it re-encoded such a DTS join whole and stopped on a
TrueHD one.
A transition is the one thing in this program that cannot be smart rendered. Everything else here exists to leave the recording's own pictures alone. A transition asks for pictures that are in neither recording — a frame that is half of one and half of the other, a frame a quarter of the way to black — so every frame it covers is made here, whatever the two clips have in common. Asked for two seconds, it costs two seconds of encoding at each join and nothing anywhere else.
It belongs to the clip before it, which is where the reference tool puts it and the only place that reads in the order the file is written. On the last clip there is nothing to give way to, so anything but a fade becomes a fade to black at the end of the file.
| Fade through black or white | The clip before goes down to the colour over half the time, the clip after comes up out of it over the other half. Each gives the part of itself the transition covers, and the output is as long as it would have been without one |
| Dissolve, wipe, slide | The two clips are on screen together for the whole of it. Both give that many seconds, and the output is that much shorter |
Not a choice: it is what the two kinds are. A fade through black never has both clips up at once, so asking it to consume both at once would be asking it to throw one of them away; a dissolve has nothing to show but both at once.
A fade on a reel whose pictures vary shares out the stretch it has. Its
frames are counted in the master's, but the stretch runs between two of the
reel's own pictures, where the body before it ends and where the body after it
opens, and on a variable-rate reel that is not a whole number of master
frames. Laid whole, the last frame ran up to a frame past it, and every
picture after it was written that much late: a 30 fps test clip with jitter
faded in from black was 30 to 38 ms behind its sound for the rest of it, a
test screen recording averaging 1.2 pictures a second 575 ms. Going out, the next reel was pushed
along with its sound, which then opened before its range. On a master whose
timeline is divided finely enough (see Grid), such a fade now writes as many
frames as before, each the picture it was, spread evenly across the reel's own
span (cut_conform::pictures_afresh). A fade on a constant-rate reel, and a
dissolve, wipe or slide, are still laid in whole master frames.
The sound is cut against the range — one boundary decision per end, one anchor
per range — so a range split in two to make room for a fade would put a seam in
the sound where the picture has none. Instead the range keeps its bounds and
gains a segment, and Segment::retouch says what that segment is.
The two exceptions are the two ends of an overlapping crossing, where the range really does lose those seconds:
- The clip after a dissolve gives up its first
seconds— they are on screen during the crossing, written by the crossing — so its range starts that much later, sound and all. - The clip before gives up nothing. Its pictures are consumed by the crossing but its sound plays through it, which is what keeps the sound in step with the pictures on either side.
The segment begins on a picture. The body of a range writes the pictures that
begin inside it; the transition after it opens on the picture on screen at its
first instant. A transition whose length is not a whole number of frames — the
default second at 29.97 is 29.97 frames — began part-way through a picture, so
both wrote that one: every picture of the transition came a frame late and the
clip's own last picture was left out. Where the body is not written afresh whole,
its end is moved to the start of the picture on screen there
(plan::on_screen_at) before it is planned.
The clip after resumes on the picture after the last one the crossing
showed. An overlapping crossing writes a whole number of the clip after's
pictures, and the range after it used to resume at its start plus the
crossing's length in seconds — which, on those same 29.97 frames, is inside
the last of them, so that picture was written twice or the one after it was
lost. The crossing now counts the pictures it writes, and a copied clip after
resumes on the first picture to begin after the last of them, read off its
packets (plan::picture_after); a clip written afresh on the master's grid
resumes that many of its frames on.
Reading those packets has to get past what comes before the first key picture.
An ffmpeg -c copy of a broadcast into an .mkv keeps the open-GOP leading
pictures in front of it, and Matroska, which cannot time a block before 0,
hands them over with no time at all. Taken for pictures that could not be read,
they made every read from the head of such a file answer nothing, and the clip
after fell back on the crossing's length in seconds: up to 0.8.29 a remuxed
H.264 recording joined whole after a half-second slide resumed a picture late,
one of its pictures never shown and the rest a frame off their sound. Those
leading pictures are now passed over, as no range shows them. Where a read
still answers nothing (a DVD's reference pictures carry no presentation time),
the clip resumes on the frames the crossing counted rather than on the
seconds.
And not every recording has a grid to work a bound out on. A picture held
for three fields moves every picture after it half a frame — soft telecine, a
DVD of a film, an interlaced broadcast now and then — and a remux need not say
so; a recording with no fixed rate has no grid at all. For those, and for
MPEG-1, MPEG-2 and VC-1 always (plan::pictures_off_the_grid), the end of the
body before a crossing is read off the pictures (plan::picture_on_screen),
a field-coded picture's two fields counted as one. On a soft-telecine clip
joined to itself with a one-second dissolve, output pictures 350 and 351 were
both its picture 84.
The same goes for the body before a crossing into a master whose reels keep
their own instants (above), whatever the clip: ended on the output's frames, it
stopped part-way into a picture, which stood for a sliver and opened the
crossing again.
Measured on 2026-10-09: soft-telecine joins from the command line went from 8 wrong in 12 to none in 67, variable-rate ones from 6 in 6 to none in 10. Three more doublings went with them: a reel written afresh on the master's grid showed a picture twice at every transition, a range short enough to have crossings at both ends did so at each, and a field-coded recording had its second field taken for a picture's start. A file with no transition in it is written byte for byte as before.
So the sound is not mixed. The clip before plays through the crossing and the clip after starts where it ends. That is a decision and not an omission: mixing two tracks would mean decoding both and encoding the result, which makes every sound track in the file one this program writes rather than one it copies — for a second of crossfade in an hour of programme. What softens the change instead is a fade, which costs nothing that is not already being spent.
The captions do not change over where the sound does. A caption plane shows
one recording's lines, and what a dissolve, a wipe or a slide leads the eye to
is the clip after, so from the instant the crossing begins — the instant the
clip after's pictures start to show — the captions are the clip after's. A
broadcast's ARIB captions and a 4K recording's TTML subtitles follow the same
rule (cut::take_caption, cut::read_ttml_crossing):
- The line the clip after has up where its pictures start goes up on the crossing's first frame. What it puts up or takes down inside the crossing goes out at its own time.
- The clip before is carried up to the clip after's first word inside the crossing, and no further: a broadcast's first caption statement, a line or a clear, or a 4K recording's first document. A line the clip before has up then is replaced there — a 4K document is cut short at that instant — and what it would send from then on is left out. Where the clip after has a line up at the start, that is the crossing's first frame.
- Where the clip after has nothing up and says nothing inside the crossing, the clip before is carried to the end of its range, as any range is: what it sends inside the crossing, a clear among it, goes out at its own time, and a line still up at the end stands until the clip after's first. The same where the clip after has no caption track in that place.
Up to 0.8.35 the clip after's captions were read back only from where its own range begins, which is the crossing's end. Its opening line came up a crossing late, a line it put up and took down inside the crossing was lost, and the clip before's last line stood over the clip after's pictures for the whole crossing. A captioned recording joined to a cut of itself with a one-second dissolve had the clip after's opening line at 11.02 seconds rather than 10.00; with three seconds, two of its lines and a clear were lost, and the clip before's line stood from 9.58 to 14.22. A 4K title joined to a cut of itself with a three-second dissolve lost the clip after's first line whole.
A disc's subtitles go with the pictures the same way, where there is
anything to go: they are carried from the master only, so a crossing changes
them only when the clip after is the master. Then its PGS or DVD subtitle up
where its pictures start goes up on the crossing's first frame — a DVD
subtitle still comes down when it did in the recording — and what it puts up
inside the crossing goes up at its own time (cut::read_graphics_before and
cut::read_subpictures_before, read on to where the range begins). The clip
before has none in the file to give way. Where the master is the clip before,
its subtitles run to the end of its range, through the crossing, as they
always have. Up to 0.8.36 the master after a crossing had its subtitles read
back from the crossing's end: a disc title joined onto a cut of itself with a
three-second dissolve lost a subtitle that was up for 2.1 seconds inside the
crossing and put the next one up 0.78 seconds late; a one-second wipe put the
first one up a second late. The same holds for .sup and .idx/.sub side
files.
A fade is not a crossing. The two clips' pictures follow one another through
the colour, and their captions follow one another with them, as they always
have. A join with no transition or a fade is written byte for byte as before;
at a dissolve or a wipe, only the caption stream differs, and the disc's
subtitle stream where the master is the clip after. The seam window's
preview (crossview.rs) draws no captions at all.
The fade at a cut inside one recording is CutOptions::audio_fade, one number
for the run. A join between two recordings is not that. How the programme that
is ending should end and how the one that is starting should start are two
questions, and one number cannot answer them: a programme that ends on its own
theme wants a long way down and nothing at all on the way up.
So Transition carries two lengths. fade_out is how long the sound of the
clip before takes to leave, fade_in how long the clip after takes to come
back. They are independent of the picture: a join whose kind is None can
carry a fade. A crossing takes material off both clips to happen in
(Transition::takes) and a fade is written over material that is staying, so
it changes no length.
A fade out lands on the last sample the track still has rather than on the
end of the range. A broadcast recording's audio often stops half a second
before its pictures do, and a ramp aimed at the range's end is still a quarter
of the way up when the sound runs out — a fade out that stops dead at quarter
level. boundary_patches takes the smaller of the range's end and the end of
the frames in hand. (That half second is filled from the next clip's own
sound — see min_start — which happens with or without a fade.)
cut::fade_lengths builds the pair — head and tail — for every range of the
job in the order they are written: audio_fade at a seam inside one recording,
and the join's own two numbers where one recording gives way to the next. The
two ends of the output are not seams and get nothing, which audio::fades_for
settles from the range's place in the job rather than from that table.
Every one of them is the same operation: for each sample of the output, choose
between two numbers or mix them, by a function of where the sample is and how
far through the crossing it is. crate::blend is the only place any of it is
written — four times over would be four places to get the chroma subsampling
wrong.
| Fade | Each sample moved towards the colour. Studio black and studio white, 16 and 235, not 0 and full scale: the pictures around it were coded in the same range, and a frame that goes past where they can reach is a frame that flashes |
| Dissolve | Each sample of one mixed with the same sample of the other |
| Wipe | An edge travelling across the frame, both pictures standing still |
| Slide | The same edge with both pictures travelling: the one before is pushed off the side the next one comes in from |
The edge is rounded rather than truncated into each plane's own samples, so the luma edge and the chroma edge are at the same place on screen at every instant. A half-sample apart is a coloured fringe travelling across the picture.
Everything is integer arithmetic — a per-sample float multiply over a 4K frame
is the difference between a transition that encodes as fast as it writes and
one that does not. The formats accepted are planar YUV, eight or ten or twelve
bits, which is what nearly every recording decodes to, and RGB. Planar RGB
(gbrp and its kind) is blended as it is. Packed RGB of eight or sixteen bits
a sample (a QuickTime Animation master's rgb24 or argb, a PNG .mov's
rgb48, the bgr0 FFV1 hands back) is turned into planar RGB around each
blend and back; both ways only rearrange the samples, and the alpha, which
libswscale moves by one on the way, is copied across by hand. In RGB a fade
goes to nought and to full scale, which are RGB's black and white. A fourth,
alpha plane (ProRes 4444, a PNG .mov with transparency) is dissolved and
wiped with the rest; up to 0.8.46 the crossing's alpha was whatever the new
frame's memory held, and an RGB master was refused at the crossing while the
window that sets it showed it. A palette, packed RGB of five or six bits
(rgb555, rgb565) and x2rgb10, which do not come back exact, and anything
else are refused rather than guessed at.
The names are the reference tool's, which are the ones a person who has used one of these before will look for: None, Back, Bounce, Circle, Elastic, Exponential, Power, Sine, Quadratic, Cubic, Quartic and Quintic, each in In, Out, In-out or Out-in. What each of them is, is the standard curve of that name; there is nothing invented here.
Two of them leave the interval by design — Back overshoots and Elastic rings past both ends — and that is left alone rather than clamped at the source. A dissolve's mix clamps where it uses it; a wipe's edge does not, because an edge a little off the frame is simply an edge off the frame. Every curve in every mode starts at nought and arrives at one, which is checked: a transition that did not would show a step at one of its ends, which is the thing it exists to remove.
A still — a title, a card — drawn over the transition and nowhere else, coming up over the first quarter of it and going down over the last so that it is never cut on or off.
Nowhere else, and that is the whole reason it is offered at all. Drawing a picture over a programme means re-encoding every frame it covers, and a smart renderer that quietly re-encoded an hour because somebody put a caption on it would not be one. The frames a transition covers are being written afresh already.
The image is read through libavcodec like everything else — PNG, JPEG, BMP, anything it has a decoder for — and scaled twice: once into the output's own pixel format for the colours, and once into RGBA for the alpha channel. An image with no alpha of its own is solid. It is stretched to the frame rather than placed in a corner, because where to put a smaller one is a question with no answer that suits everybody and an image with an alpha channel answers it itself.
The image is RGB, and is made into the frames' YUV with their matrix.
libswscale makes YUV out of RGB with BT.601's unless told another, so a title
over a high-definition master went in through the wrong matrix and showed in
other colours than the image -- a pure red some 8 % darker. It is now given the
matrix the master states, read by size only where it states none
(blend::read_laid_in, conform::read_matrix): an HD master tagged BT.601
gets BT.601 and an SD master tagged BT.709 gets BT.709. Up to 0.8.45 the cut
read the master's matrix by its size alone; over an untagged SD master nothing
changes.
A palette image (an indexed PNG, an 8-bit BMP) is RGB too, but libswscale turns a palette into YUV with BT.601's numbers whatever matrix it is told: up to 0.8.46 such an image over an HD master came out with red at 255,24,0 where the same image saved as RGB was right. Its colours are now looked up into RGBA first, at its own size, and it goes on from there as an RGB image does: an opaque one gives the RGB image's output to the byte. A GIF is decoded to RGBA already.
A join's transition is set in a window of its own, with the seconds either
side of the seam composited and playing. crossview.rs is the path that
produces them.
None of the arithmetic is written here. Every sample of the preview and
every sample of the output go through the same blend.rs, and how far
through the crossing each instant is comes from the same transition.rs.
The way to keep a preview and an output from disagreeing is for there to be
one of them.
The sound's fade is the same bargain. The preview is handed the seconds as
Heard::fades and rides audio::fade_shape over them -- the curve the cut
writes with (audio::fade_run), in the other one's units: that counts samples
because it is writing them, this counts seconds because it is playing. How
many seconds is right is not a thing anybody settles by reading, so it has to
be answerable in this window, the same as the pictures.
What is written twice is the schedule — which instant shows what. The
cutter builds it as plan segments around a whole edit (ranges_with_transitions);
the preview has no edit, only two recordings and a setting, so it builds the
same thing around one seam. The tests at the foot of crossview.rs are what
hold the two answers together.
| Preview | Output | |
|---|---|---|
| Composited into | the preview's own size, yuv420p |
the master clip's shape |
| Scaler | FAST_BILINEAR — a picture is looked at once and dropped |
BICUBIC — it is paid for the length of the reel |
| Sound | the two clips in turn to one output device (play_audio_across), each on the track in the place the master's first kept track holds (heard_nth) |
the same order and the same track (Threads), written |
A reel of another matrix is scaled into the preview through the same
cut_conform::matrices and convert_matrices the output uses, with the
master's matrix as the one to reach, so an SD reel joined to an HD one shows
in the window in the colours it is written in. The composite is then tagged
with that matrix before it becomes a JPEG. A new frame at the preview's size
states none, the JPEG reads an unstated matrix by size, and a few hundred
pixels across is BT.601: up to 0.8.44 every join staged at an HD master was
shown through the wrong matrix in this window, while the output was right.
The transition image is laid into the window's composite in that same matrix
(crossview::overlay_of). Read by the preview's own size it went in through
BT.601 and was then shown through an HD master's BT.709: up to 0.8.45 a red
title was orange in this window and red in the output. A master stating a
matrix no reel is converted to or from (YCgCo, a reserved value) is read by its
size, as the editor's stills read it. A master whose pictures are RGB or a
palette is composited in BT.601, stated or not, and the composite is tagged so:
the scaler turns its own pictures and every RGB reel's into YUV through BT.601,
and a YUV reel is brought from its own matrix to BT.601. Read by its HD size, a
QuickTime Animation master showed BT.601 samples through BT.709 in 0.8.45, and
left untagged in 0.8.46 it still did so in a window wide enough for a picture
1280 pixels across, which is read as HD.
The composite is done in studio-range yuv420p and the conversion to JPEG
comes after it. The other way round, the studio black Shade::yuv gives
would land at a different brightness in the full-range yuvj420p a JPEG
wants.
tests/run_join_tests.sh, seventeen checks, fixtures generated so it needs
nothing but ffmpeg.
| Two clips of a shape | The file is the two lengths, holds every picture of both, its sound runs to the end, and it decodes with nothing to say |
| Three clips | In the order given |
| One that does not match | Every picture the master's width, the file as long as the two clips, the mismatch named before a byte was written |
| A fade | The file keeps its length, and the middle of the fade is black |
| A dissolve | The file comes out its own seconds shorter |
| A wipe | Writes and decodes |
| A crossing against a short range | Held to whichever of its two ends allows less, so the clip after it loses nothing the crossing did not show |
| A cut of one recording | Byte for byte what it was, and the run says nothing about joining |
On real material: two twenty-second stretches of a broadcast recording came out 39.91 s and 1,194 pictures against 598 + 596 written separately, with the sound ending within a frame of the pictures and no complaint from the decoder. A 720x480 clip joined to a 1440x1080 master came out 1440x1080 throughout, 22.02 s against a wanted 22.04, and the burnt-in timecode of the second clip read 6.039 at six seconds into it.