Repository navigation
Conversation
The renderer keeps the last decoded frame for redraws, several megabytes at 1440p and above. A memory warning now drops it and flushes the texture cache; the screen keeps showing the frame until the next one arrives. The warning records the app's footprint, the stream size and the state, so reports can tell the app's own memory from system pressure.
A frame whose repair deadline passed was dropped even when no newer frame waited behind it, so dropping it unblocked nothing and the keyframe asked for in its place came later than a repair would have. On a still desktop frames come every few hundred milliseconds and WiFi power saving holds traffic as long: one iPad dropped 503 of its 525 frames this way and asked for a keyframe each time. Such a frame is now held and asked for again, less often each time, until a newer frame arrives (then it gets one more repair window) or a second has passed. Repair held at most eight frames, 67 ms on a 120 Hz iPad, shorter than the repair deadline, so frames whose repair was on its way were evicted. The bound is now 32 frames. Over USB, where nothing is repaired, a partial frame was dropped 100 ms after its first fragment. Through a slow usbmuxd a large keyframe takes longer than that to arrive, so the iPad dropped it and asked for another. The timeout now runs from the latest fragment, and a delivered keyframe retires older partial frames without asking for another one. The Rust mirror and the shared traces follow; a WiFi simulation measures drops and keyframe requests.
The USB video queue held two frames. When usbmuxd on a slow PC fell behind, the next frame cleared the queue, cut the frame being written short and asked for a keyframe. The keyframe is the largest frame, so it overflowed the queue in turn and was cut short itself; the iPad received partial keyframes over a lossless cable, reported them as lost fragments and asked for yet another keyframe. A full queue now drops the newest frame and its dependents instead of the queued ones, a frame that has started is always finished, and a keyframe always passes, replacing what is still queued. The queue holds six frames, room for a keyframe burst to drain. A link that keeps losing frames asks for keyframes at once, then after 250 ms, 500 ms, 1 s and 2 s, and immediately again after five clean seconds; a deferred request is sent even if no further frame comes. A paused-clock simulation drives the real link through a throttled tunnel. At 0.55 MB/s, where the stream averages 0.53 MB/s, 0.3.1 decoded no frame in 20 s and forced 317 keyframes; now all 1200 frames decode.
The cable loses nothing, so over USB the iPad's fragment loss, repairs and partial-frame depth say nothing about the link: a partial frame is one still arriving, or one the host could not finish. 0.3.1 read them as network loss and cut a slow PC's bitrate two rungs at once (loss_pct=42.8 in one PC's log, 47.5 in the slow-tunnel test). Over USB the controller now ignores those figures. It steps down one rung only when the tunnel's queue keeps dropping frames for two reports in a row, or jitter rises, and a report with drops does not count as clean. The transport passes the tunnel's drop count for USB sessions.
Keeping queued frames and backing off keyframe requests suits a slow tunnel, but not one that stands still: after the e2e test's 500 ms read stall, the queued video and the last requested keyframe were stale, and recovery took 113 ms instead of 9-12 ms. A write that waits 100 ms or more means the tunnel stood still; a datagram drains in milliseconds even through a slow usbmuxd. The rest of the frame in flight and everything queued is then dropped, and a keyframe requested at once with the backoff starting over. Recovery is back to 13-15 ms.
The reassembler's counters missed a frame lost whole (no fragment ever arrived), and without repair they counted a partial frame only after a replacement keyframe had restored the picture, so the next frame was held by mistake. The wire frame_seq can't replace them: the host numbers frames as captured and skips frames its encoder had no time for. The decoder now reads the codec's own count of coded pictures: H.264 frame_num, exact; HEVC POC, whose step is learned between frames with contiguous frame_seq (constant on NVENC, AMF and x265). A keyframe ends the hold; a restarted stream resets the count.
A failed session kept its format cached, so every later keyframe's identical parameter sets matched it and were ignored: the picture stayed frozen until reconnect. The format is now forgotten, so the next keyframe tries again (one attempt per keyframe). Meanwhile the last picture is held with keyframe requests and the Restoring the picture note, and the outage is logged once.
A start that failed latched the player off, and the reset or configuration change that could have cured it never cleared the latch. Each such notification now allows one more start; a start that fails again stays off until the next one.
…x/stream-smoothness # Conflicts: # RELEASE_NOTES.md
…x/ipad-robustness # Conflicts: # RELEASE_NOTES.md
Over USB the controller stepped down only after drops in two reports in a row, and every clean report reset that. A tunnel that dropped frames in every other report never stepped down: 1,200 frames dropped over 120 s left the stream at 15 Mbps. It now steps down one rung when the tunnel dropped frames in at least two of the last three reports. A single burst still does not, and the clean window before stepping up is unchanged.
A keyframe replaced up to six queued frames and reported none as dropped. Under a steady overload, periodic keyframes cleared the queue before it ever overflowed, so the tunnel never reported a drop: in the simulation a quarter of the frames never reached the iPad while ABR raised the bitrate to 20 Mbps. Replaced frames now count as dropped; a tunnel that keeps up has nothing queued when a keyframe comes. The same run now steps down to 4 Mbps, and every rate the tunnel carries still climbs to the ceiling. The keyframe backoff also survived a pipeline restart on the same tunnel: at its longest wait, the new stream's first request was deferred by almost two seconds. It now starts over whenever the session or stream epoch changes. The USB simulation feeds ABR the tunnel's drops every 500 ms, as the transport does.
A complete keyframe needs no earlier frame, yet it waited in order behind an older frame held for its repair: received at 320 ms, shown at 345 ms. In repair mode a complete keyframe now retires everything older, held, partial or waiting behind a gap, without asking for another keyframe, and is delivered at once. The Rust mirror and the shared traces follow. The WiFi simulation now decides each datagram's fate from which datagram it is, so changing the receiver no longer reshuffles every later loss and runs compare like for like. It also counts delta frames shown with a missing reference. A frame retired by a newer keyframe is not missed, so the still-desktop test checks drops and keyframe requests instead.
The wire's frame_seq was the capture number. The encoder takes only the latest captured frame, and 14% of captured frames were never encoded while streaming (up to 40% on one PC), so the numbers skipped where proto documents a contiguous per-epoch counter. On WiFi the iPad held the frame after each skip for half a frame period, waiting for one that never existed, and could not tell a skip from a whole frame lost in transit. The transport now numbers the access units it sends 1, 2, 3... per epoch, at the send point, and the repair ring and logs use the same number; the capture number stays in the debug log. A frame dropped as oversized still uses its number, so the gap marks a real loss. 0.3.1 iPads only stop waiting: they already accept any start and every number in order. In the WiFi simulation, 0.3.1's numbering against contiguous numbering: p90 latency 12.0 to 5.4 ms at 60 fps and 7.6 to 4.8 ms at 120 fps.
A malformed SPS (a scaling-list delta of Int32.max) overflowed and trapped. Every Exp-Golomb value in the SPS, PPS and slice-header parsers is now checked against the H.264 and HEVC ranges; anything else can't be parsed, which never holds the picture. A new stream epoch is a new encoder, so the frame's epoch now reaches the decoder: it restarts the count, forgets the HEVC POC step and waits for the new stream's keyframe. Changed parameter sets restart the count too. When two frames with contiguous frame_seq disagree with the learned POC step, POC is marked unreliable for the stream and a frame lost whole is no longer read from it (logged once), so a nonconstant encoder can't cause false freezes.
Recovery waited for frames, so a silent stream got no keyframe retries. While the last clean picture is held, a timer now asks again every 500 ms (500-600 ms apart, past the PC's 500 ms limit on forced keyframes). After 2 s with no decoded picture it rebuilds the decoder session and asks the PC to restart the stream (KeyframeReason.resume); after 5 s it reconnects through the existing lost-connection path. Only a picture decoded after the keyframe that ends the hold counts as progress, so a rejected keyframe or a frame already in flight doesn't reset the clock.
On a starved CI runner XCTest's idle wait and hit test outlasted the HUD's 5 s timer between revealing the HUD and tapping Keyboard, so the button was gone mid-tap (#75: host capture fell to 12 fps and encode to 3 fps at the time). The test now taps both Keyboard presses through tapHUDControl, which reveals and taps at the control's position at once and tries again if the HUD hid first. Its three-finger reveal keeps the fingers together mid-screen, now shared as HUDReveal.threeFingersTogether.
With frames numbered contiguously per epoch, a forward gap in frame_seq is a frame lost in transit, for either codec, including a keyframe lost after the encoder reopens within the same epoch. The iPad can rely on that only if it knows the host numbers this way. HELLO_ACK's host_caps gains HOSTCAP_CONTIGUOUS_FRAME_SEQ (1 << 5, mirrored as HelloAck.hostCapContiguousFrameSeq). The transport that numbers frames sets it; other hosts leave it clear. A new golden vector carries it: a new iPad reads it, and every bit an older iPad tests reads as before.
A KeyframeRequest with reason Resume, which the iPad sends after 2 s of frozen picture, forced a rate-limited IDR like any other reason, so a broken encoder session kept running until the iPad reconnected at 5 s. It now also reopens the encoder through the existing reopen path (as for a bitrate change: same epoch, then a forced IDR), at most once per 2 s. Other reasons are unchanged.
… on a lower seq An encoder reopen keeps the epoch and the numbering, so a lost keyframe of the new encoder let its first P-frame through against the old references whenever the codec's count happened to follow on. A PC that numbers sent frames contiguously per epoch (HELLO_ACK host cap, assumed 1 << 5 until fix/stream-smoothness defines it) now makes any forward gap a lost frame, whatever the codec. A lower frame_seq within an epoch is a restart: it clears the count, the learned HEVC POC step and the unreliable flag, and waits for a keyframe. 0.3.1 PCs keep the codec-continuity checks.
The 2 s restart request went out once, and the control channel drops any keyframe request within 500 ms of the last, so it was usually coalesced into the previous retry and never sent. From 2 s on each retry asks for the restart instead, in the same slots (the PC reopens at most once per 2 s; the decoder is still rebuilt once). Retries move from 500 to 600 ms apart, so jitter can't land one inside the PC's 500 ms window.
Every frame a keyframe replaced counted as congestion. A frame can also be queued because the sender ran twice before the link's writer was scheduled, with the tunnel idle: with the frame before each keyframe batched that way, a tunnel with room to spare counted 40 drops in 20 s and the bitrate fell from 15 to the 4 Mbps floor in 13 s. Replaced frames still count as dropped, since their pictures are lost. ABR now reads a separate congestion count that leaves out frames replaced while the writer was idle. A busy writer means the frame waited behind real writing: the tunnel is behind. Waiting time was not needed, and would have required a frame rate the link does not know. The healthy schedule now climbs to the 20 Mbps ceiling; the overload at 450 KB/s still steps down to 4 Mbps.
A held frame arriving just before the timer's tick used each retry slot as a plain keyframe request, and only the timer could rebuild the decoder or ask the PC to restart, so dependent frames every 600 ms starved both and the iPad reconnected at 5 s. Every keyframe request made during a hold now takes its reason from the hold's age (a restart request from 2 s), from the timer, a held frame or a decode error alike, and the one-time decoder rebuild happens when the hold first reaches 2 s whether or not a request may go out yet. Tests can drive the recovery timer by hand.
Telemetry quit tests hold the upload thread at explicit points (a stalled write, the final write, an hour-long receive) and release it after the quit returns, instead of timing the quit against SHUTDOWN_WAIT plus a margin. The held test server answers only once the quit has begun. The display lock wait and record retries read an injected clock; the tests pause it. The reporter's dead-reporter wait no longer times the process spawn. The updater helper's stand-in host closes when the helper says it is ready instead of after 0.5 s. The USB e2e stalls its reader for 60 encoded frames instead of 500 ms and checks the recovery in frames: the recovery keyframe comes from the end of the stall, fewer than half the stall's frames arrive before it, and nothing older arrives after it.
It writes 513 batches back to back. On mainpc's disk that took over a minute, and Windows' unfair lock let it hold the outbox write lock that long, so other tests' writes missed their 30 s waits. It now runs alone among the outbox tests.
The first frame after DXGI_ERROR_ACCESS_LOST reset the retry ladder, so a desktop taken again right after each return (a UAC prompt, the lock screen, a full-screen game) was re-created at full speed in a loop. The ladder's level now holds until the duplication has worked for 5 s. The first retry after a healthy stretch is still immediate, and each return still ends its outage for the 10 s give-up window.
When the preferred hardware encoder failed, the host tried NVENC, AMF and Quick Sync in turn whatever the PC had, so PCs without an AMD card logged AMF DLL errors and an NVIDIA + Radeon PC probed Quick Sync. The fallback now reads the present adapters' vendors when it runs (so an eGPU or driver added since is tried) and skips encoders whose vendor is absent. A hybrid laptop lists both cards. With no adapter list it still tries all three. Ported from #78 (2e1522b, 8a0e22c) onto the encoder-open limits.
… try An encoder that fails to open on a PC without its card or driver made FFmpeg log ERROR lines, though the next encoder took over and the stream worked. While an encoder with another one after it opens (every first open but libx264's), FFmpeg's ERROR lines from that thread are logged as WARN, keeping ffmpeg_level. The flag is set on the thread that makes the call, the encoder lane's when the open is time-limited. If every encoder fails, one ERROR names each encoder and its reason. Reopens and libx264 keep their levels.
PR #125 gave the release job AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_SUBSCRIPTION_ID without naming them in its artifact scan, so test_workflow_artifacts failed. They stay empty until Artifact Signing is set up, and a required name that is empty fails the scan outside pull requests. artifact_secrets.py gains --optional: those names are scanned when set and never reported missing.
FFmpeg 7.1.1's NVENC is built with nv-codec-headers 13.0.19 (Video Codec SDK 13.0), which needs NVIDIA driver 570.0 or newer on Windows, and FFmpeg prints the minimum as (unknown). PCs on older drivers ended on software encoding with only the generic driver advice. When NVENC fails with ENOSYS and every NVIDIA adapter's driver (DXGI's Windows version, converted to NVIDIA's numbering: 32.0.15.6607 is 566.07) is older than 570, the host logs it once and the CPU fallback notice says to update the NVIDIA driver to 570 or newer. ENOSYS alone also means an unsupported parameter, so a newer driver keeps the generic notice. Ported from #78 (298380a, 4dbf147) without the link.
"Virtual display could not be enabled (installer task missing?)" guessed the same cause for every failed Enable. The line now carries reason= from what the Enable reported: task_missing, task_failed (with device= for the state the task found, e.g. problem_43), timeout (it, or an earlier one still stuck, did not finish), or other. The Stream page's notice already says what to do for each.
Per-frame seq and fragment made every send-failure line unique, so the repeat collapser could not fold them. The line now carries link=usb or link=wifi and the error only.
The collapser evicted the oldest-started run, which could be a line that was still repeating. That run was then summarized and restarted again and again. Evict the run seen least recently instead.
A missed service poll during a USB session raised "Your iPad is plugged in, but Apple Devices is not running" even though the session proves the cable works. The notice now needs no active USB session.
If the worker thread failed to spawn, the launch re-check of a saved release never answered: the offer stayed hidden and Update now did nothing. Clear the re-check in that case, as an offline check does, and log the failure once.
Reviewing #123: a stuck description read on a network share made every later check Unknown, even for Windows (pid 4) or another copy of EternalMonitor. The image path needs no file I/O, so classify from it on the checking thread and read only the file's description on the lookup thread. A slow read now leaves the program named by its file instead of Unknown, and a description once read is kept, so the warning no longer flips between "unknown" and "another program" around the deadline. The lookup thread logs through the caller's subscriber, so the log tests see its lines.
When a frame held for a second is retired, only the incomplete frames behind it up to the first pending keyframe go with it. The keyframe keeps its own deadline and is shown when its last fragment arrives. With a complete frame behind it, a held frame now goes one repair window after the first newer frame began, as before this branch, rather than after the complete frame began. A frame held with newer frames already behind it goes as soon as a complete frame arrives.
A name given to artifact_secrets.py both in --secrets and --optional was treated as optional, so an empty required secret passed outside pull requests. Overlap is now a usage error. test_workflow_artifacts looked for each optional binding anywhere in the workflow header or job, so a secret bound only in another step's env passed without reaching the scanner. It now accepts only the workflow, job and scan step env blocks, and a fixture workflow covers each place.
… them The fallback skipped a hardware encoder whose vendor DXGI did not list. Windows does not list every card that can encode: an NVIDIA card in TCC mode beside an Intel or AMD display adapter lost NVENC to software when Quick Sync or AMF failed. And a partial adapter list (one adapter described, the next failing) suppressed a working encoder. Now the listed vendors' encoders are tried first, then every other hardware encoder, then libx264, so the worst case is the old order and a PC whose listed card encodes never probes the others. Any error while listing the adapters, including one adapter that can't be described, keeps the old order. An unknown vendor moves no encoder.
Workflow and job env no longer count: a step env entry overrides them, so an empty value there silently disabled the scan. Comments and block scalar contents are not bindings.
Reviewing #131: the description read for the port's holder was kept by image path alone, so a different program that later ran from the same path was shown under the first program's description indefinitely. Keep it by the holder's pid, start time and path instead. A new process, or a reused pid that started later, is named by its file until its own description is read on the lookup thread. The start time comes from the handle the path is already read through, so the checking thread still does no file I/O.
Scan the Azure signing secrets in the release artifact check as optional
Early MetricKit payloads can hold the binary's load address in the text-segment offset, so the offset alone can't place the frame. Each frame is now binary+0x<offset>@0x<address>, and the final scrub keeps the address, which the line formats from a number. The same hex in a binary name is still scrubbed.
64-bit codes such as EXC_GUARD and JETSAM reasons have 16 hex digits with a letter, so the final scrub turned them into code:<id> and the reason the app was killed was lost. The scrub now keeps the code value in the termination field, which is already reduced to namespace and code, and nowhere else.
A .failed connection logged "NWConnection failed" and then fail() logged the same failure again, through the connection that took the link or as the link's own line. A send failure did the same with "NWConnection send". The receive path lost its extra line in #126; the other two now log only through fail(). FramedLink takes its support-log writer as an init parameter so a test can count the lines.
Keep waiting for repairs while no complete frame is blocked
Fix host log noise, the USB notice, the update re-check and port-holder naming
Windows host field fixes: no-display wait, access-lost backoff, encoder probing, NVIDIA driver notice
Harden six tests that failed on loaded machines
Polish iPad crash lines and log a failed USB connection once
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EternalMonitor 0.3.4
This is the 0.3.4 release branch. Every change below went through its own PR with review and full CI before it landed on
release/0.3.4-candidate. Merging this PR brings 0.3.4 tomain. Taggingv0.3.4afterwards runs the release and TestFlight workflows.Customer-reported problems fixed
Reliability
latest.jsonthat can never block or degrade a working PC or iPad (Let the update manifest stage update prompts and set safe per-PC defaults #112).Still landing before this is marked ready
Each will land on the candidate the same way, and this description will be updated.
Release gate (run on the final head of this PR)