Skip to content

Harden the lost P-frame and Copy diagnostics tests - #138

Merged
whoisaldo merged 2 commits into
release/0.3.4-candidatefrom
test/ios-ui-flakes
Oct 30, 2026
Merged

whoisaldo merged 2 commits into
release/0.3.4-candidatefrom
test/ios-ui-flakes

Conversation

@whoisaldo

Copy link
Copy Markdown
Member

Two iPad tests failed on CI in runs that didn't change the code they cover. Both failures came from test timing. Neither is an app defect.

StreamRecoveryTests.testALostPFrameHoldsThePictureAndAsksUntilTheNextIDR

Failure: run 40969597183 (candidate head fd9b791, scripts and workflows only), caps 0 iteration:

  • line 153: the first request was decodeError, not gapLoss. The 1.9 s window held 4 × decodeError and no gapLoss.
  • line 166: a resume arrived after the IDR.

Root cause, from the CI log in the evidence artifact:

  1. The test sent frame 4 right behind the lost frame 3. Two parts of the app then see the same loss within a millisecond, on two queues. The reassembler delivers frame 4 and drops frame 3, which asks with gapLoss. The decoder rejects frame 4 on its frame_num or seq gap and asks with decodeError. ControlChannel keeps only the first request in any 500 ms. On a loaded runner the decoder's queue won, so gapLoss was never sent. The PC handles both reasons the same way: handle_keyframe_request only logs the reason, then forces an IDR. Either way one request goes out at once, so this is test over-specification, not a defect.
  2. The IDR was sent after a detached task had sent 17 frames with 100 ms sleeps. On CI that loop took about 2.9 s instead of 1.7 s. The hold started at 23:18:20.735. Decoder retries came on time: 21.336, 21.941, 22.559. At 22.759 the log has "Picture held 2024 ms; rebuilding the decoder", and the IDR's format description shows up only at 23.694. A hold past 2 s correctly escalates to Resume. That Resume sat in the fake PC's socket and was read by the "after the IDR" window.

Fix (ios/EternalMonitorTests/StreamRecoveryTests.swift:154-176, FakeUDPHost.swift:129-143): each step now waits for the iPad's own requests instead of a fixed time.

  • After the lost fragment, nothing else is sent until the reassembler's request arrives, and it must be gapLoss. The test then waits 600 ms, past the 500 ms spacing.
  • The following frames start. The test collects the decoder's first three requests (at once, then 600–700 ms apart), and all three must be decodeError.
  • The IDR is sent straight away, before the next request is due. The frame sender is cancelled and no longer delays it. A frame it still sends is older than the IDR and is ignored.
  • pictureRestoring is waited for instead of read once. It is refreshed at 4 Hz.

Every other assertion is unchanged: nothing after the loss is shown, the hold exists, retry spacing, no requests after the IDR, the hold ends, and no -12909.

ConnectScreenTests.testDiagnosticsCanBeCopiedAndSharedWithoutAConnection

Failure: run 40969661205 (#127's merge head 5b11562; #127 doesn't touch this test or DiagnosticsView). Line 244: "Diagnostics copied" never appeared.

Root cause, from the evidence artifact's synthesized events and UI tree:

  • XCTest tapped Copy at (417, 839.5), the centre of the Copy row when it computed the point.
  • At failure, Copy was at y 847.5–891.5, 30 pt lower, and Export at 803.5–847.5. The upload status cell above them was 74 pt tall, showing the three-line "Automatic upload is not configured; installation_id=…".
  • That line starts as the one-line "Checking automatic upload" (a 44 pt row). It is replaced when DiagnosticsUpload.status() returns from its queue. On the loaded runner that happened between XCTest finding Copy and its tap, so the tap missed.

Fix (ios/EternalMonitorUITests/ConnectScreenTests.swift:243-251): wait until "Checking automatic upload" is gone, then let swipeUp(toReveal:) confirm the Copy row is hittable and has stopped moving, then tap once. The copy itself is checked exactly as before.

Results (Xcode 16.4, iOS 18.6, iPad Pro 11-inch (M4) simulator, ten yes loops running)

Test Runs Result Load (1 min) before → after
StreamRecovery lost P-frame (fixed) 20 20/20 passed 153 → 227
ConnectScreen diagnostics (fixed) 20 20/20 passed 249 → 391
StreamRecoveryTests class (fixed) 1 5/5 passed 8
ConnectScreenTests class (fixed) 1 10/10 passed 7 → 32

Reproducing the CI failures with added delays (uncommitted)

The delays:

  • 20 ms usleep between the reassembler delivering a frame and dropping the older partial one;
  • the test's frame sender sleeping 170 ms instead of 100 ms, which is about the 2.9 s CI's starved sender took;
  • the upload status arriving 0.5–3 s late, at random.
Test Runs Result
StreamRecovery, old test + delays 5 5/5 failed at lines 153 (decodeError first) and 166 (resume after the IDR), as on CI
StreamRecovery, fixed test + same delays 5 5/5 passed
ConnectScreen, old test + random status delay 10 1 failed at line 244. Same tap point (417, 839.5) and same rows (Copy at 847.5, 74 pt status line) as CI
ConnectScreen, fixed test + same delay 20 20/20 passed

The fixed tests still fail when the feature is broken (uncommitted)

Break Result
Reassembler doesn't ask when it drops a frame (FrameAssembler.requestKeyframe) StreamRecovery fails at line 157 ([] vs [gapLoss]), both caps
A decoded IDR doesn't end the hold (VideoDecoder keeps hold) Fails at line 181 ([decodeError, resume] after the IDR) and 182 (hold still set), both caps
Decoder admits frames after a lost reference (admitted returns true) Fails at 164 (19 frames shown, not 2), 165, 167 and the Restoring wait, both caps
Copy diagnostics doesn't confirm (copied = true removed) ConnectScreen fails at line 252

Failure history

All attempts, reruns included, of the last 400 CI runs since each test existed (10-18 and 10-29). Failed iOS jobs that list these tests in "Failing tests":

Run/attempt Branch Date Test Line (statement)
40969661205/1 fix/lossy-test-flake 10-29 Diagnostics 244, "Diagnostics copied" (this cause)
40969597183/1 release/0.3.4-candidate 10-29 Lost P-frame 153 and 166 (this cause)
40822985750/1 fix/no-gl-window-renderer 10-29 Diagnostics 187, share-sheet header, old 5 s wait
40757753531/1 fix/stable-timing-tests 10-28 Diagnostics 169, failed to terminate the app at launch
40643474436/1 fix/time-box-driver-calls 10-28 Diagnostics 187, share-sheet header
40642527476/1 fix/stable-timing-tests 10-28 Diagnostics 169, failed to terminate
40641985142/1 ci/release-pipeline-hardening 10-28 Diagnostics 187, share-sheet header
40525920042/2 fix/testflight-signing 10-27 Diagnostics 186, share-sheet wait
40438379284/1 fix/hybrid-second-screen 10-27 Diagnostics 187, share-sheet header
39925383477/1, /2 site/redesign 10-23 Diagnostics 143 and 132 of the PR merge ref, share-sheet section
39733394480/1 release/0.3.2-candidate 10-22 Diagnostics artifact expired
39165557516/1 main 10-18 Diagnostics artifact expired
39162826400/1 fleet/diagnostics-private-merge 10-18 Diagnostics artifact expired

The earlier Diagnostics failures were share-sheet and app-termination waits, which later commits already lengthened or fixed. The lost Copy tap appears only in run 40969661205. The lost P-frame test failed once, in 40969597183.

Known app issue, not changed here

The Diagnostics screen's Export and Copy rows jump about 30 pt down when the upload status line loads: one line ("Checking automatic upload") becomes three. A tap made at that moment can miss Copy or land on Export. It only shows when DiagnosticsUpload.status() is slow to return, as on the loaded runner. This PR changes only the test.

Frames sent right behind the lost one let the reassembler and the decoder
see the loss on two queues within a millisecond, and the 500 ms request
spacing keeps whichever asks first, so the first reason was decodeError
on a loaded runner. The IDR followed a fixed run of 17 frames, which a
starved sender stretched past the 2 s Resume.

The test now waits for the reassembler's request before sending more
frames, and sends the IDR right after the decoder's third request instead
of after the frames.
The upload status line above the buttons grows from one line to three
when it loads. On a loaded runner it loaded between XCTest finding Copy
and its tap, which then landed on the row's old position and missed.
@whoisaldo
whoisaldo requested a review from Alitleis123 October 30, 2026 02:24
@whoisaldo whoisaldo added the 0.3.4 Part of the 0.3.4 release label Oct 30, 2026
@whoisaldo
whoisaldo marked this pull request as ready for review October 30, 2026 02:53
@whoisaldo
whoisaldo merged commit e5896c0 into release/0.3.4-candidate Oct 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0.3.4 Part of the 0.3.4 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants