Skip to content

ci: recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts - #2908

Draft
bitsandfoxes wants to merge 3 commits into
test/playmode-event-timeoutfrom
ci/residual-flakes
Draft

bitsandfoxes wants to merge 3 commits into
test/playmode-event-timeoutfrom
ci/residual-flakes

Conversation

@bitsandfoxes

@bitsandfoxes bitsandfoxes commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

CI now recovers from three more flakes that were still failing runs: a Unity LicensingClient IPC timeout on Windows, sentry-cli giving up during a sentry.io 5xx burst, and a test event left in the offline cache after the app quit.

In a re-measure of the last 21 days (75 first attempts, excluding this stack), these accounted for 15 of the 28 failed jobs that nothing below this PR covers:

  • Six Windows jobs exited 199 with IPC channel to LicensingClient doesn't exist; aborting. Unity had just started the LicensingClient (it logs the PId), then gave up waiting for its channel after 5 s. The next launch finds the client running.
  • Five symbol uploads ran out at retry number 5, max retries: 5 while sentry.io returned 503/504, four of them in one main run. That's sentry-cli's default, about 40 s of backoff.
  • Two iOS message captures timed out the shutdown flush ("Timeout when trying to flush queue."), so the event stayed in the offline cache. In one of them it was sent on the next launch, 9 s after the 300 s poll for it had given up. Two more iOS jobs failed the event poll with a 503 or 404. The earlier read was that app-runner's poll gave up on those, but Get-SentryTestEvent already keeps polling through both and only reports the last error. So these weren't a poll problem.

How it works now:

  • GetUnityRetrySignature gets licensing-ipc: exit 199 plus that log line, one immediate retry. It doesn't go through the license loop, which sleeps 20–90 s and is meant for when there's no free seat. It also applies to test runs, since a license checkout fails before any project code runs.
  • The integration test's CliConfiguration sets SENTRY_HTTP_MAX_RETRIES=20 (unless it's already set) for the uploads the Editor spawns: desktop, and Android through Gradle, including in Docker. The iOS compile step sets it for xcodebuild, whose build phase runs the upload. That gives roughly 2–4 minutes instead of 40 s. Users' SDK defaults don't change. assert-symbol-upload.ps1 warns with (signature: sentry-cli-http-retry) when sentry-cli had to retry.
  • When an action other than crash-capture logs the flush timeout, the test warns with (signature: shutdown-flush-timeout) and relaunches the app with crash-send, the same launch the crash test already uses to send its cached crash. The event the test then finds is the one the original launch serialized, with its release, tags, user and breadcrumbs, so the assertions don't change. The relaunch happens once per job. A second flush timeout fails the context, so an SDK change that times out every flush stays red. It also doesn't count against the Android hung-launch relaunch.

Left out:

  • "Resolves the throw frame to its source line" failed 5 times with the frame unresolved or resolved to the wrong line. Sentry doesn't reprocess stored events, and days later those events are still wrong, while another event from the same build resolved fine. That's a symbolication defect, not a timing issue, so it isn't retried. It needs its own investigation.
  • A busy artifact upload (2), lost runners (2), and one each of an emulator adb failure, an iOS hung launch, a Windows editor access violation and a UPM ECONNRESET.

Stacked on #2906.

CI evidence: run 40228677752 is green 56/56. It ran with a throwaway commit (since dropped) that faked each case once and carried #2907. licensing-ipc fired in every build job, the SDK build included, and recovered. sentry-cli logged max retries: 20 in every desktop and Android build (Linux inside Docker) and in all 4 Compile iOS jobs (xcodebuild), and the injected 503 was annotated sentry-cli-http-retry. shutdown-flush-timeout relaunched message-capture in the Windows crashpad/Mono, macOS cocoa and Linux breakpad runs, and their tests passed (e.g. 58/58, 77/77).

Rebased onto #2907: run 40296170756 is green on the first attempt in 49.2 min.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant