Repository navigation
ci: recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts - #2908
Draft
bitsandfoxes wants to merge 3 commits into
Draft
bitsandfoxes wants to merge 3 commits into
bitsandfoxes wants to merge 3 commits into
Conversation
bitsandfoxes
force-pushed
the
ci/residual-flakes
branch
from
October 25, 2026 20:55
05bf824 to
7a19a83
Compare
bitsandfoxes
force-pushed
the
test/playmode-event-timeout
branch
from
October 26, 2026 08:15
96055df to
0a63f48
Compare
bitsandfoxes
force-pushed
the
ci/residual-flakes
branch
from
October 26, 2026 08:59
7a19a83 to
0ebdd30
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI now recovers from three more flakes that were still failing runs: a Unity LicensingClient IPC timeout on Windows, sentry-cli giving up during a sentry.io 5xx burst, and a test event left in the offline cache after the app quit.
In a re-measure of the last 21 days (75 first attempts, excluding this stack), these accounted for 15 of the 28 failed jobs that nothing below this PR covers:
IPC channel to LicensingClient doesn't exist; aborting. Unity had just started the LicensingClient (it logs the PId), then gave up waiting for its channel after 5 s. The next launch finds the client running.retry number 5, max retries: 5while sentry.io returned 503/504, four of them in one main run. That's sentry-cli's default, about 40 s of backoff.Get-SentryTestEventalready keeps polling through both and only reports the last error. So these weren't a poll problem.How it works now:
GetUnityRetrySignaturegetslicensing-ipc: exit 199 plus that log line, one immediate retry. It doesn't go through the license loop, which sleeps 20–90 s and is meant for when there's no free seat. It also applies to test runs, since a license checkout fails before any project code runs.CliConfigurationsetsSENTRY_HTTP_MAX_RETRIES=20(unless it's already set) for the uploads the Editor spawns: desktop, and Android through Gradle, including in Docker. The iOS compile step sets it for xcodebuild, whose build phase runs the upload. That gives roughly 2–4 minutes instead of 40 s. Users' SDK defaults don't change.assert-symbol-upload.ps1warns with(signature: sentry-cli-http-retry)when sentry-cli had to retry.(signature: shutdown-flush-timeout)and relaunches the app withcrash-send, the same launch the crash test already uses to send its cached crash. The event the test then finds is the one the original launch serialized, with its release, tags, user and breadcrumbs, so the assertions don't change. The relaunch happens once per job. A second flush timeout fails the context, so an SDK change that times out every flush stays red. It also doesn't count against the Android hung-launch relaunch.Left out:
ECONNRESET.Stacked on #2906.
CI evidence: run 40228677752 is green 56/56. It ran with a throwaway commit (since dropped) that faked each case once and carried #2907.
licensing-ipcfired in every build job, the SDK build included, and recovered. sentry-cli loggedmax retries: 20in every desktop and Android build (Linux inside Docker) and in all 4 Compile iOS jobs (xcodebuild), and the injected 503 was annotatedsentry-cli-http-retry.shutdown-flush-timeoutrelaunched message-capture in the Windows crashpad/Mono, macOS cocoa and Linux breakpad runs, and their tests passed (e.g. 58/58, 77/77).Rebased onto #2907: run 40296170756 is green on the first attempt in 49.2 min.
🤖 Generated with Claude Code