Repository navigation
ci: retry known infra failures once, by signature - #2904
Draft
bitsandfoxes wants to merge 4 commits into
Draft
bitsandfoxes wants to merge 4 commits into
bitsandfoxes wants to merge 4 commits into
Conversation
bitsandfoxes
force-pushed
the
ci/infra-retries
branch
from
October 25, 2026 16:16
f262f2e to
262495c
Compare
bitsandfoxes
force-pushed
the
ci/infra-retries
branch
from
October 25, 2026 17:39
262495c to
56e67a0
Compare
bitsandfoxes
force-pushed
the
ci/fewer-editor-launches
branch
from
October 26, 2026 06:25
6ee554e to
dc3fd9a
Compare
bitsandfoxes
force-pushed
the
ci/infra-retries
branch
from
October 26, 2026 07:01
56e67a0 to
e01f78f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI now retries a handful of infrastructure failures once, each only when its log signature shows up, instead of failing the job and waiting for someone to rerun it.
These are the failures in the last 30 days (and in this PR stack's own runs) that have nothing to do with the change under test and pass on a rerun:
xvfb-runturns the 137 into exit 5 when its cleanup fails, so the existing kill retry never fires.Batchmode quit successfully invoked, thenCaught fatal signalafterCleanup mono, exit 133).Received status code 429 from server).| 139 (Segmentation fault)).pwsh find-unity.ps1aborts with a core dump in "Build Sentry.Unity Solution". MSBuild ran it in ~15 projects at once, and 12 of them died.Error on ZipFile), and a Redroid container that never boots.How it works now:
RunUnitypicks the retry by signature (GetUnityRetrySignature) instead of by exit code 137 alone, and retries each signature at most once per call. Exit 5 only counts askilledbehinddocker/xvfb-runand when the log has none of Unity's own exit lines.shutdown-crashis retried rather than treated as success, because the editor's shutdown work (saving the Library we cache afterwards) didn't finish. Test runs (-runTests) still only retrykilled, so a crash during tests stays red. License misses get their own counter, so the "It took N tries" notice keeps counting license tries only.FindUnityrunsfind-unity.ps1through theMSBuildtask (build/find-unity.proj), so MSBuild runs it once per build and hands every project the cached result. It sets the same properties as before.setup-unity.ps1retries the CLI download (-MaximumRetryCount 3) andunity install(3 tries, 30 s / 90 s), then still checkseditors --installed.test-run-android.ymlinstalls the emulator and system image withsdkmanagerbefore the emulator action, and retries up to 3 times only onError on ZipFile. The action's own install is then a no-op. If Redroid hasn't booted after 150 s (it usually takes 10–15 s), the container is removed and started once more. The emulator target/channel and the Redroid image/args are defined once in the job env.::warning::… (signature: <name>), so(signature:counts them all.Left out: a retry for an emulator that doesn't boot, since there's no such failure in the data and it'd need a duplicated emulator step. The 6000.6 editors that get
Killed~3 s after start (the container has no memory limit) also aren't covered. They're already caught by the 137 retry.Stacked on #2903.
CI evidence: run 40207302078, with a throwaway commit (since dropped) that replaced the first attempt of one launch per job with each failure signature, or failed the first install, sdkmanager download or Redroid start. All 56 jobs are green, and every signature fired once and recovered:
killed(exit 5) in the SDK build, both Create jobs, and the Linux, iOS and WebGL builds;shutdown-crashin the Linux, iOS and WebGL builds;gradle-429in both Android builds;tool-segfaultandeditor-install-failedin the Windows, x86, Mono and macOS builds;sdkmanager-corrupt-zipin Run Android 2021.3 andredroid-boot-timeoutin Run Android 6000.6 (77 passed each). "Build Sentry.Unity Solution" ranfind-unity.ps1once instead of ~15 times.Rebased onto #2907: run 40283224833 is green on the first attempt in 40.6 min.
🤖 Generated with Claude Code