Skip to content

ci: retry known infra failures once, by signature - #2904

Draft
bitsandfoxes wants to merge 4 commits into
ci/fewer-editor-launchesfrom
ci/infra-retries
Draft

bitsandfoxes wants to merge 4 commits into
ci/fewer-editor-launchesfrom
ci/infra-retries

Conversation

@bitsandfoxes

@bitsandfoxes bitsandfoxes commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

CI now retries a handful of infrastructure failures once, each only when its log signature shows up, instead of failing the job and waiting for someone to rerun it.

These are the failures in the last 30 days (and in this PR stack's own runs) that have nothing to do with the change under test and pass on a rerun:

  • Unity gets killed, but xvfb-run turns the 137 into exit 5 when its cleanup fails, so the existing kill retry never fires.
  • Unity crashes on shutdown after a successful build (Batchmode quit successfully invoked, then Caught fatal signal after Cleanup mono, exit 133).
  • Gradle gets rate limited by Maven Central (Received status code 429 from server).
  • il2cpp or another Bee tool segfaults (| 139 (Segmentation fault)).
  • pwsh find-unity.ps1 aborts with a core dump in "Build Sentry.Unity Solution". MSBuild ran it in ~15 projects at once, and 12 of them died.
  • The Unity editor install, the emulator download (Error on ZipFile), and a Redroid container that never boots.

How it works now:

  • RunUnity picks the retry by signature (GetUnityRetrySignature) instead of by exit code 137 alone, and retries each signature at most once per call. Exit 5 only counts as killed behind docker/xvfb-run and when the log has none of Unity's own exit lines. shutdown-crash is retried rather than treated as success, because the editor's shutdown work (saving the Library we cache afterwards) didn't finish. Test runs (-runTests) still only retry killed, so a crash during tests stays red. License misses get their own counter, so the "It took N tries" notice keeps counting license tries only.
  • FindUnity runs find-unity.ps1 through the MSBuild task (build/find-unity.proj), so MSBuild runs it once per build and hands every project the cached result. It sets the same properties as before.
  • setup-unity.ps1 retries the CLI download (-MaximumRetryCount 3) and unity install (3 tries, 30 s / 90 s), then still checks editors --installed.
  • test-run-android.yml installs the emulator and system image with sdkmanager before the emulator action, and retries up to 3 times only on Error on ZipFile. The action's own install is then a no-op. If Redroid hasn't booted after 150 s (it usually takes 10–15 s), the container is removed and started once more. The emulator target/channel and the Redroid image/args are defined once in the job env.
  • Every retry logs ::warning::… (signature: <name>), so (signature: counts them all.

Left out: a retry for an emulator that doesn't boot, since there's no such failure in the data and it'd need a duplicated emulator step. The 6000.6 editors that get Killed ~3 s after start (the container has no memory limit) also aren't covered. They're already caught by the 137 retry.

Stacked on #2903.

CI evidence: run 40207302078, with a throwaway commit (since dropped) that replaced the first attempt of one launch per job with each failure signature, or failed the first install, sdkmanager download or Redroid start. All 56 jobs are green, and every signature fired once and recovered: killed (exit 5) in the SDK build, both Create jobs, and the Linux, iOS and WebGL builds; shutdown-crash in the Linux, iOS and WebGL builds; gradle-429 in both Android builds; tool-segfault and editor-install-failed in the Windows, x86, Mono and macOS builds; sdkmanager-corrupt-zip in Run Android 2021.3 and redroid-boot-timeout in Run Android 6000.6 (77 passed each). "Build Sentry.Unity Solution" ran find-unity.ps1 once instead of ~15 times.

Rebased onto #2907: run 40283224833 is green on the first attempt in 40.6 min.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant