Repository navigation
Make CI stable and faster #2913
Copy link
Copy link
Open
Description
Why
CI on main is too flaky and too slow to trust:
- Only 29–30% of PR runs (33–40% on
main) are green on the first attempt. Most red runs pass on an unchanged rerun. - A green PR run takes 41 min at p50 and 80 min at p90.
- Unity license retries cost about 9 min per build job, 7,150 min over 30 days. A PR run starts 14 build jobs against 7–8 floating licenses shared with the team.
- Caches sit at ~11 GB against the 10 GB quota, so LRU eviction keeps runs cold.
A red run that goes green on a rerun teaches everyone to ignore red, and real regressions hide in the noise.
What
A stack of small PRs, each fixing one cause:
- Fail fast: stop the run when the SDK build fails, and cap job run time with
timeout-minutes. - Retries: retry known infra failures once, by a specific log signature that is printed when it fires. Examples: LicensingClient IPC timeout, an editor killed at startup, a Unity shutdown crash, Gradle 429, a segfaulting build tool, sentry-cli 5xx, an ingest 5xx on the crash report. Product and test bugs get fixed, not retried.
- Root-caused flakes:
- Native symbolication: sentry-cli returns before sentry.io has processed the upload, so events arrive before their symbols. The upload assertion now waits until sentry.io lists the uploaded debug files.
- iOS first launch: a launch less than ~2 min after the simulator boots can stall for minutes. The simulator now boots while the app compiles.
- Unity 6000.6 in Docker: Unity kills the PID in a
Library/burst.pidcopied from the Create job, which turned out to be the editor's ownxvfb-run. The stale pid files are now deleted before each build. - An Android test app that never initialises is relaunched once.
- App Hang events are ignored in the iOS Crash Capture lookup.
- The playmode integration tests get 5 s to receive an event.
ConfigureOptionsrun a second time cleared the options configuration. This is an SDK bug users can hit too, with a CHANGELOG entry and EditMode tests.
- Speed:
- Two fewer Unity launches per integration test build, 52 → 24 launches per PR run.
- Integration test and native SDK caches keyed on their inputs.
- The leftover .NET 9 install and the workload restore are gone.
- The iOS size baseline and the simulator terminate stall are off the Run iOS critical path.
- The license retry is jittered and capped at 30 min.
Results so far
Ten back-to-back reruns of the unchanged stack tip:
Before (main 626d790) |
Stack tip | |
|---|---|---|
| First-attempt green, PR | 29–30% | 8/10 |
| PR wall time p50/p90 | 41/80 min | 32.0/35.2 min |
| Unity launches per PR run | 52 | 24 |
| License-retry sleep | ~9 min per build job | 217–1,029 s per run in total |
- One of the two red runs was the
ConfigureOptionsbug, fixed since in ci: fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes #2911; its final head is green on the first attempt in 27.3 min. - The other was a one-off Unity-internal Bee IPC error (
Request cancelled while reading the binlog). - A full-matrix run (5 Unity versions) of the stack is green.
Merge order
The PRs are stacked; each one is based on the one below it. Merge from the bottom up. sentry-unity squash-merges, so after each merge the next PR is rebased onto main and its CI reruns.
app-runner goes first. The stack pins its PRs' exact commits, and merging them with merge commits keeps those pins valid:
- Skip the iOS simulator terminate right after boot and bound the others app-runner#65 (pinned by ci: take the iOS size baseline and the terminate stall off the Run iOS path #2910)
- fix(adb): Force-stop the app before each launch and after a run timeout app-runner#64 (pinned by test: relaunch an Android test app that never initialised once #2901)
- feat: Add Filter parameter to Get-SentryTestEvent app-runner#63 (pinned by test: ignore App Hang events in the iOS Crash Capture lookup #2899)
main(app-runner)
sentry-unity, from the tip of the stack down to main:
- ci: fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes #2911 Fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes, plus the
ConfigureOptionsfix - ci: take the iOS size baseline and the terminate stall off the Run iOS path #2910 Take the iOS size baseline and the terminate stall off the Run iOS path (needs app-runner#65)
- ci: key the native SDK caches on their own inputs and skip the workload restore #2909 Key the native SDK caches on their own inputs and skip the workload restore
- ci: recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts #2908 Recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts
- test: give the playmode integration tests 5 s to receive an event #2906 Give the playmode integration tests 5 s to receive an event
- ci: cap job run time with timeout-minutes #2905 Cap job run time with
timeout-minutes - ci: retry known infra failures once, by signature #2904 Retry known infra failures once, by signature
- test: launch Unity two fewer times per integration test build #2903 Launch Unity two fewer times per integration test build
- ci: key the integration test caches on their inputs #2902 Key the integration test caches on their inputs
- test: relaunch an Android test app that never initialised once #2901 Relaunch an Android test app that never initialised once (needs app-runner#64)
- ci: jitter the Unity license retry and cap it at 30 minutes #2900 Jitter the Unity license retry and cap it at 30 minutes
- test: ignore App Hang events in the iOS Crash Capture lookup #2899 Ignore App Hang events in the iOS Crash Capture lookup (needs app-runner#63)
- ci: fail fast when the SDK build fails #2898 Fail fast when the SDK build fails
- ci: drop the leftover .NET 9 install from the Linux native SDK build #2907 Drop the leftover .NET 9 install from the Linux native SDK build
main
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Projects
- StatusShow more project fieldsNo status
Activity
linear-code commentedon Oct 27, 2026
UNITY-241