Skip to content

Make CI stable and faster #2913

Open
Open

Description

@bitsandfoxes

Why

CI on main is too flaky and too slow to trust:

  • Only 29–30% of PR runs (33–40% on main) are green on the first attempt. Most red runs pass on an unchanged rerun.
  • A green PR run takes 41 min at p50 and 80 min at p90.
  • Unity license retries cost about 9 min per build job, 7,150 min over 30 days. A PR run starts 14 build jobs against 7–8 floating licenses shared with the team.
  • Caches sit at ~11 GB against the 10 GB quota, so LRU eviction keeps runs cold.

A red run that goes green on a rerun teaches everyone to ignore red, and real regressions hide in the noise.

What

A stack of small PRs, each fixing one cause:

  • Fail fast: stop the run when the SDK build fails, and cap job run time with timeout-minutes.
  • Retries: retry known infra failures once, by a specific log signature that is printed when it fires. Examples: LicensingClient IPC timeout, an editor killed at startup, a Unity shutdown crash, Gradle 429, a segfaulting build tool, sentry-cli 5xx, an ingest 5xx on the crash report. Product and test bugs get fixed, not retried.
  • Root-caused flakes:
    • Native symbolication: sentry-cli returns before sentry.io has processed the upload, so events arrive before their symbols. The upload assertion now waits until sentry.io lists the uploaded debug files.
    • iOS first launch: a launch less than ~2 min after the simulator boots can stall for minutes. The simulator now boots while the app compiles.
    • Unity 6000.6 in Docker: Unity kills the PID in a Library/burst.pid copied from the Create job, which turned out to be the editor's own xvfb-run. The stale pid files are now deleted before each build.
    • An Android test app that never initialises is relaunched once.
    • App Hang events are ignored in the iOS Crash Capture lookup.
    • The playmode integration tests get 5 s to receive an event.
    • ConfigureOptions run a second time cleared the options configuration. This is an SDK bug users can hit too, with a CHANGELOG entry and EditMode tests.
  • Speed:
    • Two fewer Unity launches per integration test build, 52 → 24 launches per PR run.
    • Integration test and native SDK caches keyed on their inputs.
    • The leftover .NET 9 install and the workload restore are gone.
    • The iOS size baseline and the simulator terminate stall are off the Run iOS critical path.
    • The license retry is jittered and capped at 30 min.

Results so far

Ten back-to-back reruns of the unchanged stack tip:

Before (main 626d790) Stack tip
First-attempt green, PR 29–30% 8/10
PR wall time p50/p90 41/80 min 32.0/35.2 min
Unity launches per PR run 52 24
License-retry sleep ~9 min per build job 217–1,029 s per run in total

Merge order

The PRs are stacked; each one is based on the one below it. Merge from the bottom up. sentry-unity squash-merges, so after each merge the next PR is rebased onto main and its CI reruns.

app-runner goes first. The stack pins its PRs' exact commits, and merging them with merge commits keeps those pins valid:

  1. Skip the iOS simulator terminate right after boot and bound the others app-runner#65 (pinned by ci: take the iOS size baseline and the terminate stall off the Run iOS path #2910)
  2. fix(adb): Force-stop the app before each launch and after a run timeout app-runner#64 (pinned by test: relaunch an Android test app that never initialised once #2901)
  3. feat: Add Filter parameter to Get-SentryTestEvent app-runner#63 (pinned by test: ignore App Hang events in the iOS Crash Capture lookup #2899)
  4. main (app-runner)

sentry-unity, from the tip of the stack down to main:

  1. ci: fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes #2911 Fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes, plus the ConfigureOptions fix
  2. ci: take the iOS size baseline and the terminate stall off the Run iOS path #2910 Take the iOS size baseline and the terminate stall off the Run iOS path (needs app-runner#65)
  3. ci: key the native SDK caches on their own inputs and skip the workload restore #2909 Key the native SDK caches on their own inputs and skip the workload restore
  4. ci: recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts #2908 Recover from LicensingClient IPC timeouts, sentry-cli 5xx and flush timeouts
  5. test: give the playmode integration tests 5 s to receive an event #2906 Give the playmode integration tests 5 s to receive an event
  6. ci: cap job run time with timeout-minutes #2905 Cap job run time with timeout-minutes
  7. ci: retry known infra failures once, by signature #2904 Retry known infra failures once, by signature
  8. test: launch Unity two fewer times per integration test build #2903 Launch Unity two fewer times per integration test build
  9. ci: key the integration test caches on their inputs #2902 Key the integration test caches on their inputs
  10. test: relaunch an Android test app that never initialised once #2901 Relaunch an Android test app that never initialised once (needs app-runner#64)
  11. ci: jitter the Unity license retry and cap it at 30 minutes #2900 Jitter the Unity license retry and cap it at 30 minutes
  12. test: ignore App Hang events in the iOS Crash Capture lookup #2899 Ignore App Hang events in the iOS Crash Capture lookup (needs app-runner#63)
  13. ci: fail fast when the SDK build fails #2898 Fail fast when the SDK build fails
  14. ci: drop the leftover .NET 9 install from the Linux native SDK build #2907 Drop the leftover .NET 9 install from the Linux native SDK build
  15. main

Activity

  1. linear-code commented on Oct 27, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    • Status
      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions