Skip to content

ci: fix the symbolication, iOS first-launch, ingest 5xx and burst.pid flakes - #2911

Draft
bitsandfoxes wants to merge 7 commits into
ci/ios-critical-pathfrom
fix/ci-stable
Draft

bitsandfoxes wants to merge 7 commits into
ci/ios-critical-pathfrom
fix/ci-stable

Conversation

@bitsandfoxes

@bitsandfoxes bitsandfoxes commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes the flakes that still failed CI runs on top of the stack, each traced to its cause.

  • Unsymbolicated native frames. sentry-cli debug-files upload returns before sentry.io has processed the files. In run 39645739252 the macOS GameAssembly debug companion was uploaded at 17:54:36 and listed by the server at 17:57:27; the crash event came in at 17:56:02, was symbolicated without it and never reprocessed. assert-symbol-upload.ps1 now polls files/dsyms/?debug_id= until each uploaded debug companion, IL2CPP mapping and source bundle is listed (portable PDBs count via their embedded sources feature), for at most 300 s. A timeout, a missing token or a 401/403 is a ::warning::, not a failure. The assert steps get SENTRY_AUTH_TOKEN for that (Compile iOS gets it as a new workflow_call secret).
  • iOS first launch hangs. Over 62+ Run iOS legs, a first launch less than 100 s after the simulator boot stalled in 6/12 legs, 100–130 s in 1/11, at 130 s or later in 0/51; waiting for simctl bootstatus doesn't help (Skip the iOS simulator terminate right after boot and bound the others app-runner#65). Run iOS now starts with Build iOS, boots the simulator while Compile iOS runs, and waits for the app artifact with wait-for-artifact. When the artifact is already there (a rerun), the first launch waits until 150 s after boot. Job timeout 25 → 45 min to cover the wait.
  • Crash report dropped on a sentry.io 5xx. sentry-cocoa deletes an envelope that the server answers with a 5xx (Received response status code: 503 in run 40264124350), so the crash event never arrives. crash-capture now reruns once per job when the send output shows a 5xx.
  • Unity 6000.6 build killed with exit 137 in Docker. Unity 6000.6 kills the PID in Library/burst.pid at startup. The project comes from Create in another container, where that PID was 259; in the build container PID 259 is the xvfb-run wrapping the editor (run 40275305182). The orphaned editor then failed the retry with "another Unity instance is running". build-project.ps1 removes Library/*.pid before building.
  • A retried configure-and-build launch built without Sentry's options configuration. OptionsConfigurationItem.SetScript re-created the configuration asset on every ConfigureOptions. On 6000.6 AssetDatabase.CreateAsset destroys the old object, and ??= kept the destroyed reference; on 2021.3 the reference stays live but no longer points to the asset. Either way, a second ConfigureOptions saved OptionsConfiguration/CliOptionsConfiguration as {fileID: 0}, so the build had no auth token (seen in run 40314070230 attempt 10, when the shutdown-crash retry re-ran that launch). SetScript now reuses the asset already at the path; EditMode tests cover both references, and fail before on both Unity versions. Users running ConfigureOptions twice hit the same bug, so it has a CHANGELOG entry.
  • Two gaps on the retry path. The symbol-upload assertion now accepts sentry-cli's File upload/processing complete, which is all a rebuild prints when every chunk is already on sentry.io, and fails on some symbols did not process correctly. The shutdown-crash retry now also fires on exit 133 (SIGTRAP) after Batchmode quit successfully invoked, for a shutdown stall that kills Unity before it logs the signal.

Stacked on #2910.

CI evidence: run 40314070230, rerun in full back to back on 383a6c3 (before the last three commits): 8 of 10 attempts green on the first try, p50 32.0 / p90 35.2 min for the green ones (baseline 41 / 80). All 20 symbol waits in attempt 1 finished in 0–14 s, none timed out. The two red attempts:

  • Attempt 8, Build Android 6000.6: a Unity-internal Bee error (Request cancelled while reading the binlog) reported as compiler errors without compiling. First sighting in 26 runs.
  • Attempt 10, Build Linux 6000.6: the shutdown-crash retry of the configure-and-build launch ran ConfigureOptions a second time. OptionsConfigurationItem.SetScript re-creates the configuration assets, and ??= keeps the deleted objects, so both references were saved as {fileID: 0} and the build had no auth token. Fixed in this PR (see above).

A throwaway commit (since dropped) that ran the configure-and-build launch twice in every build job, as a retry would: run 40386193023 is green in 33.2 min, and every build called CliConfiguration::Configure() on the second launch with no Auth Token missing. The clean head: run 40391327148 is green on the first attempt in 27.3 min.

🤖 Generated with Claude Code

@bitsandfoxes
bitsandfoxes force-pushed the fix/ci-stable branch 2 times, most recently from 17dd6c7 to bc44cac Compare October 26, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant