Skip to content

ci: jitter the Unity license retry and cap it at 30 minutes - #2900

Draft
bitsandfoxes wants to merge 1 commit into
ci/crash-capture-filterfrom
ci/license-retry-jitter
Draft

bitsandfoxes wants to merge 1 commit into
ci/crash-capture-filterfrom
ci/license-retry-jitter

Conversation

@bitsandfoxes

@bitsandfoxes bitsandfoxes commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

When an editor launch can't get a Unity license, RunUnity now waits a random 20–90 s before the next try instead of a fixed 60 s, and gives up after 30 min instead of 60 min with an error that says the license seats are exhausted.

Unity licenses are leased per editor process, not per job. Every build job gets package-release at about the same moment and then starts 3–5 editors in a row, so the acquisitions come in a burst. In 938 green licensed jobs (2026-09-25 to 10-24), 65% hit at least one "no license" retry, sleeping ~9 min per build job on average (p95 26–39 min), 7,150 runner-min in 30 days. With a fixed 60 s interval, jobs that miss together retry together, and they keep running into the same full server. And when the license server is down, a job sat in the loop for up to an hour before failing: the 2026-10-12/13 outage held jobs for up to 2h08m.

How it works now:

  • Each "No valid Unity Editor license found" miss sleeps a random whole number of seconds in 20–90 (mean 55, close to the old 60), so jobs that missed together spread out over the next minute and a half.
  • The total license wait per RunUnity call is capped at 1,800 s. The last sleep is shortened to fit, as before.
  • When the cap is hit, the step fails with ::error::Unity couldn't acquire a license after N tries in 1800 seconds. The floating license entitlements are likely exhausted (no free Unity license seat)., so a red run says why. It says "likely" because the "No valid Unity Editor license found" line it matches may also show up when the license server is unreachable or not configured.
  • A single miss still retries. Waiting costs runner time, not seats, so it doesn't fail fast.
  • The retry loop no longer checks the license cap as its own exit condition. The license branch already throws once time runs out. Before, a Unity 6.6 self-SIGKILL (exit 137) that landed after the cap skipped its one retry, and halving the cap would have made that more likely.
  • The It took N tries notice/warning is unchanged.

This is step 1 of fewer license acquisitions per build job. Removing the add-sentry and configure-sentry editor launches comes separately.

CI evidence, comparing the 17 licensed jobs of this branch's green run (run, 56/56 jobs passed) with the parent branch's green run an hour earlier (run):

  • Retry sleeps: all 14 were 60 s before. The 20 here ranged from 24 to 90 s (mean 53 s), so jobs that miss together no longer retry together.
  • Jobs with at least one license retry: 10/17 before, 11/17 here. Retries per run: 14 before, 20 here. That's run-to-run noise at Sunday load; one run can't show the effect on the burst. The 60-day numbers above are the baseline to re-measure against once this has been on main for a while.
  • The 30-minute cap wasn't reached; that only happens during an outage.

Rebased onto #2907: run 40268408803 failed Run Android 2021.3 runtime on an app that never initialised (fixed in #2901), then passed on a rerun of the failed jobs.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant