Skip to content

ci: cap job run time with timeout-minutes - #2905

Draft
bitsandfoxes wants to merge 1 commit into
ci/infra-retriesfrom
ci/job-timeouts
Draft

bitsandfoxes wants to merge 1 commit into
ci/infra-retriesfrom
ci/job-timeouts

Conversation

@bitsandfoxes

@bitsandfoxes bitsandfoxes commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Every CI job now has a job-level timeout-minutes, so a hung job fails within its usual worst case instead of running for up to 6 hours.

Without a job timeout, GitHub stops a job after 6 h. Two jobs in the last 30 days ran that long, and a hung job that holds a Unity editor keeps its license seat leased. Step timeouts don't cover this: while measuring this stack, a Run iOS job's "Run iOS Integration Tests" step with timeout-minutes: 20 was still running at 39 min, and had to be cancelled by hand (job 121176550259). The new 25-min job timeout would have stopped it.

Each number covers the job's green p95 from the last 30 days, plus the worst case the stack below allows: a 30-min license wait per Unity launch (#2900), one retry per infra signature (#2904), and a consumer that stops waiting for package-release once the SDK build fails (#2898).

Job Timeout (min) Why
Build Unity SDK 90 p99 72.5. It waits up to ~35 min for the native SDKs (sdk.yml already has 30), plus build and three Unity launches with one full license wait
Build Android/iOS/Linux/macOS/Windows/WebGL/x86/Mono 120 Their timer runs while they wait for package-release, which can land late in a legitimately slow 90-min SDK build. After that they need ~25 more min for their own launches
Create Test Project 40 One launch, so one full license wait (30) plus pull and create. Max seen 34
Compile iOS 45 On a build-without-Sentry cache miss it runs both 20-min compile steps
Run iOS 25 20-min step plus setup and the failure upload
Run Android 35 The 30-min test step plus device setup (Redroid restart, sdkmanager retries)
Run desktop / WebGL 25 The 20-min run step plus setup
UPM Package validation 10 It needs: the SDK build now, so it no longer waits. 19 s on a green run
create-unity-matrix, Build Size 5

The step timeouts stay as they are. Shorter ones would cut off the tests' own 300 s event polls, so a missing event would end as "The action has timed out" instead of the Pester failure that names it. cancel-previous-workflow gets none: its only conflict-free spot next to the UaaL branch's runs-on change would be after its steps. Each timeout-minutes is the first key under its job, which keeps it away from the runs-on lines ci/uaal-integration-tests changes, and the two branches merge cleanly.

Stacked on #2904.

CI evidence: run 40217553065 passed 56/56 in 43 min with these limits. It's the green run for the whole stack below on its final commits, without any throwaway commits.

Rebased onto #2907: run 40287391159 is green on the first attempt in 33.0 min.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant