Skip to content

ci: key the integration test caches on their inputs - #2902

Draft
bitsandfoxes wants to merge 7 commits into
ci/android-launch-recoveryfrom
ci/cache-content-keys
Draft

bitsandfoxes wants to merge 7 commits into
ci/android-launch-recoveryfrom
ci/cache-content-keys

Conversation

@bitsandfoxes

@bitsandfoxes bitsandfoxes commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

The integration test caches are now keyed on what goes into them, instead of on the run that wrote them, so a push to main no longer writes a fresh copy of every Unity Library.

The repo sat at 10.67 GB of its 10 GB cache quota. 7.49 GB of that was 16 it-library-* entries, all from a single run, because the keys end in -${{ github.run_id }}: every push to main saves a new set and evicts whatever was least recently used. What gets evicted are the caches that actually save time. There's no sdk= entry on main left (all 74 of them sit on PR refs, 25 distinct keys saved over and over), and only 13 of the 35 build-nosentry-* entries survive. That's why "Build without Sentry SDK" misses 50–80% of the time on main, the native SDK Build misses ~70%, and "Compile Xcode project without Sentry" misses ~95%. Over the last 60 days that's roughly 9,500, 2,840 (on the critical path) and 870 macOS minutes, and every "Build without Sentry" miss is also an extra Unity editor launch and license acquisition.

How it works now:

  • The it-library-<os>-<major> keys end in a hash of the downloaded IntegrationTest project's ProjectSettings/ProjectVersion.txt (the full editor version and changeset), its Packages/manifest.json and the parts of test/Scripts.Integration.Test that get copied into the project (Editor/, Scenes/, Scripts/, DependencyConflictPackage/, plus create-project.ps1 and add-sentry.ps1). A change to one of the other harness scripts doesn't touch the Library, so it doesn't re-save ~7 GB either. Both project files come out the same for every run of the same editor version. The key gets computed before Sentry is added to the manifest, and the save step reuses it (cache-primary-key) and only runs on a miss. The restore-keys prefix is unchanged, so a changed input still starts from the newest Library and Unity reimports what changed. The restore-only ubuntu jobs (WebGL, Android, iOS) use the same key, so they get the Linux job's Library as an exact hit.
  • The build-nosentry-* keys include the full Unity version as well (the same ProjectVersion.txt hash, so bumping one Unity version only rebuilds that version), and the compiled iOS one reads its version from scripts/unity-versions.json, since there's no project in that job yet when it restores. Until now a Unity patch bump kept serving the build-size baseline from the old patch.
  • The IntegrationTest-Project cache in test-create.yml is gone. It had no entry when we looked (it gets evicted like everything else), creating the project takes ~0.5 min, and its key didn't include the full Unity version either.
  • Library-unity-of-bugs (548 MB) is only saved on main, like the others. Every PR whose run found it evicted saved its own copy.
  • sdk.yml only checks out the submodules on a cache miss (~1 min per job, ~1,900 min over 60 days). Nothing after a hit uses them, and the key is built from git submodule status --cached, which reads the index, so it doesn't change.

Existing entries don't match the new keys, so the first main run after merging misses everything once and saves a single set; the old -<run_id> entries can be deleted or left to age out.

What it doesn't fix: if every entry is kept, the steady state is still about 11 GB (Libraries ~8 GB, a full set of 35 build-nosentry-* ~2.4 GB, Library-unity-of-bugs 0.55 GB, sdk= ~0.07 GB per key set), so LRU will keep evicting about 1 GB. PR runs only use 2021.3 and 6000.6, so what goes are the entries only main uses (2022.3, 6000.0, 6000.3), and they get saved again on the next main push: about 1 GB per push instead of 7.5 GB, and the sdk= and PR-version caches stay. Getting below 10 GB for good needs one more cut, e.g. leaving Library/Bee out of the Library caches (about two thirds of a Library locally) or dropping the macOS Library cache (2.74 GB), after an A/B on build time.

Measured, with a throwaway commit (since dropped) that let this PR branch save caches like main does:

  • First run (the jobs re-run after an unrelated pwsh crash in the SDK build): every content key missed and was saved once, 8 it-library-* (3.1 GB) and 14 build-nosentry-*. Android, iOS and WebGL share the it-library-ubuntu-* entries with Linux. Library-unity-of-bugs was an exact hit, so it wasn't saved again.
  • Second run, same commit: all 15 Library restores, 12 build-without-Sentry restores and both runtime compiled-iOS restores were exact hits. All 25 save steps were skipped, and "Update submodules" was skipped in all 7 sdk jobs.
  • gh api repos/getsentry/sentry-unity/actions/cache/usage: before 10.20 GB / 126 entries. During the two runs it peaked at 11.10 GB, since the PR's own set sat next to main's old -<run_id> entries. The old entries stop being restored and age out.

Rebased onto #2907: run 40275305182 failed Build WebGL 6000.6 on a stale Library/burst.pid (fixed in #2911), then passed on a rerun of the failed jobs.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant