[Broccolito]

Why main went red with every step green

Two pushes to main on 2026-10-18 had test (ubuntu-latest) cancelled at the 40-minute limit: 39156748556 (40.0 min) and 39158205693 (40.2 min). Every step succeeded, "Complete job" included. Both had a cold Rust cache.

It was not a key change, and PR and main runs share the cache already. I read the rust-cache lines from the jobs API for six runs that day. All six computed the same key, v0-rust-ubuntu-latest-test-Linux-x64-edf29d0d-c299e55f, on the same runner image:

Run	Event	Time (UTC)	rust-cache restore	Job
39117814789	push main	10:04	Cache hit	29.4 min
39150548727	PR release/1.93.2	17:39:11	Cache hit	29.2 min
39156748556	push main	19:00:45	No cache found	40.0 min, cancelled
39157053442	PR	19:04	No cache found	32.1 min
39158205693	push main	19:19	No cache found	40.2 min, cancelled
39161387447	push main	20:01	Cache hit (saved 19:40 by 39156748556)	30.1 min

The entry existed at 17:39 and was gone by 19:00. GitHub evicted it.

• At 18:16 and 18:18 the release branch's Linux GUI packages / Windows GUI packages dispatches saved two cross caches: 1.2 + 1.4 GB, scoped to release/1.93.2 and readable by nothing else.
• That pushed the repository past its 10 GB cache limit (/actions/cache/usage reads 12.35 GB right now).
• GitHub evicts in order of last access. The least recently used entry was the ubuntu test cache: the release PR had read it at 17:39:11, one second before serve's, and six seconds before the others.

Racing made it worse but did not cause it. The two main runs started 19 minutes apart, both before either had saved. So both paid the cold price.

A second, recurring cause. Swatinem/rust-cache hashes every installed toolchain into its key: rustup toolchain list, then rustc -vV of each (getRustVersions in its src/config.ts). That includes the runner image's own stable, which nothing here builds with, because rust-toolchain.toml pins 1.92.

• Image 20260901.588 → 20261002.596 moved that stable from 1.98.1 to 1.99.0.
• The key went from …-15238ee1-c299e55f to …-edf29d0d-c299e55f.
• Every ubuntu run built cold from 2026-10-10 until main next saved on 10-12, 37.0–39.2 min each.

Why a cold cache turns into a red X. The 40-minute budget was set on 2026-08-27, when this job took 15 minutes warm. Since then the integration-binary step and the socket gate have doubled it, and the budget stayed. Measured over the completed runs of 2026-10-10..18, classified by each job's own rust-cache restore line:

OS	warm	cold
ubuntu	21.3–31.0 min (n=52)	30.5–40.2 min (n=10; eight above 37)
macos	14.3–38.2 min (n=60)	20.1–37.6 min (n=6)
windows	15.5–29.7 min (n=54)	21.3–31.6 min (n=8), and one cold main run still in cargo test when 40 hit (39057096529)

Main's first run after any key change is cold by design: a dependency in Cargo.lock, the toolchain, or a CARGO_* variable. So the budget has to hold a cold run.

What changes

• rust.yml test: timeout-minutes 40 → 60. That is 1.5× the measured cold maximum, which is the margin the guards comment says these budgets keep. A hang still costs an hour, not six. The comment carries the table above.
• Pinned toolchain only, before every rust-cache step (test, serve, frontend.yml api-contract, apps-smoke.yml). A small step removes every toolchain except rustup show active-toolchain. Runner image updates no longer change the keys.
• save-if: main on the rust caches that lacked it (serve, api-contract, apps-smoke), as test already did. During the 10-11/12 eviction, each PR's first serve run wrote its own 1.9 GB copy that no other branch could read (38176616426, 38204107268, 38235367922, 38240320573). That compounded the eviction.
• GUI package workflows: no persisted cross cache. Measured over every dispatch since 2026-09-24:
  • A release branch's save was never read again (release/1.92.0, 1.93.1 and 1.93.2 each dispatched once).
  • Saving from main only would not fit either. Main's own caches come to about 8.3 GB of the 10. rust.yml's cross-check restores the same cross-<triple>- prefix on every run, so the pair would stay resident and push a test cache out instead.
  • Cold builds take 34.7–37.5 min (Linux, n=6) and 30.8–39.1 min (Windows, n=4), against a 100-minute limit. That is the same from-scratch build release.sh already does.

Nothing under tests/ or src/ changes. The integration-binary step, its exclusion table and the network namespace are untouched.

Expected one-time cost. Pruning the toolchain changes every rust-cache key once. This PR's own runs and the first push to main after merging build cold, under the new limit. Main then saves the new keys.

Measured result

This PR's own run (39371819622) was cold on every OS by design. The new keys hash one toolchain. Each job's log shows info: uninstalling toolchain stable-<host>, then Rust Versions: 1.92.0 alone and No cache found:

Job	New key	Cold duration	Limit
test (ubuntu-latest)	...-ed8f1928-c299e55f	32.6 min (lib+bins 13.1, integration 8.7, socket gate 4.7, clippy 5.4)	60
test (macos-latest)	...-4276b576-c299e55f	31.6 min	60
test (windows-latest)	...-a733e789-c299e55f	24.8 min	60
serve	...-4bbc99e1-c299e55f	14.1 min	45

The prune step itself took under 0.1 min on all three OSes.

The two pushes to main after merging ran back to back and were both cold. That is the same situation as the 2026-10-18 pair, where the second push starts before the first has saved. Both passed:

Push	Run	test (ubuntu-latest)	macos	windows	Steps (ubuntu)
2291a0254 (this PR)	39384759671	40.6 min, success	41.6	24.9	lib+bins 15.8, integration 11.0, socket gate 6.0, clippy 6.5, save 0.6
06ca6bd24 (#419)	39384909493	41.8 min, success	41.0	33.6	lib+bins 16.1, integration 11.3, socket gate 6.1, clippy 6.8, save 0.5

All four ubuntu and macOS times are above the old 40-minute limit, so the old budget would have cancelled every one of them. They used 68–70% of the new limit.

• The first run saved v0-rust-ubuntu-latest-test-Linux-x64-ed8f1928-c299e55f at 07:50.
• The second run's save reported "Unable to reserve cache … another job may be creating". That is harmless: the first run saved the same key.

Main's caches are now the six new rust entries plus npm, about 8.3 GB. The old-key entries still counted in /actions/cache/usage (11.3 GB) are the least recently used, so they are what GitHub evicts first.

The next push to main is the first warm run on the new keys.

Not changed, noted

• rust.yml cross-check's cache save fails on every run: "/usr/bin/tar" failed with exit code 2 on the container's root-owned files. So it never persists anything; it builds cold, max 8.6 / 18.5 min of 30. If that save is ever fixed, it needs main-only saving first, or every PR writes 2.6 GB.
• computer-use-package-acceptance.yml (dispatch-only, last run 2026-09-24) still saves its caches on whatever ref it is dispatched on.
• setup-node's npm caches (about 0.16 GB per OS) are still written per PR when package-lock.json changes.

🤖 Generated with Claude Code