ci: bound apt-get so a stalled mirror can't burn a 6h job

[prql-bot]

apt-get update has no wall-clock bound, and on 2026-08-18 a musl build job on main stalled inside it for 359 minutes before GitHub killed it at the 6-hour cap. This routes the five apt-get call sites through a new apt_install.sh that bounds each call and retries it, and caps all four build-prqlc* jobs — two in tests.yaml, two in release.yaml — as a backstop. An update that can't complete is a warning rather than a build failure, because the packages we install resolve from the runner image's baked-in lists anyway. The stall reproduced twice on this PR's own CI, which is what set both the timeout value and that last decision — see Verification.

What happened

On 2026-08-18 at 16:27 UTC, build-prqlc-c (ubuntu-24.04, x86_64-unknown-linux-musl, default) on main reached Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease inside sudo apt-get update and never came back. The next line in its log is The operation was canceled — 359 minutes later, at the 360-minute default job timeout (run 32160030161). Two sibling aarch64-unknown-linux-musl legs in the same run died the same way — those two are release.yaml's build-prqlc/build-prqlc-c, reached from tests.yaml via nightly.yaml → nightly-release on every main push, which is why the cap has to cover both files. That job normally takes under a minute: every other completed run of it across the last 25 tests runs finished in 0–3 minutes.

It was still happening while this was written — build-prqlc-c on run 32223676248 (the main push for #6208) and build-prqlc on run 32227330677 (#6209) were each over an hour into the same step.

Why it matters beyond the wasted runner hours

A job killed at the timeout cap reports cancelled, not failure. cancelled is also what a superseded concurrency-group run reports, so nothing watching for a red default branch can tell the two apart — tend-ci-fix gates on conclusion == 'failure' and skipped it. The practical effect: no push-triggered tests run on main has concluded successfully since 2026-08-17T08:49Z. The four commits merged on 08-18 (#6202, #6204, #6205, #6207) have never had a green tests on main, and nothing surfaced that.

The change

• .github/workflows/scripts/apt_install.sh (new) — runs apt-get update under sudo timeout -k 30 300, retrying up to 3 times, then installs the same way. timeout is invoked under sudo rather than the reverse so it signals apt-get directly, and -k 30 is what makes the budget a bound rather than a request: plain timeout sends SIGTERM and then waits for the child indefinitely, and apt defers termination signals while dpkg is working. Worst case is ~22 minutes, inside the 30-minute job cap.
• An update that never completes is a ::warning::, not an error — only the install decides the step's exit status, and it annotates its own failure. The ubuntu-24.04 image ships with /var/lib/apt/lists populated, so a mirror that won't serve InRelease shouldn't fail a step that can resolve without it. Refreshing is still attempted first, since a stale index can 404 at download time. The install still fetches the .deb from that same mirror under the same bound, so a full outage is still red — just in minutes rather than hours.
• Five call sites now use it: the musl-tools and gcc-aarch64-linux-gnu installs in build-prqlc and build-prqlc-c, and the musl-tools install in test-rust.yaml. Relative-path invocation from a composite action is the same pattern set_version.sh already uses in these two files.
• A job-level timeout-minutes on all four build-prqlc* jobs, as a backstop for a hang from any other cause. tests.yaml gets 30 — its slowest successful leg is 11 minutes (macOS), so roughly 3× headroom, and it matches the value test-devcontainer already uses. release.yaml gets 90: those legs build profile: release, their slowest successful leg is 24 minutes (macos-15-intel), and on the aarch64 legs apt_install.sh runs twice per job (the runner.os == 'Linux' step for musl-tools, then the target-gated step for gcc-aarch64-linux-gnu), so the bounded worst case there is ~45 minutes of apt plus a ~12 minute build. A 30-minute cap on those would turn routine green release builds red.

test-rust gets the apt fix but no job-level cap — it's a much longer and more variable job and I don't have evidence for a safe value.

What this does and doesn't fix

It doesn't make a stalled Ubuntu mirror work. It bounds the damage: instead of holding a runner for six hours and reporting cancelled, the job now spends at most ~22 minutes per apt_install.sh invocation (~45 on the two aarch64 legs, which call it twice), and finishes green if the packages are installable from the indices already on the image. It goes red — with an ::error:: naming the failing packages and apt's exit status (124/137 means the install hit its own bound, anything else is apt refusing the packages) — only when the packages genuinely can't be installed. The underlying flakiness is on GitHub's side.

Verification

The stall reproduced twice on this PR's own CI, which is what set both the timeout value and the warn-don't-fail behaviour.

At 7f73bdda the budget was 120s/attempt, and build-prqlc (ubuntu-24.04, x86_64-unknown-linux-musl) failed after all three attempts — correctly, and in 6m20s rather than 6h. Reading that job's log showed why 120s was wrong: azure.archive.ubuntu.com was unreachable (every index Ign:), apt spent ~40s retrying it before failing over to archive.ubuntu.com, and was then killed mid-download of the package indices. So the degraded-but-working path needs meaningfully more than two minutes. 5dc802e9 raises it to 300s, and 9bd6e53c adds -k 30 so the bound holds even against a process that ignores SIGTERM (verified against a trap '' TERM child: timeout 3 never returns, timeout -k 2 3 returns at 5s with exit 137).

At 9bd6e53c it reproduced again, and this time 300s wasn't the issue — the mirror was down for the whole 15 minutes. Both musl legs went red (build-prqlc, build-prqlc-c). All three attempts got as far as Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease and then sat there until the budget ran out — the exact signature from main, so the bound worked, but a red build is still the wrong outcome for a step that didn't need the network. 7729df04 makes the update best-effort.

The premise for that, checked on a stock ubuntu-24.04 runner with no apt-get update run first:

$ apt-cache policy musl-tools
musl-tools:
  Installed: (none)
  Candidate: 1.2.4-2
  Version table:
     1.2.4-2 500
        500 mirror+file:/etc/apt/apt-mirrors.txt noble/universe amd64 Packages

apt-get install -s -y plans a complete install for both musl-tools and gcc-aarch64-linux-gnu from those baked-in lists, and /var/lib/apt/lists holds 22 _Packages indices on a fresh image. The failing run's own log agrees: Hit:2 https://archive.ubuntu.com/ubuntu noble InRelease means apt already had a cached index for the pocket the packages come from.

Also checked:

• Control flow replayed against a stubbed sudo/timeout/apt-get on PATH, all four paths: update-ok + install-ok runs one update then the install and exits 0; update-always-fails + install-ok emits three ::warning:: attempt lines plus the summary warning, then installs and exits 0; update-fails + install-fails emits ::error::apt-get install failed (exit 1) and exits 1; update-ok + install killed at its bound exits 124, with the status carried through to the annotation.
• shellcheck and bash -n on the script — clean.
• prettier --check and yaml.safe_load on all four edited YAML files — clean.
• Job durations sampled from the last 40 tests runs via the jobs API (151 successful build-prqlc* legs): slowest is 10 min (macos-15), musl legs 0–1 min.

[prql-bot]

The diagnosis holds up independently — I confirmed the 360-minute cancelled leg in the jobs API, and sampled 151 successful build-prqlc* legs across the last 40 tests runs: the slowest is 10 minutes (macos-15), musl legs 0–1 minutes, so timeout-minutes: 30 has real headroom and won't turn green builds red. The relative-path invocation is sound at all five call sites — set_version.sh already runs that way from both composite actions, and every consumer (tests.yaml, release.yaml) checks out to the workspace root with no working-directory override. No apt-get call site was missed; the only other one in the repo is the advisory msg: string in Taskfile.yaml.

One substantive gap, inline: timeout without --kill-after doesn't actually guarantee a bound. It sends SIGTERM and then waits indefinitely for the child to exit — I verified this locally, where timeout 3 against a TERM-ignoring child never returned and ran out a 2-minute budget. That matters here because the bound is the feature: the new comment claims the pathological case is bounded "at ~15 minutes rather than 6 hours", and apt defers termination signals while dpkg is working, so the install call is the likeliest one to sit through its TERM. The consequence isn't symmetric across call sites either — build-prqlc* have the timeout-minutes: 30 backstop to catch it, but test-rust.yaml deliberately has none, so a TERM-ignoring apt there is back to burning the 6h cap and reporting cancelled.

Two smaller notes, neither blocking:

• release.yaml's build-prqlc and build-prqlc-c carry the same unbounded-job exposure and didn't get the backstop. They share the composite actions, so they do get the apt fix — but a hang from any other cause still runs out the 6h default there. I'm not proposing a number, since those run profile: release across five targets and I don't have duration samples for them.
• The PR description still says the script runs apt-get update under sudo timeout 120; 5dc802e raised it to 300. Worth syncing so the description and the code agree.

How the numbers were checked

Durations, from repos/PRQL/prql/actions/runs/<id>/jobs over the 40 most recent completed tests runs, filtered to conclusion == "success" — 151 samples. Max per leg:

10  build-prqlc (macos-15, aarch64-apple-darwin, default,test-dbs)
 9  build-prqlc-c (macos-15, aarch64-apple-darwin, default,test-dbs)
 6  build-prqlc-c (windows-latest, x86_64-pc-windows-msvc, default)
 5  build-prqlc (windows-latest, x86_64-pc-windows-msvc, default)
 1  build-prqlc (ubuntu-24.04, x86_64-unknown-linux-musl, default)
 1  build-prqlc-c (ubuntu-24.04, x86_64-unknown-linux-musl, default)

The same query surfaced the hang: build-prqlc-c (ubuntu-24.04, x86_64-unknown-linux-musl, default) at 360 minutes, cancelled.

Worst case against the 30-minute cap, after 5dc802e, on the leg that actually runs apt (musl): 3 × 300s update + 2 × 10s sleep + 300s install ≈ 20.3 min, plus a ~1 min build. Fits — but that's the arithmetic -k is what makes true.

The timeout check: a script doing trap '' TERM; sleep 600, run as timeout 3 ./ignore_term.sh, produced no exit line and consumed the full 2-minute command budget. timeout -k 2 3 is the form that terminates.

[prql-bot]

-k 30 closes the gap from the last review — both call sites now have a hard bound, and I re-checked the mechanism independently: timeout -k 2 3 against a trap '' TERM child returns at 5s with exit 137, where plain timeout 3 never returns. Threads resolved. shellcheck and bash -n on the updated script are clean, and the worst case still fits the 30-minute job cap (3 × 330s update + 2 × 10s sleep + 330s install ≈ 22 min).

The release.yaml note from the earlier review stands as a follow-up rather than a blocker — it's a separate concern from what this PR sets out to fix.

Commenting rather than approving because I authored this PR; a maintainer's call on merging. CI is still in flight at this head (52 checks pending, none red yet).

[prql-bot]

The warn-don't-fail turn is the right call for this failure mode, and the stubbed-PATH replay covers all three paths. One design point, plus some drift the switch left behind.

The retry budget now sits on the call that can't fail. After this commit the update — which no longer decides the step's exit status — gets three attempts and a sleep between them, while the install, which does decide it, gets one. But the install fetches the .deb from the same mirror under the same 300s bound, so a stall there is an immediate red with no retry: the retry logic ended up protecting the call whose failure is now cosmetic. Two attempts each would fit the cap (2 × 330 + 10 + 2 × 330 + 10 ≈ 22 min). I'm not pushing that, because it moves the arithmetic against the timeout-minutes: 30 a reviewer would be signing off on, and there's a fair argument the other way — a mirror that won't serve InRelease probably won't serve the package either, so a second install attempt may just spend five more minutes reaching the same red. Worth a maintainer's call rather than mine.

Inline, four things in the header comment and the exit path. The "failing fast keeps that signal honest" paragraph now argues against the paragraph this commit added below it; ~17 minutes is the update loop's bound rather than the script's (~22, which is what the description says); and "resolve from the baked-in indices with no network at all" describes resolution only — the install still needs the mirror, which is the boundary that decides whether this change actually converts a red into a green.

I've synced the description, which promised an ::error:: naming the cause — this commit removed the script's only one, so an install failure now surfaces as GitHub's generic Process completed with exit code 1. That's the illegible-failure shape this PR exists to fix, so the inline suggestion restores an annotation rather than dropping the claim.

Commenting rather than approving because I authored this PR. CI at this head: 19 checks pending, none red. build-prqlc (ubuntu-24.04, x86_64-unknown-linux-musl) went green in under two minutes, so the mirror has recovered and the warn path isn't being exercised on this run — it's covered by the replay, not by a real stall. The release.yaml note from the first review still stands as a follow-up, not a blocker.

[prql-bot]

feee3437 applies all four points from the last review correctly. I re-replayed the exit path against a stubbed sudo/timeout/apt-get on PATH across all six combinations of update-rc × install-rc: update-ok/install-ok exits 0 silently; update-fail/install-ok emits 4 ::warning:: lines and still exits 0; install-fail emits ::error::apt-get install failed (exit N) for: musl-tools and carries N (1 and 124 both) through to the script's exit. The || status=$? form is the right idiom here — if ! cmd would have clobbered $? to 0 inside the branch — and the trailing if with a false condition exits 0, so the happy path is unaffected.

One new finding, which I've now pushed a fix for rather than leaving as a follow-up. It supersedes the release.yaml note I filed as non-blocking on the first review — the evidence turns out to be much stronger than I framed it then.

The timeout-minutes: 30 backstop doesn't cover two of the three jobs that hung in the incident this PR cites. I pulled the job list for run 32160030161 rather than reading the description's summary of it, and the three cancelled jobs are:

job	defined in	capped by this PR?
build-prqlc-c (ubuntu-24.04, x86_64-unknown-linux-musl, default)	tests.yaml	yes
nightly / nightly-release / build-prqlc (ubuntu-24.04, aarch64-unknown-linux-musl)	release.yaml	no
nightly / nightly-release / build-prqlc-c (ubuntu-24.04, aarch64-unknown-linux-musl)	release.yaml	no

The two aarch64 legs aren't a separate workflow that happens to share code — tests.yaml calls nightly.yaml on main, which calls release.yaml as nightly-release, so they run inside the same tests run on every main push. tests.yaml's own build-prqlc matrix has aarch64-unknown-linux-musl commented out ("They run on release.yaml regardless"), which is why the cap as written can't reach them. They get the apt bound, since they go through the same composite actions — but the job-level backstop for a hang from any other cause is exactly what they don't get, and they're where two-thirds of the observed 6-hour damage happened.

I picked timeout-minutes: 90 rather than 30, because two things make these jobs genuinely slower than the tests.yaml ones:

• They build profile: release across five targets. Sampling successful nightly-release / build-prqlc* legs across the last 40 tests runs, the slowest is 24 min (macos-15-intel, x86_64-apple-darwin), then 16 (windows-latest), 13, 12. A 30-minute cap would sit 25% above a routinely-observed duration and start turning green release builds red.
• On the aarch64-unknown-linux-musl legs, apt_install.sh runs twice per job — once from the runner.os == 'Linux' step for musl-tools, then again from the inputs.target == 'aarch64-unknown-linux-musl' step for gcc-aarch64-linux-gnu. Both conditions hold on that leg, so the script's ~22 min worst case is ~45 min for the job, plus a ~12 min build ≈ 57 min. Any cap below that would convert a fully-stalled mirror straight back into a cancelled, which is the outcome this PR exists to eliminate. 90 clears it, and still turns a 6-hour hang into a 1.5-hour one.

Worth noting the script's header comment says "~22 minutes" — that's per invocation and correct as written, but it's not the per-job bound on those two legs. I haven't touched the comment, since the script is the right place for the script's own bound.

Commenting rather than approving because I authored this PR — a maintainer's call on merging. CI at feee3437: 23 checks pending, none red.

How the durations were sampled

Over the 40 most recent completed tests runs, from repos/PRQL/prql/actions/runs/<id>/jobs, filtered to conclusion == "success" and job names matching nightly-release / build-prqlc. Max per leg, in minutes:

24  build-prqlc (macos-15-intel, x86_64-apple-darwin, default,test-dbs)
16  build-prqlc (windows-latest, x86_64-pc-windows-msvc)
13  build-prqlc-c (windows-latest, x86_64-pc-windows-msvc)
13  build-prqlc (macos-15, aarch64-apple-darwin)
12  build-prqlc (ubuntu-24.04, aarch64-unknown-linux-musl)
10  build-prqlc-c (macos-15, aarch64-apple-darwin)
 9  build-prqlc-c (ubuntu-24.04, aarch64-unknown-linux-musl)
 4  build-prqlc (ubuntu-24.04, x86_64-unknown-linux-musl)
 3  build-prqlc-c (ubuntu-24.04, x86_64-unknown-linux-musl)

The macOS legs aren't cache-assisted here — save-if in both composite actions is (github.ref == 'refs/heads/main') && contains(inputs.target, 'musl') — so 24 min is already close to a cold-cache figure rather than a warm-cache one.

The exit-path replay, six cases, against stubs on PATH:

update_rc=0 install_rc=0   -> exit=0    0 warnings  no error
update_rc=0 install_rc=1   -> exit=1    0 warnings  ::error::apt-get install failed (exit 1) for: musl-tools
update_rc=0 install_rc=124 -> exit=124  0 warnings  ::error::apt-get install failed (exit 124) for: musl-tools
update_rc=1 install_rc=0   -> exit=0    4 warnings  no error
update_rc=1 install_rc=1   -> exit=1    4 warnings  ::error::apt-get install failed (exit 1) for: musl-tools
update_rc=1 install_rc=124 -> exit=124  4 warnings  ::error::apt-get install failed (exit 124) for: musl-tools

[prql-bot]

This cap is right for the jobs it lands on, but it can't reach the ones that produced two of the three hangs in the run the PR cites. nightly / nightly-release / build-prqlc (ubuntu-24.04, aarch64-unknown-linux-musl) and its build-prqlc-c sibling are defined in release.yaml, reached from this same workflow via nightly.yaml -> nightly-release on every main push — and the matrix comment 14 lines below ("They run on release.yaml regardless") is exactly why tests.yaml has no aarch64 leg to cap.

I've pushed timeout-minutes: 90 onto both release.yaml build jobs rather than 30: their slowest successful leg is 24 min (macos-15-intel), and the aarch64 legs invoke apt_install.sh twice — the runner.os == 'Linux' step and the target-gated step both fire — so their per-job apt worst case is ~45 min, not ~22. Details in the review body.

[prql-bot]

On the one point from that review that wasn't inline — the retry budget sitting on the update rather than the install — I've left it as-is, and I think that's the right call rather than just a deferral.

The rebalance is arithmetic-neutral (2 × 330 + 10 twice ≈ 22 min, same as 3 update attempts plus one install), so cost isn't the objection. What's missing is evidence. Every stall observed so far — the 359-minute leg on main, and both reproductions on this PR's own CI at 7f73bdda and 9bd6e53c — is an index-refresh stall: each attempt reached Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease and sat there. The install has never been the thing that hung, because until 7729df04 a failed update was fatal and the script never reached it. So a second install attempt would be protecting a failure mode that hasn't been seen, at the cost of five more minutes on a run that a mirror outage is sending red either way.

The asymmetry is also less lopsided than it reads. The update retries aren't only protecting the update — a stale index 404s at download time, which is an install failure, so the attempts feed the call that decides the exit status even though they don't share its budget. And the two are no longer symmetric in what they need: a .deb fetch is a single-file GET that either connects or doesn't, where InRelease refresh across ~22 indices has many more places to stall.

If a real install stall does show up now that the script reaches it, the ::error:: added in feee3437 makes it identifiable — exit 124/137 is the bound firing, anything else is apt refusing the packages — which is the evidence that would justify the change. Happy to make it now if a maintainer would rather have it pre-emptively; it's a one-line loop change with no effect on the timeout-minutes arithmetic.

CI at feee3437, the commit I pushed: 55 checks green, none red, including both x86_64-unknown-linux-musl build-prqlc* legs and test-rust. Three were cancelled by supersession when ed26f16f landed on the branch, and the nightly-release matrix was still pending at the poll cap — unverified rather than green.

[prql-bot]

Confirming the release.yaml cap from the last review as the PR's author, and closing out CI, which was still pending when that review was written.

ed26f16f puts timeout-minutes: 90 at job level on both build-prqlc and build-prqlc-c — checked by parsing the file rather than reading the diff, since a step-level indentation slip would be silently accepted YAML: yaml.safe_load gives jobs.build-prqlc.timeout-minutes == 90 and the same for build-prqlc-c. Nothing else in the file changed.

The useful part is that the two legs the incident actually hung on ran green under the new cap on this head. In run 32240625139, nightly-release / build-prqlc (ubuntu-24.04, aarch64-unknown-linux-musl) took 3.1 min and its build-prqlc-c sibling 2.5 min — the same two jobs that ran out the 6-hour default in run 32160030161. So the capped jobs are the ones that hung, reached through the same tests.yaml → nightly.yaml → nightly-release path on this PR's own run, not by inference from the workflow files.

Every gating check on ed26f16f has now settled green — 54 success, 15 skipped, none red or cancelled, including the full nightly-release matrix that was unverified at feee3437. That's the first head on this branch where the release matrix has been confirmed rather than left pending.

I haven't touched the ~22 minutes figure in the script's header comment, agreeing with the review that it's correct as a per-invocation bound; the per-job figure for the twice-invoking aarch64 legs (~45 min of apt) is where a reader needs it, in the PR description.

Checks at ed26f16f

Job durations from repos/PRQL/prql/actions/runs/32240625139/jobs, all conclusion == "success":

3.1  nightly-release / build-prqlc    (ubuntu-24.04, aarch64-unknown-linux-musl)
2.5  nightly-release / build-prqlc-c  (ubuntu-24.04, aarch64-unknown-linux-musl)
2.8  nightly-release / build-prqlc    (ubuntu-24.04, x86_64-unknown-linux-musl)
3.8  nightly-release / build-prqlc-c  (ubuntu-24.04, x86_64-unknown-linux-musl)
1.1  build-prqlc                      (ubuntu-24.04, x86_64-unknown-linux-musl)
0.9  build-prqlc-c                    (ubuntu-24.04, x86_64-unknown-linux-musl)

All four apt-invoking musl legs are well inside both caps, so this run exercises the happy path — the mirror is healthy and the warn path isn't being hit here. The stall behaviour itself remains covered by the two reproductions at 7f73bdda and 9bd6e53c and by the stubbed-PATH replay, not by this run.

The remaining rollup entry is the review check, which is this response job itself.

[max-sixty]

this is way too complicated relative to the benefit

@tend-agent we should have a much higher bar for changes that only affect CI runner time; looks like this only happened once? please adjust our guidance

[tend-agent]

Agreed, and the guidance is adjusted: max-sixty/tend#1015.

running-in-ci's Weighing a Fix already ranked efficiency a distant third, but its examples were all about the bot's own compute, so a repo's CI runner time read as a category with no bar on it. It now says runner time is the same category and names two bars for a change whose only benefit is minutes: one incident is not a pattern (several jobs stalling on the same mirror in one afternoon, plus two reproductions on the PR fixing it, is one occurrence — which is this case), and the remedy is capped at a one-line knob such as timeout-minutes:, not a new script threaded through five call sites. The bar binds the reviewer too — this PR got four review rounds that each read the latest diff and never asked whether the whole thing was worth its weight.

One finding here that is separable from the over-build, noted in that PR but not fixed by it: a job killed at the 6-hour platform cap reports cancelled, not failure, so tend-ci-fix never sees it. That's a gap in what the bot notices rather than a CI-time optimisation, so it stands or falls on its own.

[prql-bot]

Fair close — I authored this, and the bar in max-sixty/tend#1015 would have stopped it at the first commit. I'd drafted a PRQL-side overlay saying much the same thing before that landed; closed it as redundant (#6213), since the bundled skill is the right home.

One correction on the facts, since you asked directly. It's happened more than once — though not often enough to change the verdict. Scanning tests runs on main for jobs past 60 minutes, the 360-minute cancelled shape on the musl build-prqlc* legs shows up in four runs across ~3.5 months:

run	started	jobs cancelled at 360m
26818821087	2026-06-02	1
27778015378	2026-06-18	2
32160030161	2026-08-18	3
32223676248	2026-08-19	1

The last two are ~14 hours apart and are one incident, so that's three windows, roughly monthly. Each self-corrects on a re-run and costs runner minutes only — so it still doesn't earn a retry script across six workflow files, which is the part I got wrong. Worth recording only because it means the next sweep that finds this will find real recurrence evidence, and the bar it needs to fail on is proportionality, not novelty.

Method, and one caveat

repos/PRQL/prql/actions/runs/<id>/jobs over the 60 most recent completed tests runs on main, filtered to jobs whose completed-minus-started exceeds 60 minutes. Widened to 400 runs (2026-04-30 → present) using run-level duration over 300 minutes, since a 6h job forces a 6h run; that surfaced the two June runs, whose long jobs I then confirmed individually.

Caveat: the apt-stall cause is confirmed only for the August window. The June runs match on job shape, duration and conclusion, but I didn't pull their logs — so a common cause there is inferred, not verified.