Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-28 20:16 UTC)
Window: 2026-10-27 20:16 → 2026-10-28 20:16. 417 runs in the window, 300 listed (3 pages, back to 04:51). Depot slow-start is back at its worst: 62% of Depot jobs in this slice took over 15 s to reach their first step (p90 144 s, max 329 s). The one merge_group run took 11.48 min, and green PR runs took 7.6 / 10.7.
Since the last run (16:18 → 20:16)
- Merge queue: 1 merge (Keep Go hosted pins after go mod tidy drops the zip line #1748, enqueue → merge 12.3). Its merge_group run took 11.48 created →
ci-ok(24h p50 5.08).release-readiness, a 10-second gate in front ofcoverageande2e-build, waited 307 s to boot, so they started at 5.31 instead of ~0.25. Read Go workspaces the way go builds them #1763 entered the queue at 20:10, and its run was still going at 20:16. - Critical path: the e2e fan-out (
e2e_redirect_npm_buildfinished last at 11.38). It was created only at 6.68, behind the slow gate ande2e-build. Filed #1795: moverelease-readinessto hostedubuntu-latestlike Run ci-ok on a GitHub-hosted runner #1747 did forci-ok. Replaying the 19 completed runs gives −2.8 min on the mg run, p90 −1.1, p50 ~0. - PR feedback: green full runs created →
ci-ok7.6 / 10.7 (n=12, 5.0–13.0), vs 5.91 / 7.07 at 16:18. The 13.0 had 10 e2e legs boot in 189–273 s. 3 red runs onfeat/golang-workspace-writer(its own failures; that branch still ranci-okon Depot) and 1 onfix/go-patch-delivery-e2e(hosted-e2e). - Push to main: 1 run, 5.82 (p50 3.33);
release-readinessboot 152 s,clippy117 s. - Depot boot cost: ~400 runner-min of first-step wait across 447 jobs in 4 h. Commented on #1512.
- Hosted
ci-ok(Run ci-ok on a GitHub-hosted runner #1747) queued 2–7 s throughout and stayed off the slow-start problem.
24h table
| metric | 24h |
|---|---|
| PRs merged | 23 (20 in the 13:56–14:56 burst, #1748 at 18:02, #1744 and #1648 on 10-27 evening). 0 evictions |
| Merge queue enqueue → merge | burst 9.1 / 13.6 (n=15, carried); #1748 12.3 |
merge_group created → ci-ok (success) |
burst 5.08 / 5.35 (n=20, carried); #1748 11.48 |
PR run created → ci-ok (green full runs) |
this slice 7.6 / 10.7 (n=12) |
push to main created → ci-ok |
3.33 / 4.02 (n=11, carried); 18:02 run 5.82 |
| Linux queue wait p50 / p90 | 0.02 / 0.02 (Depot, n=450, max 0.03); slow-start 62% (275/447, p50 35 s, p90 144 s, max 329 s) |
| Windows / macOS queue wait p50 / p90 | Windows 0.03 (mg, n=1); macOS 0.03 / 0.05 (10-28 nightly, carried) |
| Job-min/day (Linux / Windows / macOS) | ~3.5k / 0.1k / 0.03k (est.); since 16:18: 1,133 L in 20 runs (mg 69, push 21, PR ~1,040), 2.7 W |
Merge-queue settings (ruleset 24668460, read 20:16): build 5 / merge 5, min 1, min wait 5 min, ALLGREEN, timeout 90, SQUASH. Unchanged.
Sample (20:16 run): 300 runs listed, jobs for all 20 non-stub CI runs since 16:18, 2 PR timelines, 0 job logs (step timestamps were enough: the slow jobs were slow before their first step), ~35 REST requests. GraphQL is still blocked here.
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-28 20:16 | 23 PRs merged (1 since 16:18: #1748; 0 evictions) | since 16:18 12.3 (n=1, #1748); burst 9.1 / 13.6 (carried) | 11.48 (n=1, #1748: release-readiness booted 307 s, e2e fan-out last at 11.38); burst 5.08 (carried) |
7.6 / 10.7 (n=12 green full runs, 5.0–13.0) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 queue (n=450, max 0.03); slow-start 62% (275/447, p50 35 s, p90 144 s, max 329 s) | ~3.5k / 0.1k / 0.03k (est.); since 16:18 mg 69 L, push 21 L, PR ~1,040 L in 18 runs |
| 2026-10-28 16:18 | 23 PRs merged (20 since 12:17, burst 13:56–14:56; 0 evictions) | 9.1 / 13.6 (n=15 timelines; first 5 entries 5.3–5.8, entries 6–13 9.1–13.8 waiting for a slot) | 5.08 / 5.35 (n=20, 4.58–6.60, all first-attempt green; coverage (2) last in 12/20, Gradle hosted leg in 7/20) |
5.91 / 7.07 (n=8 green full runs, 4.90–7.88) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 queue (n=810, max 0.15); slow-start 23% (181/779 Depot, p50 6 s, p90 31 s, max 104 s) | ~3.0k / 0.1k / 0.03k (est.); since 12:17 mg 779 L / 54 W in 20 runs, push 95 L in 11, PR 325 L in 8 full runs |
| 2026-10-28 12:17 | 12 PRs merged (0 since 08:17; queue idle since 21:16) | — since 08:17 (none); 24h ≈6.8 / 9.3 (carried) | none since 21:16; 24h ≈5.1 / 7.1 (carried) | 5.13 / 7.08 (n=7 green full runs, 4.73–7.08; the 7.08 is a slow coverage (2) on a test-adding PR) |
— (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 queue (Depot, n=185, max 0.02); slow-start 3% (5/185, p50 5 s, p90 6 s, max 29 s) | ~2.3k / 0.05k / 0.03k (est.); since 08:17 PR 247 L in 17 runs (7 full, 10 stubs) |
| 2026-10-28 08:17 | 12 PRs merged (0 since 04:16; queue idle since 21:16) | — since 04:16 (none); 24h ≈6.8 / 9.3 (carried) | none since 00:16; 24h ≈5.1 / 7.1 (carried) | 4.90 / 6.25 (n=21 green full runs, 4.53–11.0; the 11.0 waited 5.2 min on a stuck cancel) | 0.03 / 0.05 (nightly, n=4) | 0.02 / 0.02 queue (Depot, n=687, max 0.03); slow-start 8% (45/582, p50 5 s, p90 11 s, max 35 s) | ~2.6k / 0.05k / 0.03k (est.); since 04:16 PR 790 L in 23 runs (incl. 129 for #1768 ×3), nightly 94 L / 53 W / 28 M (failed, bun_lockb) |
| 2026-10-28 04:16 | 12 PRs merged (0 since 00:16; queue idle) | — since 00:16 (none); 24h ≈6.8 / 9.3 (carried) | none since 00:16; 24h ≈5.1 / 7.1 (carried) | 4.83 / 6.22 (n=7 green full runs, 4.67–6.22; 1 hatch-flake re-run at 8.57) | — (no macOS jobs in slice) | 0.02 / 0.02 queue (Depot, n=300, max 0.03); slow-start 0% (0/300, p50 5 s, p90 6 s, max 10 s) | ~2.6k / 0.1k / 0.07k (est., carried); since 00:16 PR 404.6 L in 11 full runs (incl. 33.8 re-run) |
| 2026-10-28 00:16 | 12 PRs merged (2 since 20:16: #1744 20:57, #1648 21:16 on its third enqueue; 0 evictions in the slice) | since 20:16 8.8 (#1744) and 5.5 (#1648's last enqueue; 87.7 from its first); 24h ≈6.8 / 9.3 | 8.12 (#1744, rustls bump, coverage (2) last at 7.97) and 5.07 (#1648); 24h ≈5.1 / 7.1 |
6.5 / 7.6 (n=6 green full runs, 5.05–7.63; #1747's hosted ci-ok run 5.05) |
— (no macOS jobs in slice) | 0.02 / 0.02 queue (Depot, n=219, max 0.03); slow-start 36% (79/219, p50 6 s, p90 39 s, max 80 s) | ~2.6k / 0.1k / 0.07k (est., carried); since 20:16: mg 86.4 L in 2 runs, push 30.3 L in 2 runs, PR 254.5 L in 6 full runs |
| 2026-10-27 20:16 | 17 PRs merged (1 since 16:17: #1702 at 20:05; #1648 evicted twice, 19:57 and 20:18, on coverage (1) mode_migration_pypi) |
since 16:17 9.3 (n=1, #1702, which waited behind #1648's failing group); 24h 6.1 / 7.6 today (carried) | 7.08 (n=1, #1702: ci-ok waited 2.1 min for a Depot boot after the last job ended at 4.93); 24h 5.13 / 6.3 (carried) |
9.80 (n=1 green full run, 17:07, with a 2.6-min ci-ok boot) |
— (no macOS jobs in slice) | 0.02 / 0.02 queue (Depot, n=172, max 0.03); slow-start 64% (110/172, p50 31 s, p90 91 s, max 179 s) | ~2.6k / 0.1k / 0.07k (est., carried); since 16:17: 2 mg runs + 2 cancelled ≈ 125 L, push 16.2 L, PR ≈ 175 L in 3 full runs |
| 2026-10-27 16:17 | 16 PRs merged (9 today 13:07–14:58: #1738, #1734, #1735, #1740, #1733, #1739, #1736, #1742, #1741; 7 on 10-26 after 16:17) | today 6.1 / 7.6 (n=9, 5.2–8.5, 0 evictions); 10-26 post-burst 9.1 / 10.2 | 5.13 / 6.3 (n=9, 4.75–6.68; coverage (2) last in 7/9) |
green full runs 4.87 (n=2, both 4.87; 2 more red at 4.68 / 6.40, PDM unearth before #1738 reached the branch) |
— (no macOS jobs in slice) | 0.02 / 0.02 (Depot, n=441, max 0.22); slow-start 16% of mg jobs (36/231) | ~2.6k / 0.1k / 0.07k (est.); since 12:17: mg 343 L / 24.5 W in 9 runs, push 79.7 L in 8 runs, PR 147.4 L in 4 full runs |
| 2026-10-27 12:17 | 25 PRs merged (0 since 18:54; queue idle 17.4 h) | — since 08:16 (none); burst 43.9 / 50.0, post-burst 9.1 / 10.2 (carried) | none since 18:54; 6.55 / 9.14 (carried) | 6.1 (n=4 full runs, 5.68 / 6.07 / 6.12 / 6.38; 2 delayed ~1.3 min behind a superseded run; coverage (2) last in 3/4) |
— (no macOS jobs in slice; 0.04 / 0.05 carried) | 0.02 / 0.02 (Depot, n≈230, max 0.35) | ~2.9k / 0.08k / 0.07k (est.); since 08:16 PR 285 L in 8 full runs (4 red on PDM unearth) |
| 2026-10-27 08:16 | 25 PRs merged (0 since 18:54; queue idle 13.4 h) | — since 04:16 (none); burst 43.9 / 50.0, post-burst 9.1 / 10.2 (carried) | none since 20:16; 6.55 / 9.14 (carried) | 4.72 (n=3 full runs, 4.63 / 4.72 / 4.92; 1 red on upstream PDM break) | 0.04 / 0.05 (nightly, n=4) | 0.02 / 0.02 (Depot, n=93 PR + 191 nightly, max 0.05); slow-start 0% (0/93) | ~2.9k / 0.08k / 0.07k (est.); since 04:16 PR 106 L in 3 full runs (36.0, 34.9, 35.2); nightly 10-27 254 L / 80 W / 65 M (failed, PDM unearth) |
| 2026-10-27 04:16 | 25 PRs merged (0 since 18:54; queue idle 9.4 h) | — since 00:16 (none); 24h burst 43.9 / 50.0, post-burst 9.1 / 10.2 (carried) | none since 20:16; 24h 6.55 / 9.14 (carried) | 4.70 (n=3 full runs, 4.62 / 4.70 / 6.43; coverage (2) last in all 3) |
— (no macOS jobs in slice; 0.05 carried) | 0.02 / 0.02 (Depot, n=93, max 0.03); slow-start 3% (3/93) | ~3.8k / 0.17k / 0.03k 24h (carried); since 00:16 PR 111 L in 3 full runs (36.4, 35.1, 39.5) plus 3 stubs and 5 audit runs |
| 2026-10-27 00:16 | 25 PRs merged (0 since 20:16; queue idle since 18:54) | — since 20:16 (none); 24h burst 43.9 / 50.0, post-burst 9.1 / 10.2 (carried) | none since 20:16; 24h 6.55 / 9.14 (carried) | 6.80 (n=3 full runs, 4.68 / 6.80 / 9.02; the 9.02 is a DNS re-run) | — (no macOS jobs in slice; 0.05 carried) | 0.02 / 0.02 (Depot, n=108, max 0.03); slow-start 6% (6/108) | ~3.8k / 0.17k / 0.03k 24h (carried); since 20:16 PR 112 L in 3 full runs (37.3, 37.1, 72.4 incl. re-run) plus 1 cancelled run (27.1) and 6 stubs |
| 2026-10-26 20:16 | 25 PRs merged (7 since 16:19: #1722, #1719, #1730, #1715/#1723/#1724 in one group, #1731; queue idle since 18:54) | since 16:19 9.1 / 10.2 (n=7, 6.1–10.3, 0 evictions); 24h n=25, p90 ≈50 (burst carried) | 6.55 / 9.14 24h (n=37); since 16:19 6.57 (n=7, 4.63–9.65) | 24h full runs 5.57 / 8.54 (n=27); since 16:19 5.72 / 8.55 (n=11, max 12.77 on 29/31 slow hosts) | 0.05 / 0.05 (nightly, n=4) | 0.02 / 0.02 (Depot, n=2,546, max 0.95); slow-start 38% of mg jobs 24h (547/1455), 53% since 16:19 (114/217) | ~3.8k / 0.17k / 0.03k (mg 2,339 L / 124 W in 47 runs; PR 1,192 L; push 171 L; nightly 93 / 50 / 30) |
| 2026-10-26 16:19 | 18 PRs merged (all 15:10–16:17, first burst since 10-24 02:54; 1 eviction, #1719 dequeued) | 43.9 / 50.0 (n=18; first 3 entries 5.1–7.7; max 67.1 #1706) | 6.07 / 9.03 (n=30; 2 at 20.5 / 22.3 on hung rust-cache restores) | 24h ≈4.7 / 8.2; since 12:33 6.4 / 8.5 (n=10 full runs during the burst, 4.57–9.68) | 0.05 / 0.05 (nightly, carried) | 0.02 / 0.02 (Depot, n=1,760, max 0.95); slow-start 36% (326/900 mg jobs) | ~3.0k / 0.16k / 0.03k (mg 1,956 L / 106 W in 40 runs, 54% discarded; PR 497 L since 12:33; push 97 L; nightly 93 / 50 / 30) |
| 2026-10-26 12:33 | 0 PRs merged (queue idle 57.6 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.78 (n=1 green full run since 08:18; 24h 4.58 / 4.77, n=12) | — (no macOS jobs in slice; 0.05 carried) | 0.02 / 0.02 (Depot, n=31, max 0.02) | ~0.51k / 0.05k / 0.03k (since 08:18 PR 34.8 L in 1 full run plus 4 stub and 4 audit runs; nightly 10-26 still in window) |
| 2026-10-26 08:18 | 0 PRs merged (queue idle 53.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.57 (n=1 green full run since 04:16; 24h 4.58 / 4.75, n=13) | 0.05 / 0.05 (nightly, n=4) | 0.02 / 0.02 (Depot, n=122, max 0.03) | ~0.55k / 0.05k / 0.03k (since 04:16 PR 33.9 L in 1 full run plus 4 stub and 4 audit runs; nightly 10-26 92.5 L / 50.2 W / 30.4 M (failed, Windows shard 1); the 10-25 nightly rolled out) |
| 2026-10-26 04:16 | 0 PRs merged (queue idle 49.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.55 (n=3 green full runs since 00:16, 4.53–4.57; 24h 4.58 / 4.77, n=13) | — (no macOS jobs in slice; ≤0.03 carried) | 0.01 / 0.01 (Depot, n=93, max 0.01) | ~0.55k / 0.05k / 0.03k (since 00:16 PR 106.7 L in 3 full runs (34.95, 35.63, 36.15) plus 4 stub and 6 audit runs) |
| 2026-10-26 00:16 | 0 PRs merged (queue idle 45.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.65 (n=3 green full runs since 20:15, 4.48–4.73; 24h 4.58 / 4.77, n=11) | — (no macOS jobs in slice; ≤0.03 carried) | 0.02 / 0.02 (Depot, n=93, max 0.02) | ~0.48k / 0.05k / 0.03k (since 20:15 PR 108 L in 3 full runs (35.5, 35.8, 36.7) plus 4 stub and 7 audit runs) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~14 | #1670 (settings) 90-min check timeout let a stuck check freeze main for 77 min | ~46 min sooner recovery per stuck-check incident | No incident since. Needs a repo admin. |
| ~11 | #1248 (settings) merge-queue build concurrency | ~46 entry-min of slot wait in the 10-28 15-PR burst, up to 8.3 min per entry | Commented 16:18. Needs a human decision. |
| ~4 | #1512 (settings) slow / lost Depot hosts | ~2–6 min per merge under slow starts | Commented 20:16: slow-start 62%, p90 144 s, max 329 s; mg 11.48, PR 7.6 / 10.7. Needs Depot. |
| ~1 | #1720 (coverage tradeoff) Windows-only regressions reach main | avoids ~0.2–0.3k job-min of fix churn per regression | agent:needs-human; waiting on a maintainer. |
| ~0.8 | #1795 new release-readiness on hosted ubuntu-latest |
−2.8 min on today's slow mg run, p90 −1.1 per run, ~0.1k Depot min/day | Filed 20:16. S effort; ready for the implementer. |
This run (20:16): filed #1795, commented on #1512, closed none.
Open ci-perf / ci-janitor PRs: none.
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
#1747 ci-ok on a GitHub-hosted runner (#1745) |
10-28 14:02 | Delivered. In 31 runs with hosted ci-ok since, ci-ok finished 0.07–0.12 min after the last job (queue 1–3 s), while Depot slow-start was 23% (p90 31 s, max 104 s). The 8 Depot ci-ok runs in the same burst, before the change reached them, took 0.12–0.17. Before #1747, ci-ok waited up to 2.6 min in slow-start episodes. #1745 is closed. |
| #1760 fix ETXTBSY flake in the hatch version-gate test (ci-janitor) | 10-28 14:02 | 0 hatch failures in 39 CI runs since (20 mg, 11 push, 8 PR). Too early to call. |
| #1775 nightly vlt migration tests for eject (#1774, ci-janitor) | 10-28 14:01 | The next nightly (10-29) will show it. |
| #1780 retry the Deno install in the deno test image (ci-janitor) | 10-28 14:56 | Robustness; 0 Deno install failures since (few runs). |
| #1734 tiny e2e capstones on 3 runners instead of 9 (#1732) | 10-27 13:13 | Delivered. merge_group runs went from 50 to 44 jobs and from 36.5 to 34.2–35.1 Linux job-min per run (n=6 that add no tests), about −1.5 per run (the PR runs had shown −1.3 to −1.7). Capstones are off the critical path; mg ci-ok p50 5.13. |
| #1740 retry yarn classic e2e installs on transient DNS (ci-janitor) | 10-27 13:17 | 0 yarn-classic failures in 27 full runs since (17 + 10 this slice, all green). Holding. |
#1738 pin unearth<0.19 for the PDM e2e |
10-27 13:13 | Delivered. 0 PDM failures on main-based runs since; before, 4 of 8 PR runs (08:16–12:17) were red on it. |
| #1730 cap hung rust-cache restores at 5 min (ci-janitor) | 10-26 18:12 | Too early to call: 0 hangs in ~10 runs since. Before: 3 hangs of 14.2–15.6 min in one 72-min burst (~330 queue entry-min). |
| #1708 stop late draft runs cancelling real PR CI (ci-janitor) | 10-26 15:10 | Holding: 0 draft-run cancels in the 20 PR CI runs after 15:10 (the 1 cancelled PR run was a superseding push on #1730). |
| #1716 Pipenv pylock test follows uv (ci-janitor) | 10-26 15:14 | Robustness; 0 Pipenv leg failures in 30 merge_group runs since. |
| #1710 Windows nightly yarnrc path (#1689) | 10-26 15:14 | Delivered. In the 10-27 nightly, test (windows-latest, 1) passed in 21.32 min and shard 2 in 17.22 (run), after the 10-24, 10-25 and 10-26 nightlies failed on shard 1. The nightly is still red, but only on the unrelated PDM unearth break. |
| #1704 shard coverage across two runners (ci-perf, self-picked from dashboard) | 10-24 02:38 | Delivered, beat its estimate. merge_group ci-ok with the shards: 4.47, 4.48, 4.93 (n=3; run, run, run) vs 6.18–6.47 after #1697 and before #1704: about −1.8 min per merge (expected −1.5). Shards take 3.78–4.62 min (was one 5.88–6.17 job). Cost +0.7–1.4 Linux job-min per mg run (35.9–36.6 vs 35.2), less than the +4 expected. Push to main ci-ok 3.03 / 3.08 (was 3.12–3.27). PR push→ci-ok since 04:16: 4.45–4.95 (n=3) vs 6.37 p50 before. |
| #1697 Gradle agent suites in their own e2e leg (self-picked from dashboard) | 10-23 22:18 | Delivered. The Gradle legs now take 2.82–2.98 and 1.77–1.97 min (was 4.58–4.92 combined, 7.40 on a slow host). ci-ok in the 3 merge_group runs that include it: 6.18 / 6.23 / 6.47 vs 6.50–7.58 on fast hosts before (about −0.3 to −1 min). Gradle is off the critical path; coverage now sets it. |
| #1682 start coverage and e2e-build without waiting on clippy (#1681) | 10-23 14:54 | Delivered. In merge_group runs, coverage now starts at 0.20–0.30 min (was 1.43–1.47). ci-ok with #1682: 6.50, 6.73, 7.35, 7.58, 10.03 (n=5; p50 7.35, the 10.03 on 100%-slow hosts) vs 7.53–8.15, p50 7.76 before. On fast hosts ~1.1 min saved against 1.3 expected. |
| #1686 Windows nightly path asserts (ci-janitor) | 10-23 14:38 | Partly delivered. In the 10-24 nightly Windows shard 2 passed, but shard 1 still failed on -p socket-patch-core --lib (job). Follow-up #1710 (#1689) is open and green in dispatch. |
| #1690 gem old-Bundler legs (follow-up to #1687) | 10-23 14:54 | Delivered. No gem leg failures in the 10-24 nightly (its only failure was Windows shard 1). |
| #1692 compatibility workflows off Node 20 pins (ci-janitor) | 10-23 14:47 | Maintenance; no CI-time effect expected. |
| #1673 retry the yarn-classic corepack probe (#1671) | 10-23 00:13 | Before: 2 of 40 tests failed on one DNS timeout and evicted #1653 (~94 job-min). After the merge: 0 yarn-classic failures in 28 jobs (7 full runs through 04:16, then 21/21 green since). Holding. |
| #1663 fix uv settings test on Windows nightly (ci-janitor) | 10-22 19:29 | Did not make the Windows nightly green. The 10-23 nightly failed both Windows shards (cli --test user_patches, core --lib, path-form asserts). Follow-up #1686 passed both shards in its dispatch run. |
| #1652 keep transport cause in registry errors | 10-22 17:38 | Diagnostics; no CI-time effect expected. |
| #1654 retry crates.io in coverage's cargo e2e | 10-22 17:12 | 0 infra coverage failures in 53 merge-group + 19 push runs + 10 PR runs since 04:16. The one 04:16–08:16 failure was a PR's own new test. The 10-22 21:25 PR failure is carried. Holding. |
| #1643 fix update-lock flake | 10-22 17:09 | 0 recurrences since. Too early to call. |
| #1647 pin floating toolchains (#1645) | 10-22 17:03 | No toolchain-drift red main since (7 h). Needs a week to judge. |
| #1659 runner image's PHP in composer e2e | 10-22 17:03 | 0 Setup PHP failures since (was 1 re-run on 10-21). Composer leg p50 unchanged. |
#1666 skip cargo shapes on failed --precise pin |
10-22 17:01 | Robustness; no CI-time effect expected. |
Method notes
-
GraphQL and
search/issuesare blocked in this environment. Merge-queue state comes from REST timelines of merged PRs. "Enqueue → merge" uses the PR's firstadded_to_merge_queueevent in the window. "Slot wait" = enqueue → the PR's first merge_groupCIruncreated_at. -
Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Slow start" = jobstarted_at→ first stepstarted_atover 15 s, Depot jobs only. -
Critical path = merge_group run
created_at→ci-okcompleted_atof the latest attempt. -
"Discarded" merge_group runs are every
CIrun except each merged PR's last one. -
Hang cost (~330 entry-min) = for each entry held, its merge time minus when its own group run finished (or, for the Fix Deno global npm tools missed by -g (#444) #1719 hang, minus ~15:30, when the held groups' runs had finished).
-
Run 10-26 20:16: 300 of 313 runs listed (3-page cap), and jobs for all 108
CIruns. "PR full runs" leave out 1-job stub runs (ci-okin under 1 min). -
Runs 10-27 00:16, 04:16, 08:16 and 12:17: the queue was idle, so the 24h merge and merge_group numbers are carried from 20:16. Only the new 4-hour slice was fetched each time. The 12:17 run dropped the 10-24 12:16 history row (18-row cap).
-
Run 10-27 16:17: 197 runs in the window (2 pages). Jobs fetched for every
CIrun since 12:17. The 24h merge numbers combine the 10-26 post-16:19 slice (carried) with today's 9 merges. The oldest history row (10-24 16:16) was dropped (18-row cap). -
Run 10-27 20:16: 157 runs in the window. Jobs were fetched for every
CIrun since 16:17, plus all merge_group/pushCIruns in the window, to getci-okstart gaps (started_at→ first step). The oldest history row (10-24 20:16) was dropped (18-row cap). -
Run 10-28 00:16: 190 runs in the window. Jobs were fetched for every non-stub
CIrun since 20:16. The 24h enqueue→merge p50 combines the 9 carried 13:07–14:58 values (5.2–8.5, p50 6.1) with Build(deps): Bump zod from 3.25.76 to 4.6.6 in /npm/socket-patch #1702, Build(deps): Bump rustls from 0.23.45 to 0.23.46 #1744 and fix: vendor hosted patches directly from lockfiles #1648 (last enqueue); it's approximate. The oldest history row (10-25 00:16) was dropped (18-row cap). -
Run 10-28 04:16: 227 runs in the window (3 pages). No merge_group or push
CIruns since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 11 completed full PRCIruns since 00:16. The oldest history row (10-25 04:16) was dropped (18-row cap). -
Run 10-28 08:17: 283 runs in the window (3 pages). No merge_group or push
CIruns since 00:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 24 non-stubCIruns since 04:16. The oldest history row (10-25 08:15) was dropped (18-row cap). -
Run 10-28 12:17: 288 runs in the window (3 pages). No merge_group or push
CIruns since 10-27 21:16, so all 24h merge-queue numbers are carried. Jobs were fetched for the 17CIruns since 08:17. The oldest history row (10-25 12:15) was dropped (18-row cap). -
Run 10-28 16:18: 300 of 369 runs listed (3-page cap; the list reaches back to 10-27 22:41, so the 10-27 20:05–21:16 merges are carried). Jobs were fetched for all 39 non-stub
CIruns since 12:17 (20 mg, 11 push, 8 PR) and timelines for 15 of the 20 merged PRs (~60 REST requests). Enqueue → merge p50/p90 is over those 15. The oldest history row (10-25 16:15) was dropped (18-row cap). -
Run 10-28 20:16: 300 of 417 runs listed (3-page cap, back to 04:51). Jobs were fetched for all 20 non-stub
CIruns since 16:18 and timelines for Keep Go hosted pins after go mod tidy drops the zip line #1748 and Read Go workspaces the way go builds them #1763. The CI perf: release-readiness — 10-second gate waits up to 5 min for a Depot boot in front of coverage and e2e (~1–3 min per run in slow-start periods) #1795 replay keeps each run's measuredclippyend and per-job boots and putsrelease-readinessat its own work time + 6 s; jobs needingclippytoo move only as far asclippyallows. The oldest history row (10-25 20:15) was dropped (18-row cap).
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code