Skip to content

ci: a bot PR's CI or canary run hands off to the bot sweep as it ends - #364

Open
88plug wants to merge 3 commits into
mainfrom
claude/bot-sweep-cron-gap
Open

88plug wants to merge 3 commits into
mainfrom
claude/bot-sweep-cron-gap

Conversation

@88plug

@88plug 88plug commented Nov 7, 2026

Copy link
Copy Markdown
Owner

The bot sweep (bot-sweep.yml) is the only workflow that fills CI on bot PRs and merges them, and it ran only from its hourly cron. That cron is not hourly in practice: GitHub started bot-sweep.yml's 22 scheduled runs from 2026-11-01 14:16Z to 2026-11-07 10:04Z between 3.7 and 9.8 h apart (gh run list --workflow bot-sweep.yml --event schedule), and the watchdog's own 47 * * * * cron has the same gaps. GitHub's documentation says scheduled events can be delayed and dropped under load. The latest gap ran from 02:47Z to 10:04Z today; bot PRs with green CI sat unmerged until someone dispatched the sweep by hand, and the watchdog opened stale:bot-sweep.yml at 09:55Z.

The CI and canary runs the sweep dispatches with GITHUB_TOKEN cannot chain to it through workflow_run: a run dispatched with GITHUB_TOKEN starts no workflow_run. On 2026-11-01 the ci.yml runs dispatched on main by github-actions[bot] at 19:13Z and 23:07Z, and the release runs it dispatched at 19:38-19:42Z, started no watchdog run, while the push- and schedule-started completions around them (Auto-release AUR, Docs, the 19:38Z and 23:34Z sweeps) each started one within 3 s. So with one bot CI in flight at a time (hack/bot-ci-dispatch), each bot PR step waited a whole cron gap.

The change:

  • ci.yml (when dispatched on autobump/* or main) and canary.yml (when dispatched on autobump/*) end with job sweep. It needs every other job, runs unless the run was cancelled, has actions: write only and is continue-on-error, so it never changes the run's result. It dispatches bot-sweep.yml on main with after_run set to its own run id, retrying a failed dispatch twice. It reads GET /rate_limit first (free) and skips the hand-off when fewer than 500 of the repository's 1,000 hourly GITHUB_TOKEN requests remain: a sweep reads about 10 per open bot PR, and the watchdog and Freshness share that budget.
  • bot-sweep.yml takes the dispatch input after_run. Its ci-fill job first polls that run (every 5 s, 5 min at most) until it completes, so hack/bot-ci-dispatch and hack/bot-merge read a finished run. With no after_run (the cron, a manual dispatch) it waits for nothing; a value that is not digits fails the step. ci-fill's timeout goes from 10 to 15 min to cover the wait.
  • The cron stays as the backstop. The chain ends: each sweep dispatches at most one bot CI and one bot canary (the existing in-flight cap of 1 each), heads change only through Freshness (3 regenerations per group per UTC day), and merges are capped at 6 a day.

Push-started CI on main does not hand off; only dispatched runs do (bot-merge's post-merge CI on main, the watchdog's, and a bot PR's).

Tests, written first and failing before the change (6 failed, then 5 more for the budget check):

  • tests/ci_wiring_test.sh: job sweep in both workflows needs every other job, has the if:, permissions, continue-on-error and one identical step; bot-sweep.yml has the after_run input and the wait step before any hack/bot-ci-dispatch. Both steps run against a fake gh: the dispatch arguments, a retried 502, three failed dispatches (warning, exit 0), 499 left (no dispatch), 500 left (dispatch), an unreadable budget (dispatch); the wait with no run, a bad id, a run that completes after a failed read, and one still open after 60 reads (warning, sweep goes on). 198 passed.
  • tests/canary_spec_test.sh: the write-permission allowance names sweep, with actions: write only. 110 passed.
  • actionlint 1.7.12 (the pinned image) and shellcheck v0.11.0 at severity warning clean; tests/watchdog_test.sh 32 passed; python3 tests/ledger_drift_check.py and tests/claims_check.sh pass.

Existing bot branches keep their old ci.yml until Freshness regenerates them from main (hack/autobump-branch-reset), so the hand-off starts on bot runs from then on; a ci.yml dispatch on main exercises it right after the merge.

One side effect to know: a scheduled sweep that is pending behind a running sweep is cancelled when a hand-off queues a newer one (concurrency group bot-sweep), so stale:bot-sweep.yml, which counts scheduled successes only, can open even though sweeps ran.

The bot sweep is the only thing that fills CI on bot PRs and merges them, and
it ran only from its hourly cron. GitHub started that cron 3.7 to 9.8 h apart
on this repository (22 scheduled runs from 2026-11-01 to 11-07, none hourly),
and the CI and canary runs the sweep dispatches with GITHUB_TOKEN start no
workflow_run, so nothing followed them: with one bot CI in flight at a time,
each bot PR waited a cron gap per step.

ci.yml (dispatched on autobump/* or main) and canary.yml (dispatched on
autobump/*) now end with job sweep. After every other job, never failing the
run, it dispatches bot-sweep.yml on main with after_run set to its own run id,
unless fewer than 500 of the hour's 1,000 GITHUB_TOKEN requests remain. The
sweep's ci-fill job first waits, 5 min at most, for that run to complete, so
the evidence it reads is final. The cron stays as the backstop.

tests/ci_wiring_test.sh checks the wiring (needs lists every other job, the
if:, actions: write only, the after_run input and the wait before any
dispatch) and runs both steps against a fake gh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant