ci(2284): a Bugbot gate derived from the findings, not from a verdict we do not control

[LukasWodka]

Closes part of tracebloc/backend#2284. Lands the mechanism for a Bugbot gate that is derived from findings rather than from a verdict we don't control, plus its test and mutation tiers. It does not yet arm the gate on any branch — see What is armed below, which is the honest answer and the reason the diff is this size and no larger.

Step 1 first: option 1 was measured, and it is worthless

The ticket said option 1 ("add Cursor Bugbot to the required contexts") was "worth measuring first since it may be nearly free". It was measured. It is not nearly free — it is worthless on both horns of a dichotomy.

Bugbot's conclusion vocabulary has two values, and failure is not one of them. Measured 2026-08-22 over 25 per-commit runs (client#779/#782/#787/#789) and the merged heads of 48 PRs across 8 repos:

conclusion	findings on that commit
success	0, every time (9/9 sampled per-commit)
neutral	findings in 13/16 sampled per-commit — and 0 findings in the other 3

So neutral is what Bugbot emits when it has findings, and it is not even a reliable findings signal in the direction that would help.

• If neutral satisfies a required context — GitHub's documented behaviour — requiring it gates nothing, because failure never occurs. This half could not be settled from org history: a scan of the last 25 merged PRs per repo across 7 repos found zero cases of any required context ever concluding neutral or skipped, so there is no counter-example either way and the claim rests on the documented semantics. Stated rather than glossed.
• If neutral does not satisfy it, requiring the context permanently bricks every PR that ever received a finding. Bugbot re-runs only on a push or an explicit bugbot run, so resolving a finding — the org's own sanctioned disposition, "file the finding, link the ticket on the thread, resolve, ship" (release-train/CLAUDE.md) — can never turn the check green again. Two prod promotions merged on 2026-08-21 in exactly that state: client#786 and frontend-app#863, both staging -> main, both Cursor Bugbot = neutral on the merged head, each carrying one resolved Medium. Promotion PRs may not be pushed to by policy, so under option 1 those two had no route to green at all.

Both horns are bad outcomes. Option 1 is not taken, and that negative result is why this PR exists.

Two supporting measurements:

• The check name does not vary: exactly Cursor Bugbot, from the App with slug cursor, in all 16 train repos.
• It is required nowhere. Read across client / backend / cli / release-train / frontend-app / tracebloc-engine / docs / design-system / .github, on develop + staging + prod, from branches/{b}/protection and from every repo's rulesets. Zero hits. The ruleset read was verified non-empty rather than assumed: 12 rulesets exist across those repos, carrying only pull_request (merge-method) and tag creation/update/deletion rules — no ruleset anywhere carries a required_status_checks rule, so classic protection is the whole picture. Confirms the ticket's finding fleet-wide, not just on client.

What the gate claims — and what it deliberately does not

(A) The load-bearing claim: a terminal Bugbot check run must exist on the PR's current head sha. Nothing requires that today, and its absence is the ticket's quiet failure mode stated precisely — "the PR merges with a High that nobody re-checked after the last push" is Bugbot not having reviewed the last push. An absence never approves.

This is a ratchet, not a new backlog: measured on the merged heads of 48 PRs across 8 repos, 48/48 already had a completed Cursor Bugbot run on the exact merged head. Green today by measurement — arm while green — and what it stops is the case that has not happened yet.

(B) Severity is parsed and reported, and an open finding at/above the threshold fails the gate.

What (B) does not add, said in the file so nobody later "fixes" the omission. required_conversation_resolution is true on every branch measured (client/backend/cli/docs/design-system/frontend-app develop, plus client and backend staging + main). That already blocks a merge on any open thread, including a Low. So (B) is not stricter than protection for open threads and is not sold as if it were. What it buys is the name: GitHub refuses such a merge "naming neither cause", which release-train/CLAUDE.md records as a real diagnosis cost. A named check saying "High finding open" is the whole of (B)'s value.

Derived, never restated

what is matched	matched on
the check run	the producing App slug (cursor), never the display name — so a Bugbot rename cannot blind the gate. Pinned by a mutation, and by a test asserting a renamed check still counts and a same-named check from another app does not.
a finding	Bugbot's own <!-- BUGBOT_BUG_ID: ... --> marker — how Bugbot itself distinguishes a finding from any other comment it makes
the severity	Bugbot's own **<Level> Severity** line

The rank order (low < medium < high < critical) is the one thing that must be declared — there is nowhere to read it from — which is exactly why an unrecognised severity token is a failure, not a harmless default. Observed vocabulary across 29 sampled findings is High and Medium only; Bugbot may emit others.

Fail closed; "cannot tell" is a finding

Every one of these exits nonzero rather than reporting clean: the GraphQL read failing; a non-JSON response; an errors[] payload at exit 0; pullRequest: null; a PR reporting no commits; last-commit ≠ headRefOid; a truncated rollup; a truncated thread page; a missing totalCount on either; an unrecognised severity; a threshold outside the declared rank; no terminal Bugbot verdict once the wait budget is spent.

Truncation follows bricked-prs.py's precedent exactly — same PAGE_CAP = 100, and >= rather than >, because at exactly 100 there is no way to tell 100 contexts from 140. It matters in the direction this gate is about: a context lost to pagination makes Bugbot look absent on a head it reviewed. A dedicated case asserts truncation is refused even when a Bugbot run is visible in the page, so the guard is not dead code on exactly the heads that have it.

The one deliberate fail-open is a draft PR — it cannot merge, and Bugbot's behaviour on drafts is not this gate's business. Same treatment and reasoning as bricked-prs.py. Pinned in both directions: a mutation that treats every PR as a draft, and one that removes the exemption; plus a case asserting the same PR not marked draft fails.

Why it waits

Claim (A) is unsatisfiable the instant a push lands, so the gate polls. Measured started_at -> completed_at over 40 Bugbot runs: min 9s, p50 164s, p90 332s, max 635s. Default budget 900s ≈ 1.4× the observed max. The wait runs concurrently with Bugbot's own work, so it adds latency only when Bugbot is slower than usual. timeout-minutes sits above the script's budget on purpose, so the script reports why it gave up instead of the runner killing it silently.

Test evidence

$ make check                              # the repo's pre-push tier
==> check: green (gitleaks runs in 'make check-all')
EXIT=0                                    # actionlint: 0 findings; ruff/shellcheck/
                                          # house-rules/action-pins/mint-scope/mutations-dry all clean

$ make selftests
80 passed, 0 failed                       (selftests-cover + every suite)
bugbot-gate-selftest: 54 assertions, all passed

$ python3 scripts/tests/bugbot-gate-mutations.py
26 mutation(s): 0 stale, 0 uncaught

$ actionlint -no-color -oneline -shellcheck shellcheck   # clean
$ ruff check --select E4,E7,E9,F                          # All checks passed

Every anchor is asserted to match exactly once, so an inert mutation fails the run rather than passing as coverage.

Two real defects the mutation tier found in its own suite

Worth calling out, because both are the failure mode this tier exists for — and neither was visible in a green log.

1. A stale bytecode cache made caught mutations report as uncaught. The suite imports the gate by path, so CPython wrote scripts/__pycache__. A .pyc is revalidated on the source's (mtime-to-the-second, byte size) — and several mutations lengthen the file by exactly the same 10 bytes (they all disable a guard by prefixing False and ). Run back to back inside one second, one mutation's bytecode was served to the next run. Fixed at the source (sys.dont_write_bytecode = True in the suite) and at the harness (-B + PYTHONDONTWRITEBYTECODE + unlinking any pre-existing cache), belt and braces, with the reasoning written down.
2. expect_unreadable accepted any Unreadable, so three cases passed for the wrong refusal — CLAUDE.md rule 10. Unreadable is raised from nine places; with a guard in fetch() removed, the read fell through to a different refusal one line down and the type still matched. The helper now requires a because= substring, and those three mutations are caught by "refused, but for the wrong reason".

What is armed, and what is not — precisely

Armed by this PR: the test tier only. Both suites run inside make selftests / make mutations, and selftests is already a required status check on .github's develop, staging and main (verified: gh api repos/tracebloc/.github/branches/develop/protection lists selftests). That is the same trick selftests.yml uses to arm a guard with no branch-protection edit.

Not armed: the gate reporting on PRs. No caller ships here, and that is a hard constraint rather than a choice. Every caller in this org pins tracebloc/.github/...@main (RFC-BACKEND-1405 Q3), and .github's main is currently 45 commits behind develop and does not carry bugbot-gate.yml. A caller landed alongside this file would reference a reusable main does not have and die with a startup_failure — a red check on the very PR introducing a gate. code-quality-caller.yml records the identical sequencing for its action-pins input.

So arming is two follow-ups, in this order:

1. Once this file is on main: add bugbot-gate-caller.yml to .github (this repo first — one repo, reviewable).
2. Once that caller has been observed green on real PRs: add the bugbot / review context to .github's branch protection.

Never the reverse — backend#1976's lesson is that a contract claiming a context reality lacks is itself a finding, and a required context no workflow reports leaves every PR waiting forever (client#665, pii-gate/pii-check). No branch protection or ruleset was changed by this PR. repo-inventory.yml is deliberately untouched for the same reason: the contract must not claim what reality lacks.

The caller for follow-up 1 is written and verified (actionlint clean); it is held back only by the @main constraint.

Known gap: the trigger this wants and cannot have

Resolving a Bugbot thread is not a pull_request event, so pull_request_review_thread: [resolved] is the right trigger for the severity half. actionlint 1.7.12 — a required check here, run with no config file and no -ignore on purpose — does not know that event name and rejects the workflow. Verified against the pinned binary: zero occurrences of review_thread in it. Adding the trigger when actionlint learns the event is a follow-up.

Until then the failure message says "resolve the thread, then re-run this check" — explicitly not "push a commit", because that is the habit a stale gate teaches.

Also worth a follow-up (not fixed here, out of scope)

make selftests fails if a __pycache__ directory exists under scripts/tests/, even though .gitignore ignores it — so any developer who imports a module from there by path breaks the fast tier until they delete it. Pre-existing; this PR avoids adding to it rather than fixing it.

---

🤖 Generated with Claude Code

---

Note

Medium Risk
New merge-gate logic that will eventually block PRs, with GraphQL reads of checks and review threads. This PR only arms the test/mutation tier; no caller or branch-protection change yet.

Overview
Lands the mechanism for turning Cursor Bugbot from advice into a named gate, without requiring Bugbot’s own check (which never concludes failure and would either gate nothing or brick resolved findings).

The reusable bugbot-gate.yml plus scripts/bugbot-gate.py poll until a terminal check from app slug cursor exists on the current head, then fail on open finding threads at or above min-severity (default high). Bugbot’s conclusion is reported, not used. Drafts pass. Truncation, unknown severities, and unreadable GraphQL all fail closed. Sibling checks from the same app cannot stand in for the review.

Not armed on PRs: no caller and no branch-protection edit (callers pin @main). Tests are armed: hermetic selftest + 26 mutations are wired into make selftests / make mutations.

^{Reviewed by Cursor Bugbot for commit b03dfd8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

This PR is currently a live instance of the finding it fixes

Snapshot taken just now, while writing this:

required context on .github/develop	state
actionlint	SUCCESS
gate	SUCCESS
quality / action-pins	SUCCESS
quality / gitleaks	SUCCESS
quality / house-rules	SUCCESS
quality / ruff	SUCCESS
quality / shellcheck	SUCCESS
selftests	SUCCESS

Cursor Bugbot: IN_PROGRESS — and not in that list.

So every required context is satisfied and this PR is one approval away from merging while Bugbot has not yet looked at the diff at all. Nothing is red, nothing is waiting, and no signal anywhere says a review is still outstanding. That is exactly claim (A): a terminal Bugbot run must exist on the current head, and it is the half of backend#2284 that no existing mechanism covers — required_conversation_resolution cannot block on a thread Bugbot has not written yet.

Not an argument to merge this faster; just the cheapest available demonstration that the gap is real rather than theoretical.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5f038a1. Configure here.}

[saqlainsyed007]

Reviewed the gate script, the reusable workflow, and both test tiers against the code. This is the mechanism for a Bugbot gate derived from findings (backend#2284), and it is exceptionally careful.

The negative result is load-bearing and measured: option 1 ("just require the Cursor Bugbot context") is worthless on both horns — Bugbot's vocabulary is success/neutral and never failure, so if neutral satisfies a required context the gate is inert, and if it doesn't it permanently bricks every PR that ever got a finding (two prod promotions merged in exactly that state). So the gate reads findings itself.

What I verified in scripts/bugbot-gate.py:

• Matched on machine-readable markers, never display names. The check is found by the producing app slug (cursor), so a Bugbot rename can't blind it; a finding by Bugbot's own BUGBOT_BUG_ID marker; severity by its **<Level> Severity** line. When the app publishes more than one check, the canonical name disambiguates and an unresolvable tie is REFUSED, not guessed — closing the Autofix-sibling path (measured unreachable today, closed anyway).
• Fail-closed on every "cannot tell" (rule 3): GraphQL failure, pullRequest: null, an inconsistent head, a truncated rollup or thread page, a query that stops asking totalCount, an unknown severity token, and no terminal run after the wait budget all exit nonzero rather than reporting clean. The truncation test is totalCount > len(nodes) — correctly NOT >= cap (an exactly-full page is complete), the exact bug Bugbot caught on this PR, now regression-guarded in the mutation tier.
• Honest and narrow. The verdict vocabulary is read by main, not sniffed from the report prose (rule 9); only PENDING is waitable; the wait is sized from measured Bugbot latency and runs concurrently; and it says plainly that (B) is not stricter than required_conversation_resolution for open threads — its value is naming the finding, not adding strictness. The draft fail-open is the one deliberate exception, documented and safe (a draft can't merge; re-runs on ready_for_review).

The tests clear the house bar: the selftest runs in the required selftests context (green), and the mutation tier proves those assertions are load-bearing — 30 mutations that edit the real gate and re-run the real suite (rule 9), each anchor exact-once, the bytecode cache disarmed (with the documented 10-byte-collision reasoning), always restored, and "caught" counted only when the suite actually reported (a harness-breaking traceback is not coverage). It even re-injects the >= PAGE_CAP regression so it can't come back quietly.

Correctly landed-not-armed: bugbot-gate.yml is workflow_call-only and wired into no branch, which is the honest land → observe green → arm sequencing. Green, no open threads. Approving.

[LukasWodka]

@saqlainsyed007 thank you — that is a genuinely thorough read, and you caught the two things that matter most about this PR: that the negative result on option 1 is load-bearing, and that the truncation test is totalCount > len(nodes) rather than >= cap. Both are exactly right.

One thing you could not have known, and it is not about your review: this approval cannot unblock the PR. saqlainsyed007 has read permission on tracebloc/.github, and GitHub only counts approvals from users with write toward a required-review rule. So the review is recorded as APPROVED while reviewDecision stays REVIEW_REQUIRED and mergeStateStatus stays BLOCKED.

Measured just now on this repo:

user	permission
saadqbal	write
LukasWodka	admin
saqlainsyed007, aptracebloc, waqaskhanroghani, shujaatTracebloc	read

Everything else on this PR is satisfied — all 8 required contexts reported and green (actionlint, gate, selftests, and the five quality / *), zero unresolved threads, no required context missing from the rollup. The review requirement is the only thing outstanding.

So @saadqbal's approval is needed to merge, and this is not a re-review request: Saqlain's review stands and I have nothing to add to it.

Filing the access asymmetry separately — a reviewer whose approval is structurally inert is a silent waste of exactly the effort above, and nothing in the UI says so.

[saadqbal]

Approving. The measured negative result on option 1 is the valuable half, and I corroborated its central claim independently — on a case this PR doesn't cite.

neutral really is what Bugbot emits when it has findings. client#797 currently carries two OPEN Bugbot Mediums, and the raw check on its head is app=cursor, name='Cursor Bugbot', status=COMPLETED, conclusion=NEUTRAL. That's your dichotomy in the wild: not failure, so requiring the context gates nothing under documented semantics — and if neutral didn't satisfy, that PR would be permanently bricked. It also confirms the two supporting measurements (exact check name, cursor slug) on a repo you sampled.

Worth adding for anyone reading the tables: gh pr checks prints neutral as skipping, so anyone eyeballing the CLI will read your neutral rows as "skipped" and conclude Bugbot didn't run. Your numbers come from conclusions rather than the CLI label, so they're right — but that display quirk is why this is easy to get wrong by hand, and it's arguably worth a line in the header.

I went looking for two holes and found both already closed, which is why I'm approving rather than asking:

• The draft→PASS exemption is only safe if callers trigger on ready_for_review, since GitHub's default pull_request types don't include it and this file is workflow_call-only. :54-59 states exactly that and calls it load-bearing rather than boilerplate.
• Resolving a thread isn't a pull_request event, so the severity half could have reproduced option 1's brick. It doesn't, and the reason is the important one: this gate clears on a check re-run, which needs no push — whereas Bugbot itself only re-runs on a push or bugbot run, which is exactly why option 1 had no route to green on a promotion. :541-546 puts that in the failure message and explicitly tells the reader not to push an empty commit. That's the difference between the two designs, and it's the right one.

The fail-closed shapes are all in the right direction: unknown severity raises Unreadable ("an unrecognised severity is not a harmless one"), a missing check on head is PENDING with "an absence never approves", and findings are anchored on Bugbot's own BUGBOT_BUG_ID marker and **<Level> Severity** line rather than on prose — I've been reading those exact markers in threads all week, so the anchors are real.

And I'd single out (B)'s honesty: saying outright that it is not stricter than required_conversation_resolution for open threads, and that what it buys is the name GitHub refuses to give, is a much more useful claim than overselling it. The diagnosis cost of "blocked, cause unnamed" is real — I've spent time on it in release-train.

[LukasWodka]

/fr-pass