Auto-apply high-confidence frontend-triage results - #6439
Open
msujaws wants to merge 1 commit into
Open
Conversation
msujaws
force-pushed
the
frontend-triage-auto-apply
branch
2 times, most recently
from
August 6, 2026 19:44
b7f1425 to
1095882
Compare
frontend-triage produces a root-cause analysis and fix plan and records it as
Bugzilla actions, but nothing applied them: a human had to click Apply in the
hackbot UI. Close that loop for the results the agent is confident about.
`AgentSpec` gains `auto_apply_confidence` — the `findings.confidence` levels whose
actions may be applied unattended — and frontend-triage opts in at `{"high"}` only,
so widening the policy later is an edit to that set rather than to the applier.
Because `confidence` is parsed out of the agent's free-form JSON block, it is
compared case- and whitespace-insensitively, and a run reporting nothing usable
never qualifies.
Confidence gates the agent's judgement, not its reach. An action's params are
model output and the apply step dispatches them against the runtime's *global*
handler registry — which can create bugs, attach files and write to Phabricator —
so restricting which tools the agent was given does not restrict what its
recorded actions reach. `auto_apply_guard` bounds that: for triage, one comment
and one add-only `keywords`/`severity` change on the bug the run was asked about.
Anything else holds the whole run, since the comment explains the field change and
the two are coalesced into one PUT.
The agent is now told what its rating causes, because it was being asked to
self-report a control input without knowing it was one.
Medium and low results are unchanged: still recorded, still visible in the UI,
still appliable by hand. So are all the other agents.
msujaws
force-pushed
the
frontend-triage-auto-apply
branch
from
August 6, 2026 19:55
1095882 to
d0e2d83
Compare
Contributor
|
I'm adding Slack support here, so I hope you'll be able to use it instead of emails when it's landed #6523 |
msujaws
force-pushed
the
frontend-triage-auto-apply
branch
from
August 7, 2026 15:33
d0e2d83 to
58b42a6
Compare
Contributor
Author
Thanks! I have removed the Slack notification commit from this PR since your approach will be much better. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
frontend-triageproduces a root-cause analysis and fix plan and records it as Bugzilla actions — but nothing applied them, so every triage needed a human to click Apply in the hackbot UI.This closes that loop for the results the agent is confident about.
Auto-apply, gated on confidence
AgentSpecgainsauto_apply_confidence: the set offindings.confidencelevels whose actions may be applied unattended, layered on top of the existingauto_apply_actionsswitch.frontend-triageopts in at{"high"}only, so widening the policy later — to also acceptmedium— is an edit to that set rather than to the applier.The decision is a pure predicate over
(spec, run)so the policy is testable on its own, and it fails closed: a run whose findings carry no usable confidence never qualifies. Becauseconfidenceis parsed out of the agent's free-form JSON block, it's compared case- and whitespace-insensitively, so"High"doesn't silently mean "never apply".Medium and low results are unchanged from today: still recorded, still visible in the UI, still appliable by hand. So are all the other agents — there's a test asserting that.
Bounding reach, not just judgement
Confidence gates how sure the agent is, not how far its actions can go. An action's params are model output, and the apply step dispatches them against the runtime's global handler registry — which can create bugs, attach files and write to Phabricator — so restricting which tools the agent was given does not restrict what its recorded actions reach.
auto_apply_guardbounds that separately. For triage it allows one comment and one add-onlykeywords/severitychange, both on the bug the run was asked about. Anything else holds the whole run rather than applying the parts that pass, since the comment explains the field change and the two are coalesced into a single PUT.Tell the agent what its confidence rating now causes
confidenceused to be advisory metadata a human read off the run. Now it's a control input that decides whether a comment reaches a real bug unreviewed — and the agent was being asked to self-report it without being told what it does, which invites grading on a curve.Testing
118 passedinservices/hackbot-api. The 3 failures intest_actions_applier.py(comment.is_markdownmissing from the expectations) and the 8 errors intest_list_runs_api.py/test_create_run_api.py(fixture 'client' not found) are pre-existing on master — verified by running master's suite unmodified in a clean worktree, which gives80 passed, 4 failed, 8 errors. This branch's failures are a strict subset: it rewrites the fourth,test_succeeded_opted_in_agent_records_and_applies.Deploying
No new configuration.
BUGZILLA_API_KEYis already set on the service and working (hackbot@mozilla.tldhas been applying comments), so nothing new is needed for the apply path.Follow-up
Firefox :: New Tab Pagebugs from staff — land this first, so a triggered run auto-applies.