Skip to content

Auto-apply high-confidence frontend-triage results - #6439

Open
msujaws wants to merge 1 commit into
mozilla:masterfrom
msujaws:frontend-triage-auto-apply
Open

Auto-apply high-confidence frontend-triage results#6439
msujaws wants to merge 1 commit into
mozilla:masterfrom
msujaws:frontend-triage-auto-apply

Conversation

@msujaws

@msujaws msujaws commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

frontend-triage produces a root-cause analysis and fix plan and records it as Bugzilla actions — but nothing applied them, so every triage needed a human to click Apply in the hackbot UI.

This closes that loop for the results the agent is confident about.

Auto-apply, gated on confidence

AgentSpec gains auto_apply_confidence: the set of findings.confidence levels whose actions may be applied unattended, layered on top of the existing auto_apply_actions switch. frontend-triage opts in at {"high"} only, so widening the policy later — to also accept medium — is an edit to that set rather than to the applier.

The decision is a pure predicate over (spec, run) so the policy is testable on its own, and it fails closed: a run whose findings carry no usable confidence never qualifies. Because confidence is parsed out of the agent's free-form JSON block, it's compared case- and whitespace-insensitively, so "High" doesn't silently mean "never apply".

Medium and low results are unchanged from today: still recorded, still visible in the UI, still appliable by hand. So are all the other agents — there's a test asserting that.

Bounding reach, not just judgement

Confidence gates how sure the agent is, not how far its actions can go. An action's params are model output, and the apply step dispatches them against the runtime's global handler registry — which can create bugs, attach files and write to Phabricator — so restricting which tools the agent was given does not restrict what its recorded actions reach.

auto_apply_guard bounds that separately. For triage it allows one comment and one add-only keywords/severity change, both on the bug the run was asked about. Anything else holds the whole run rather than applying the parts that pass, since the comment explains the field change and the two are coalesced into a single PUT.

Tell the agent what its confidence rating now causes

confidence used to be advisory metadata a human read off the run. Now it's a control input that decides whether a comment reaches a real bug unreviewed — and the agent was being asked to self-report it without being told what it does, which invites grading on a curve.

Testing

118 passed in services/hackbot-api. The 3 failures in test_actions_applier.py (comment.is_markdown missing from the expectations) and the 8 errors in test_list_runs_api.py / test_create_run_api.py (fixture 'client' not found) are pre-existing on master — verified by running master's suite unmodified in a clean worktree, which gives 80 passed, 4 failed, 8 errors. This branch's failures are a strict subset: it rewrites the fourth, test_succeeded_opted_in_agent_records_and_applies.

Deploying

No new configuration. BUGZILLA_API_KEY is already set on the service and working (hackbot@mozilla.tld has been applying comments), so nothing new is needed for the apply path.

Follow-up

  • Reporting each outcome to its team's Slack channel is split out of this PR, to be redone against the native Slack API rather than the email-to-channel bridge this branch originally used.
  • mozilla/bugbot will start triggering these runs automatically for newly filed Firefox :: New Tab Page bugs from staff — land this first, so a triggered run auto-applies.

@msujaws
msujaws requested review from a team as code owners July 30, 2026 19:53
@msujaws msujaws self-assigned this Aug 4, 2026
@msujaws
msujaws force-pushed the frontend-triage-auto-apply branch 2 times, most recently from b7f1425 to 1095882 Compare August 6, 2026 19:44
frontend-triage produces a root-cause analysis and fix plan and records it as
Bugzilla actions, but nothing applied them: a human had to click Apply in the
hackbot UI. Close that loop for the results the agent is confident about.

`AgentSpec` gains `auto_apply_confidence` — the `findings.confidence` levels whose
actions may be applied unattended — and frontend-triage opts in at `{"high"}` only,
so widening the policy later is an edit to that set rather than to the applier.
Because `confidence` is parsed out of the agent's free-form JSON block, it is
compared case- and whitespace-insensitively, and a run reporting nothing usable
never qualifies.

Confidence gates the agent's judgement, not its reach. An action's params are
model output and the apply step dispatches them against the runtime's *global*
handler registry — which can create bugs, attach files and write to Phabricator —
so restricting which tools the agent was given does not restrict what its
recorded actions reach. `auto_apply_guard` bounds that: for triage, one comment
and one add-only `keywords`/`severity` change on the bug the run was asked about.
Anything else holds the whole run, since the comment explains the field change and
the two are coalesced into one PUT.

The agent is now told what its rating causes, because it was being asked to
self-report a control input without knowing it was one.

Medium and low results are unchanged: still recorded, still visible in the UI,
still appliable by hand. So are all the other agents.
@evgenyrp

evgenyrp commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

I'm adding Slack support here, so I hope you'll be able to use it instead of emails when it's landed #6523

@msujaws
msujaws force-pushed the frontend-triage-auto-apply branch from d0e2d83 to 58b42a6 Compare August 7, 2026 15:33
@msujaws msujaws changed the title Auto-apply high-confidence frontend-triage results and report them to Slack Auto-apply high-confidence frontend-triage results Aug 7, 2026
@msujaws

msujaws commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

I'm adding Slack support here, so I hope you'll be able to use it instead of emails when it's landed #6523

Thanks! I have removed the Slack notification commit from this PR since your approach will be much better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants