mozilla · msujaws · Aug 6, 2026
@@ -99,14 +99,24 @@ Each run writes to `~/hackbot/artifacts/<run_id>/`:
 
 - **`summary.json`** — `findings` holds the structured plan (`root_cause`,
   `proposed_fix`, `target_files`, `confidence`) plus the executor handoff fields
-  `actionable`, `regressor_node` and `relevant_tests`. `actions` holds the single
-  **recorded** `bugzilla.add_comment` — written here for review, not posted.
+  `actionable`, `regressor_node` and `relevant_tests`. `actions` holds the
+  **recorded** `bugzilla.add_comment` (and, at high confidence, possibly a
+  `bugzilla.update_bug`). Recording is not posting — but see below: hackbot posts
+  a high-confidence run's actions to the bug unattended.
 - **`logs/agent.log`** — the streamed reasoning and every tool call, and the only
   record of which model actually ran.
 - **No `changes/` directory.** Its absence confirms the run stayed read-only.
 
-Two things to know before acting on a plan:
+Three things to know before acting on a plan:
 
+- **`confidence` decides whether the actions are posted.** A `high`-confidence
+  run has its recorded actions applied to the real bug automatically; `medium`
+  and `low` are held for a human to apply from the hackbot UI. Only the fields
+  the agent is trusted with (`keywords`, `severity`) can go up unattended, and
+  only against the bug the run was asked about — anything else holds the whole
+  run for review. See `auto_apply_confidence` in
+  `services/hackbot-api/app/agents.py` and `frontend_triage_guard` in
+  `services/hackbot-api/app/auto_apply.py`.
 - **`confidence` describes the diagnosis, not the fix.** It reflects how clearly
   the agent pinned a root cause in the code, never whether the fix works — it
   cannot run anything. Read `high` as "trust the diagnosis, still review the

@@ -33,6 +33,10 @@
 # and plans only: it records a comment with its findings/plan and, at high
 # confidence, may propose field updates (e.g. keyword/severity). It never
 # creates bugs or attaches files.
+#
+# `bugzilla.update_bug` needs `editbugs` on the apply account. The apply step coalesces
+# a same-bug field change with the nearest comment into one PUT, so losing that
+# privilege would take the analysis comment down with the rejected field change.
 ENABLED_ACTION_TYPES = [
     "bugzilla.add_comment",
     "bugzilla.update_bug",

@@ -36,10 +36,17 @@ not restate the whole bug. Do not claim the fix is verified — you did not run
 
 ## Confidence and field changes
 
+Your `confidence` decides whether your actions reach the bug: a **high**-confidence
+run is applied to Bugzilla automatically, while **medium** and **low** are held for
+a human to review first. Reserve `high` for when you have actually localized the
+cause in specific code — not for a plausible-sounding hypothesis.
+
 - **High** (you found the specific code and the cause is clear): record the
   plan comment. If a rule or convention clearly applies, you may also record a
   `bugzilla_update_bug` for an obviously-correct field (e.g. adding a relevant
-  keyword). Do not change `status`/`resolution`.
+  keyword). Do not change `status`/`resolution`. Record a keyword addition as
+  `{"keywords": {"add": ["…"]}}` — a bare list replaces every keyword already on
+  the bug, so it is held for review rather than applied.
 - **Medium** (plausible area, cause not pinned down): record the comment with
   your best hypothesis and the open questions that would confirm it.
 - **Low** (could not localize): record a comment stating what you checked and

@@ -64,7 +64,9 @@ async def update_bug(
 ) -> str:
     """Record an intended change to a Bugzilla bug.
 
-    Recorded into the run summary for human review — does not modify Bugzilla.
+    Recorded into the run summary rather than sent from here. Some agents have their
+    recorded changes applied without human review, so record only changes you would
+    stand behind unreviewed.
     """
     recorder.record(
         "bugzilla.update_bug",
@@ -92,8 +94,9 @@ async def add_comment(
 ) -> str:
     """Record an intended comment on a bug.
 
-    Use is_private=true for security-sensitive notes. Recorded into the run
-    summary for human review — does not post to Bugzilla.
+    Use is_private=true for security-sensitive notes. Recorded into the run summary
+    rather than posted from here. Some agents have their recorded comments posted
+    without human review, so write it as if it will be read on the bug unreviewed.
     """
     text_with_footer = text.rstrip() + "\n\n" + _COMMENT_FOOTER
     recorder.record(

@@ -2,12 +2,12 @@
 
 On run completion the recorded actions from `summary["actions"]` are always
 upserted as `run_actions` rows (one per entry) so they're visible and
-manageable in the UI. Whether they're then applied *automatically* depends on
-the agent's `auto_apply_actions` opt-in (see `app/agents.py`); either way they
-can be applied on demand (manual apply-all from the UI). Application runs each
-pending row through the handler registry in `hackbot_runtime.actions.handlers`
-and is idempotent per action — an already-`applied` row is never re-applied, so
-Pub/Sub retries and repeated manual applies are safe.
+manageable in the UI. Whether they're then applied *automatically* is decided by
+`_should_auto_apply` (see `app/agents.py`); either way they can be applied on demand
+(manual apply-all from the UI). Application runs each pending row through the handler
+registry in `hackbot_runtime.actions.handlers` and is idempotent per action — an
+already-`applied` row is never re-applied, so Pub/Sub retries and repeated manual
+applies are safe.
 """
 
 from __future__ import annotations
@@ -28,9 +28,9 @@
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app import gcs
-from app.agents import AGENT_REGISTRY
+from app.agents import AGENT_REGISTRY, AgentSpec
 from app.database.models import Run, RunAction
-from app.schemas import RunStatus
+from app.schemas import Confidence, RunStatus, parse_confidence
 
 log = logging.getLogger(__name__)
 
@@ -86,6 +86,58 @@ def _sub(match: re.Match) -> str:
     return value
 
 
+def _reported_confidence(run: Run) -> Confidence | None:
+    """The run's self-reported confidence, or None if it didn't report a usable one."""
+    return parse_confidence(
+        ((run.summary or {}).get("findings") or {}).get("confidence")
+    )
+
+
+def _should_auto_apply(
+    spec: AgentSpec | None, run: Run, rows: list[tuple[RunAction, list[dict]]]
+) -> bool:
+    """Whether `run`'s recorded actions may be applied without a human.
+
+    The whole unattended-apply policy in one place, so "why didn't this apply?" has one
+    answer, and every gate fails closed.
+
+    Judged on the persisted rows rather than `summary["actions"]`, because the rows are
+    what gets dispatched: `ensure_action_rows` never rewrites an existing row, so if the
+    two diverge, checking the summary would approve one payload while a different one
+    went to Bugzilla.
+    """
+    if spec is None or not spec.auto_apply_actions:
+        return False
+
+    # `rules/scoping.md` pairs an out-of-scope report with `confidence: low`, but nothing
+    # makes the agent do so — a `high` + `actionable: false` run would otherwise post an
+    # out-of-scope note on the strength of the confidence alone. `is False`, so a missing
+    # `actionable` doesn't read as "out of scope".
+    findings = (run.summary or {}).get("findings") or {}
+    if findings.get("actionable") is False:
+        return False
+
+    if (
+        spec.auto_apply_confidence is not None
+        and _reported_confidence(run) not in spec.auto_apply_confidence
+    ):
+        return False
+
+    if spec.auto_apply_guard is not None:
+        reason = spec.auto_apply_guard(run, [row for row, _ in rows])
+        if reason is not None:
+            log.warning(
+                "Holding run %s for review: %s (agent %s)",
+                run.run_id,
+                reason,
+                run.agent,
+            )
+            return False
+
+    return True
+
+
+
 async def ensure_action_rows(
     db: AsyncSession, run: Run
 ) -> list[tuple[RunAction, list[dict]]]:
@@ -224,11 +276,11 @@ async def _apply_pending_rows(
 
 
 async def on_run_completed(db: AsyncSession, run: Run) -> None:
-    """Record a completed run's actions, and auto-apply them if the agent opts in.
+    """Record a completed run's actions, and auto-apply them if the agent qualifies.
 
-    Called from the `apply-run-actions` push route. Actions are always recorded
-    (so the UI can show/manually apply them); they're applied automatically only
-    when the run's agent has `auto_apply_actions=True`.
+    Called from the `apply-run-actions` push route. Actions are always recorded (so the
+    UI can show/manually apply them); they're applied automatically only when
+    `_should_auto_apply` says so.
     """
     # Defense-in-depth: only a succeeded run's actions are recorded/applied. A
     # failed/timed-out run may have recorded actions before erroring, but acting
@@ -243,15 +295,17 @@ async def on_run_completed(db: AsyncSession, run: Run) -> None:
     await db.commit()
 
     spec = AGENT_REGISTRY.get(run.agent)
-    if spec and spec.auto_apply_actions:
+    if _should_auto_apply(spec, run, rows):
         await _apply_pending_rows(db, run, rows)
-    else:
-        log.info(
-            "Recorded %d action(s) for run %s; auto-apply off for agent %s",
-            len(rows),
-            run.run_id,
-            run.agent,
-        )
+        return
+
+    log.info(
+        "Recorded %d action(s) for run %s; not auto-applying (agent %s, confidence %s)",
+        len(rows),
+        run.run_id,
+        run.agent,
+        _reported_confidence(run),
+    )
 
 
 async def apply_all_pending(db: AsyncSession, run: Run) -> None:

@@ -4,10 +4,13 @@
 
 from pydantic import BaseModel
 
+from app.auto_apply import frontend_triage_guard
+from app.database.models import Run, RunAction
 from app.schemas import (
     AutowebcompatReproInputs,
     BugFixInputs,
     BuildRepairInputs,
+    Confidence,
     FrontendTriageInputs,
     TestPlanGeneratorInputs,
     TestRepairInputs,
@@ -27,6 +30,12 @@ class AgentSpec:
     # succeeds. Off by default: actions are still recorded and can always be
     # applied manually from the UI; only opted-in agents auto-apply.
     auto_apply_actions: bool = False
+    # Fail closed: when set, a run whose findings carry no usable confidence never
+    # qualifies. Widening the policy is an edit to this set, not to the applier.
+    auto_apply_confidence: frozenset[Confidence] | None = None
+    # Bounds what a run may write unattended, beyond what confidence already gates.
+    # Returns the reason a human is needed, or None. See app/auto_apply.py.
+    auto_apply_guard: Callable[[Run, list[RunAction]], str | None] | None = None
 
 
 def model_to_env(inputs: BaseModel) -> dict[str, str]:
@@ -80,6 +89,11 @@ def model_to_env(inputs: BaseModel) -> dict[str, str]:
         description="Triage a Firefox desktop frontend bug (read-only) and produce a root-cause analysis and proposed fix plan.",
         job_name="hackbot-agent-frontend-triage",
         input_schema=FrontendTriageInputs,
+        # Triage results reach a real bug unattended, so only the ones the agent
+        # localized to specific code qualify. Medium/low stays for manual apply.
+        auto_apply_actions=True,
+        auto_apply_confidence=frozenset({Confidence.high}),
+        auto_apply_guard=frontend_triage_guard,
     ),
     "test-repair": AgentSpec(
         name="test-repair",

@@ -0,0 +1,111 @@
+"""Per-agent limits on what a run may write to Bugzilla without a human.
+
+An agent's `confidence` gates its *judgement*; a guard here gates its *reach*. The
+two are separate because an action's params are model output and the apply step
+dispatches them against the runtime's global handler registry — which can create
+bugs, attach files and write to Phabricator — so restricting which tools the agent
+was given does not restrict what its recorded actions can reach.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from app.database.models import Run, RunAction
+
+_TRIAGE_FIELDS = frozenset({"keywords", "severity"})
+
+
+def _is_bug_id(value: Any) -> bool:
+    # An int or a plain run of digits, nothing looser: the handler interpolates this
+    # raw value into the REST path, whereas `int()` would also accept `"2_014_702"`,
+    # signs, whitespace and non-ASCII digits — validating a different string than the
+    # one sent.
+    if isinstance(value, bool):
+        return False
+    return isinstance(value, int) or (isinstance(value, str) and value.isdigit())
+
+
+def _field_change(field: str, value: Any) -> str | None:
+    """Why setting `field` to `value` is more than an addition, or None."""
+    if field == "severity":
+        # A single-valued field has no additive form, so a scalar is the only way to
+        # set it. The value isn't checked against Bugzilla's vocabulary: an unknown one
+        # is rejected there, surfacing as a failed action rather than a wrong write.
+        if isinstance(value, str) and value.strip():
+            return None
+        return f"severity is set to an unexpected {type(value).__name__}"
+
+    # A bare list *replaces* every keyword already on the bug; `{"add": [...]}` is the
+    # only form that adds.
+    if not isinstance(value, dict):
+        return f"{field} is set wholesale rather than added to"
+    if set(value) - {"add"}:
+        return f"{field} is edited with {', '.join(sorted(value))}, not add"
+    additions = value.get("add")
+    if not isinstance(additions, list) or not additions:
+        return f"{field}'s add is not a non-empty list"
+    if not all(isinstance(item, str) and item.strip() for item in additions):
+        return f"{field} adds something that isn't a non-empty string"
+    return None
+
+
+def frontend_triage_guard(run: Run, rows: list[RunAction]) -> str | None:
+    """Why this triage run needs a human, or None if it may apply unattended.
+
+    What `rules/frontend-triage.md` sanctions: one plan comment and, at most, one
+    obviously-correct field addition on the bug the run was asked about. A run
+    proposing anything else is held whole rather than part-applied — the comment
+    explains the field change and the two are coalesced into one Bugzilla PUT, so
+    dropping one and applying the rest would post something the agent didn't propose.
+    """
+    expected_bug_id = (run.inputs or {}).get("bug_id")
+    seen: set[str] = set()
+
+    for row in rows:
+        params = row.params or {}
+
+        if row.type not in ("bugzilla.add_comment", "bugzilla.update_bug"):
+            return f"{row.type} is not an action type it may apply unattended"
+        if row.type in seen:
+            return f"it records more than one {row.type}"
+        seen.add(row.type)
+
+        bug_id = params.get("bug_id")
+        # Required, not merely compared when present: `bugzilla.create_bug` carries no
+        # `bug_id` at all, and "no target" must not read as "target matches".
+        if bug_id is None or expected_bug_id is None:
+            return f"{row.type} names no bug to check against the run's input"
+        if not _is_bug_id(bug_id):
+            return f"it targets an unreadable bug id {bug_id!r}"
+        if int(bug_id) != int(expected_bug_id):
+            return f"it targets bug {bug_id}, not the run's bug {expected_bug_id}"
+
+        # A private comment is invisible to the reporter and the public, which defeats
+        # the review-by-visibility this design leans on. Wanting privacy is exactly the
+        # case that wants a human.
+        if row.type == "bugzilla.add_comment" and params.get("is_private"):
+            return "it posts a private comment"
+
+        if row.type != "bugzilla.update_bug":
+            continue
+
+        # `UpdateBugHandler` forwards a `comment` param straight into the PUT, so it is a
+        # second route to posting one — including a private one, past the check above.
+        # (A `comment` key inside `changes` is a third, caught by the allowlist below.)
+        if params.get("comment") is not None:
+            return "it carries its own comment rather than a coalesced one"
+        # Not `or {}`: a falsey non-mapping (`[]`, `""`, `0`) would become an empty dict
+        # and sail through as "changes nothing" instead of being held.
+        changes = params.get("changes")
+        if not isinstance(changes, dict) or not changes:
+            return "its `changes` is not a non-empty mapping of fields"
+        disallowed = sorted(set(changes) - _TRIAGE_FIELDS)
+        if disallowed:
+            return f"it changes {', '.join(disallowed)}, which it may not change"
+        for field, value in changes.items():
+            reason = _field_change(field, value)
+            if reason is not None:
+                return reason
+
+    return None
@@ -14,6 +14,33 @@ class RunStatus(str, Enum):
     timed_out = "timed_out"
 
 
+class Confidence(str, Enum):
+    """How sure an agent is of its own findings.
+
+    Reported by the agent inside a free-form JSON block, so it arrives as untrusted
+    text — use `parse_confidence` rather than the constructor.
+    """
+
+    high = "high"
+    medium = "medium"
+    low = "low"
+
+
+def parse_confidence(value: object) -> Confidence | None:
+    """`value` as a `Confidence`, or None if it isn't one.
+
+    Tolerates casing and stray whitespace, because the value is parsed out of
+    model output and "High" must not silently read as "no confidence". Anything
+    else is None: callers fail closed rather than inventing a level.
+    """
+    if not isinstance(value, str):
+        return None
+    try:
+        return Confidence(value.strip().lower())
+    except ValueError:
+        return None
+
+
 class ArtifactRef(BaseModel):
     name: str
     size: int