Skip to content

Make the hackbot action apply path safe to repeat and hard to poison - #6529

Open
msujaws wants to merge 2 commits into
mozilla:masterfrom
msujaws:apply-path-hardening
Open

Make the hackbot action apply path safe to repeat and hard to poison#6529
msujaws wants to merge 2 commits into
mozilla:masterfrom
msujaws:apply-path-hardening

Conversation

@msujaws

@msujaws msujaws commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Split out of #6439 so it can be reviewed and landed on its own: these are fixes to
the existing apply path, not part of the frontend-triage auto-apply feature.

Both bugs are reachable today, on the manual apply-all path. They matter more once
anything applies actions unattended (#6439), which is why they were found — but
they stand alone and change no user-visible behaviour.

Concurrent deliveries could double-post

Pub/Sub push is at-least-once and concurrent, so two deliveries of the same
run.completed could both read a pending row and both post its comment to the bug.
A manual apply-all racing the automatic path could do the same.

Rows are now locked before dispatch and stay locked until the result is committed, so
the second caller blocks, then reads applied and skips. Holding the lock rather than
flagging the row in-progress is what keeps this from needing recovery: a process that
dies mid-dispatch rolls back to a row exactly as retryable as it was — no in-flight
state to reclaim, and no new column to migrate. The cost is a transaction open across
the Bugzilla call, which is why the lock is per coalesced group rather than per pass.

Four inputs could 5xx the push route in a loop

There is no dead-letter topic on this subscription, so a 5xx replays the same message
for the whole retention window. Each of these now fails a row a human can read instead:

  • two concurrent first deliveries racing the row insert (now ON CONFLICT DO NOTHING)
  • a recorded action with no type, or params that aren't a mapping
  • a params shape that raises inside the coalescer
  • an unresolved {{actions.<ref>.<field>}}, which was previously sent literally into a
    real bug comment with only a log line as the signal

Notes

  • No schema change and no migration.
  • No UI change.
  • Adds 12 tests; the 3 failures and 8 errors in services/hackbot-api/tests are
    pre-existing on master (an is_markdown coalescing mismatch and a missing client
    fixture, in files this PR does not touch).
  • Conflicts with Auto-apply high-confidence frontend-triage results #6439 by construction — both touch actions_applier.py. Whichever
    lands second needs a rebase.

msujaws added 2 commits August 6, 2026 15:32
Two concurrent Pub/Sub deliveries of the same `run.completed` could both read a
`pending` row and both post its comment to the bug. Rows are now locked before
dispatch and stay locked until the result is committed, so a second delivery — or
a manual apply-all racing the automatic one — blocks, then reads `applied` and
skips.

Holding the lock rather than flagging the row in-progress is what keeps this from
needing recovery: a process that dies mid-dispatch rolls back to a row that is
exactly as retryable as it was, with no in-flight state for anything to reclaim
and no new column to migrate. The cost is a transaction open across the Bugzilla
call, which is why the lock is per coalesced group rather than per pass.

Along the way, three inputs that could 5xx the push route in a loop. With no
dead-letter topic configured the same message returns for the whole retention
window, so each now fails a row a human can read instead: a first-delivery race
on the row insert (now `ON CONFLICT DO NOTHING`), a recorded action with no type
or non-mapping params, and a params shape that raises inside the coalescer.

An unresolved `{{actions.<ref>.<field>}}` also stops being sent. It was left in
place deliberately so a human could see it, which is only safe if it never
reaches Bugzilla — and it could, once a referenced row failed.
A malformed envelope, a body that isn't JSON, or a `run_id` that isn't a UUID
raised out of the route. Pub/Sub reads the resulting 5xx as a nack, and with no
dead-letter topic configured on this subscription the same message comes back for
the whole retention window — none of these can ever become valid.

Log and ack instead. Only retryable failures should reach the client as an error.
@msujaws
msujaws requested a review from a team as a code owner August 6, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant