Make the hackbot action apply path safe to repeat and hard to poison - #6529
Open
msujaws wants to merge 2 commits into
Open
Make the hackbot action apply path safe to repeat and hard to poison#6529msujaws wants to merge 2 commits into
msujaws wants to merge 2 commits into
Conversation
Two concurrent Pub/Sub deliveries of the same `run.completed` could both read a
`pending` row and both post its comment to the bug. Rows are now locked before
dispatch and stay locked until the result is committed, so a second delivery — or
a manual apply-all racing the automatic one — blocks, then reads `applied` and
skips.
Holding the lock rather than flagging the row in-progress is what keeps this from
needing recovery: a process that dies mid-dispatch rolls back to a row that is
exactly as retryable as it was, with no in-flight state for anything to reclaim
and no new column to migrate. The cost is a transaction open across the Bugzilla
call, which is why the lock is per coalesced group rather than per pass.
Along the way, three inputs that could 5xx the push route in a loop. With no
dead-letter topic configured the same message returns for the whole retention
window, so each now fails a row a human can read instead: a first-delivery race
on the row insert (now `ON CONFLICT DO NOTHING`), a recorded action with no type
or non-mapping params, and a params shape that raises inside the coalescer.
An unresolved `{{actions.<ref>.<field>}}` also stops being sent. It was left in
place deliberately so a human could see it, which is only safe if it never
reaches Bugzilla — and it could, once a referenced row failed.
A malformed envelope, a body that isn't JSON, or a `run_id` that isn't a UUID raised out of the route. Pub/Sub reads the resulting 5xx as a nack, and with no dead-letter topic configured on this subscription the same message comes back for the whole retention window — none of these can ever become valid. Log and ack instead. Only retryable failures should reach the client as an error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #6439 so it can be reviewed and landed on its own: these are fixes to
the existing apply path, not part of the frontend-triage auto-apply feature.
Both bugs are reachable today, on the manual apply-all path. They matter more once
anything applies actions unattended (#6439), which is why they were found — but
they stand alone and change no user-visible behaviour.
Concurrent deliveries could double-post
Pub/Sub push is at-least-once and concurrent, so two deliveries of the same
run.completedcould both read apendingrow and both post its comment to the bug.A manual apply-all racing the automatic path could do the same.
Rows are now locked before dispatch and stay locked until the result is committed, so
the second caller blocks, then reads
appliedand skips. Holding the lock rather thanflagging the row in-progress is what keeps this from needing recovery: a process that
dies mid-dispatch rolls back to a row exactly as retryable as it was — no in-flight
state to reclaim, and no new column to migrate. The cost is a transaction open across
the Bugzilla call, which is why the lock is per coalesced group rather than per pass.
Four inputs could 5xx the push route in a loop
There is no dead-letter topic on this subscription, so a 5xx replays the same message
for the whole retention window. Each of these now fails a row a human can read instead:
ON CONFLICT DO NOTHING)type, orparamsthat aren't a mapping{{actions.<ref>.<field>}}, which was previously sent literally into areal bug comment with only a log line as the signal
Notes
services/hackbot-api/testsarepre-existing on master (an
is_markdowncoalescing mismatch and a missingclientfixture, in files this PR does not touch).
actions_applier.py. Whicheverlands second needs a rebase.