Skip to content

fix(dispatcher): retry transient ClaimDue failures and give the claim a longer timeout - #12

Closed
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1790543392-dispatcher-claim-retry
Closed

devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1790543392-dispatcher-claim-retry

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

On Cloud Run, cold CPU allocation and idle-pool reconnects push the first few ClaimDue calls past the 5s QueryTimeoutDuration, producing recurring failed to claim due notifications / context deadline exceeded ERROR logs even though nothing is actually wrong.

  • store.ClaimTimeoutDuration = QueryTimeoutDuration*2 (10s) now bounds ClaimDue, since it opens a transaction and runs a FOR UPDATE SKIP LOCKED select plus an update rather than a single query.
  • dispatchDueNotifications now claims via claimDueNotifications, which retries up to dispatcherClaimRetries (2) extra times with a dispatcherClaimRetryBackoff (2s) pause, only when isTransientClaimError(err):
    • context.DeadlineExceeded (incl. wrapped), pgconn.SafeToRetry/pgconn.Timeout, any net.Error, and PgError classes 08xx / 40001 / 40P01 / 57P01-03.
    • Anything else (e.g. 42P01, opaque errors) returns immediately, so a real outage still logs once per tick. Retries also stop as soon as the dispatcher ctx is cancelled, and a cancelled-ctx failure (shutdown) is no longer logged as an error.
  • The startup sweep now runs dispatchDueNotificationsAt(ctx, startup=true): if it still ends in DeadlineExceeded after retries, it logs at Warn instead of Error.

Tests: TestClaimDueNotifications (success on 2nd attempt, wrapped deadline, retry budget exhausted, permanent PgError / opaque error not retried, no retry after ctx cancel) and TestIsTransientClaimError.

Link to Devin session: https://app.devin.ai/sessions/40339faccc2c46948d4a8fb2bee608e5
Open in Devin Desktop: https://app.devin.ai/desktop/session/40339faccc2c46948d4a8fb2bee608e5?variant=devin
Requested by: @balebbae

@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@balebbae balebbae closed this Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant