Repository navigation
fix(task-execution): Supersede stale mailbox Turns on new input - #2562
Merged
Merged
Conversation
Conversation work can stop after repeated failed attempts (for example a worker lost at deploy rollout) without closing the Turn cursor. The next web or event Turn then started next to it, and every resume failed with "multiple active mailbox Turns" until retries ran out. New mailbox input now abandons every other open web or event Turn, not only auth-paused ones. Fixes JUNIOR-9M Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
3 screenshot changes — 3 changed · 0 added · 0 removed Review screenshots in Frameshift
|
dcramer
approved these changes
Nov 2, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What failed
JUNIOR-9M:
getActiveMailboxTurnIdthrowsConversation … has multiple active mailbox Turns(for examplelocal:web:67232d03…andslack:C0B595QDZLL:1793638594.088319). Every resume attempt of the Conversation throws, retries run out (conversation.work.retry.exhausted), and the Conversation stays stuck until a person posts again.Root cause
Generic conversation work can stop a Turn without closing its Turn cursor. Examples: the heartbeat clears an expired lease after a worker is lost (deploy rollout, freeze), lost-lease recovery runs out of retries, or a final-attempt Run failure in
conversation-turn.tsthat marks the Turn failed in the Conversation but leaves the checkpointrunning/paused. The next web or event message starts a new Turn next to it. New input only superseded auth-paused Turns, so the Conversation now has two open mailbox Turns. When the new Turn yields, its resume fails.Fix
When new mailbox input starts a Turn, abandon every other open web or event Turn, not only auth-paused ones. The OAuth prompt cleanup still runs only when an auth-paused Turn is superseded. Slack Turns are not mailbox Turns and stay as before.
Why a Turn that waits to resume is not superseded
New mailbox input starts a Turn only when no yielded Turn waits:
paused: yield (requestAnotherSlice), expired-lease recovery (clearExpiredConversationLease), a counted retry (recordConversationRetry), and lost-lease recovery in the worker.selectAttemptMessagesreturns no web or event input while execution ispaused. A steered web message is not an attempt interrupt while paused. Deferred input appended while paused keepspaused.retrypause from fix(task-execution): Resume a turn once after its worker stops checking in #2298 runs only inrunNextPausedTurn(Slack and dispatch workers). Mailbox Turns do not use it.resolveMailboxTurnWorkreturns no work for Slack input, and the Slack worker resumes the paused Turn and steers the mention into it.So a different open mailbox Turn at this point is stale.
conversation-turn-work.test.ts› "resumes a paused dashboard Turn before a deferred event" protects the selection rule: it fails when thepausedcheck inselectAttemptMessagesis removed.Review evidence
An end-to-end run through
processConversationQueueMessage, the real conversation-turn worker andrecoverConversationWorkreproduces JUNIOR-9M: a web Turn yields, five workers take the lease and are lost, and the heartbeat sets the Conversation tofailedwith the cursor stillpaused. Onmain, the next message yields and every resume fails with "multiple active mailbox Turns". On this branch, the stale Turn isabandonedand the next message yields, resumes and replies.That test is not in this PR:
scripts/check-test-architecture.mjsdoes not allow tests inpackages/junior/teststo run the agent or compose the worker. Its home isevals/integration/conversation/lifecycle.eval.ts, with a fixture step that loses workers. That needs new model recordings.Other checks:
AgentHistoryBranchErrorand the cursor staysabandoned.turn_started, and the dashboard reads the latest lifecycle event to show that Junior is responding.Remaining risk
multiple active agent invocationsfor child Conversations) is the same class of bug. fix(agent-invocations): Keep one active agent invocation per child #2558 fixed it: the worker ends every other active agent invocation of the child before it starts a new one.via @dcramer.
--
View Session [Sentry]