llm: retry an error the server reports mid-stream - #220
Merged
Conversation
An error event that arrives inside an open stream rides the 200 response the SDK is already reading, so the SDK builds a plain APIStatusError from it. should_retry keys on the status code, sees 200, and treats the failure as deterministic. An overloaded or internal-server error delivered that way then takes the whole run down on its first attempt, without the run-wide retry floor ever being consulted. Classify those by the payload's error.type instead, accepting the same four conditions the status roster already accepts. A status code that can speak for itself still decides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jar-ben
approved these changes
Sep 7, 2026
shellygr
changed the base branch from
jaroslav/anthropic-stream-retry
to
master
September 7, 2026 18:24
shellygr
enabled auto-merge (squash)
September 7, 2026 18:24
…dstream-error-retry # Conflicts: # composer/llm/anthropic.py # tests/test_anthropic_retry.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
should_retrydecides whether a provider failure is transient, and the run-wide retry floorresumes a graph from its checkpoint when it is. For an
APIStatusErrorthat decision reads thestatus code: 408/409/429 and every 5xx are retryable.
An error the server reports part-way through a stream never gets a status code of its own. It
arrives as an SSE
errorevent on the already-open 200 response, and the SDK builds the exceptionfrom that response.
_make_status_errorfinds no mapping for 200, so it hands back the baseAPIStatusError, carryingstatus_code == 200and naming the real condition (overloaded_error,api_error) only in the body.should_retrysees the 200, calls the failure deterministic, and itpropagates on the first attempt.
The damage goes past the one task. The property-extraction fan-out is gathered without
return_exceptions, so the first such error ends the run before formalization and discards thecomponent analysis, harness and autosetup work already done.
How
When the status code cannot speak for the failure, read the payload's
error.typeand accept thesame conditions the status roster accepts: overloaded, api_error, rate limit, timeout. A body of any
other shape stays non-retryable (the SDK leaves it as raw text when the payload does not parse), and
so does a mid-stream
invalid_request_error, which is as deterministic as its 400 twin.Builds on #217, which covers the other half of the same gap: a connection that drops mid-stream.
Merge that one first; this PR targets its branch.
Testing
tests/test_anthropic_retry.pycovers both mid-stream conditions, the mid-stream request error, anunparsed body, and a real 529 to keep the status-code path honest. Local: pytest 1189 passed (the
test_rag_db.pyerrors are a postgres container that will not start on this machine), pyright cleanon the CI target set.