Retry a rejected report grouping instead of falling back - #212
Open
shellygr wants to merge 2 commits into
Open
Conversation
A cloud run lost every high-level heading in its report because the model returned the whole GroupingResult document as a JSON string in the `groups` field. The grouping in it was complete; only the envelope was wrong, and it went straight to the single-bucket fallback. Two things now stand between a malformed response and that fallback. A before-validator on GroupingResult.groups decodes a string that parses as the expected shape, either the whole document or a bare list. A string that does not decode is handed back untouched, so pydantic rejects it as before. call_grouping_llm retries once with the rejection appended, the way rank_properties corrects a ranking. Grouping had no retry at all, so one bad response cost the report all of its headings. The correction is appended to the original request rather than sent as a follow-up turn, keeping each attempt one self-contained user message. The fallback stays as the last resort it was designed to be. Closes #211 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #211.
A cloud run produced a report whose
coverage.warningsbegan withFALLBACK GROUPING APPLIED, collapsing all 11 formalized properties into the singlegeneralbucket. The cause:The model serialized the whole
GroupingResultdocument into the one fieldmeant to hold the list. The grouping was complete and correct; only the
encoding was wrong, and it landed in the same bucket as a transport failure.
What changes
A string that parses as the grouping is decoded, not rejected. A
field_validator(mode="before")onGroupingResult.groupshandles both shapesseen and plausible: the whole document (
{"groups": [...]}) and a bare list.Anything that does not decode is returned untouched so pydantic raises the same
error it always did. This is parsing rather than a fallback, so a usable answer
is not thrown away over its envelope.
Grouping gets the corrective retry the ranker already has.
call_grouping_llmpreviously made one structured call and let any schema missreach
build's fallback. It now retries once with the rejection appended,mirroring
rank_propertiesincomposer/spec/prioritize.py. The correction isappended to the original request rather than sent as a second turn, so each
attempt stays one self-contained user message.
The single-bucket fallback is unchanged and still covers a grouping that could
not be obtained at all.
Tests
Five added. Each half of the fix is covered by a test that fails without it:
neutralizing the decoder fails both string-wrapping tests, and dropping
max_attemptsto 1 fails the retry test.1145 pass, pyright clean.