Skip to content

Retry a rejected report grouping instead of falling back - #212

Open
shellygr wants to merge 2 commits into
masterfrom
shelly/report-grouping-retry
Open

Retry a rejected report grouping instead of falling back#212
shellygr wants to merge 2 commits into
masterfrom
shelly/report-grouping-retry

Conversation

@shellygr

@shellygr shellygr commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #211.

A cloud run produced a report whose coverage.warnings began with FALLBACK GROUPING APPLIED, collapsing all 11 formalized properties into the single
general bucket. The cause:

report: grouping failed: 1 validation error for GroupingResult
groups
  Input should be a valid list [type=list_type, input_value='{"groups":[...]}', input_type=str]

The model serialized the whole GroupingResult document into the one field
meant to hold the list. The grouping was complete and correct; only the
encoding was wrong, and it landed in the same bucket as a transport failure.

What changes

A string that parses as the grouping is decoded, not rejected. A
field_validator(mode="before") on GroupingResult.groups handles both shapes
seen and plausible: the whole document ({"groups": [...]}) and a bare list.
Anything that does not decode is returned untouched so pydantic raises the same
error it always did. This is parsing rather than a fallback, so a usable answer
is not thrown away over its envelope.

Grouping gets the corrective retry the ranker already has.
call_grouping_llm previously made one structured call and let any schema miss
reach build's fallback. It now retries once with the rejection appended,
mirroring rank_properties in composer/spec/prioritize.py. The correction is
appended to the original request rather than sent as a second turn, so each
attempt stays one self-contained user message.

The single-bucket fallback is unchanged and still covers a grouping that could
not be obtained at all.

Tests

Five added. Each half of the fix is covered by a test that fails without it:
neutralizing the decoder fails both string-wrapping tests, and dropping
max_attempts to 1 fails the retry test.

  • a string-wrapped document decodes
  • a string-wrapped bare list decodes
  • a string that is not a grouping is still rejected
  • the retry fires once, carries the rejection, and the first attempt does not
  • after the retry it raises, so the caller's fallback still applies

1145 pass, pyright clean.

shellygr and others added 2 commits September 5, 2026 02:29
A cloud run lost every high-level heading in its report because the model
returned the whole GroupingResult document as a JSON string in the `groups`
field. The grouping in it was complete; only the envelope was wrong, and it
went straight to the single-bucket fallback.

Two things now stand between a malformed response and that fallback.

A before-validator on GroupingResult.groups decodes a string that parses as
the expected shape, either the whole document or a bare list. A string that
does not decode is handed back untouched, so pydantic rejects it as before.

call_grouping_llm retries once with the rejection appended, the way
rank_properties corrects a ranking. Grouping had no retry at all, so one bad
response cost the report all of its headings. The correction is appended to
the original request rather than sent as a follow-up turn, keeping each
attempt one self-contained user message.

The fallback stays as the last resort it was designed to be.

Closes #211

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report grouping falls back to one bucket when the model string-wraps its structured output

1 participant