Repository navigation
Conversation
…uctured outputs A call that exists only to get JSON back is what structured outputs are for, and Claude Fable 5.1 rejects forced tool use outright. The shape guarantee is unchanged: `output_config.format` carries the same strict JSON schema the forced tool carried, so a malformed verdict stays impossible rather than merely unlikely. `_tool()` becomes `_output_schema()`, `STRUCTURED_TOOL_CHOICE` is gone, the reply is read from the text block and still passes through the existing fail-closed `_parse_verdicts`. `judge_trade_answers.py`'s transport label follows (`_structured_tool` -> `_structured_output`), as do `tests/agent/judge_narrative.py`'s `SCORE_TOOL` -> `SCORE_SCHEMA` and both README descriptions. This is not a live bug fix. `DEFAULT_MODELS["anthropic"]` is `claude-opus-5`, which still accepts forced tool use; this is the prerequisite for pointing `TR_JUDGE_MODEL` at Fable 5.1. New coverage: `test_structured_output_judge_reads_the_text_block_and_fails_closed` asserts five degenerate replies all raise rather than score, and the harness tests now assert `tools`/`tool_choice` are absent from the sent request. 60/60 offline suites pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provenance
Recovered from uncommitted work that had been sitting in
.claude/worktrees/qa-multi-turn-trace-counting-ed1876since 2026-09-02. It was never committed, never pushed, and never had a PR.git log --all -S"_output_schema"returns nothing — none of this is on main in any form.What changes
The Anthropic judge backend stops forcing a tool call and asks for a schema-constrained reply instead.
_tool(axes)returning a forced-tool spec withstrict: true_output_schema(axes)returning the same JSON schema"tool_choice": dict(STRUCTURED_TOOL_CHOICE)output_config.format = {"type": "json_schema", "schema": ...}tool_useblock_parse_verdicts_structured_tooltransport label_structured_outputSCORE_TOOLintests/agent/judge_narrative.pySCORE_SCHEMAThe shape guarantee is unchanged — the same strict schema, the same
additionalProperties: false, the same required axis list. A malformed verdict stays impossible rather than merely unlikely.Why
A call that exists only to get JSON back is what structured outputs are for, and Claude Fable 5.1 rejects forced tool use outright.
This is not a live bug fix.
DEFAULT_MODELS["anthropic"]isclaude-opus-5, which still accepts forced tool use, so nothing on main is broken today. This is the prerequisite for pointingTR_JUDGE_MODELat Fable 5.1.New coverage
test_structured_output_judge_reads_the_text_block_and_fails_closed— five degenerate replies (no text block, empty text, non-JSON, wrong shape, missing axis) must each raise rather than score.toolsandtool_choiceare absent from the sent request, so a silent regression to forced tool use fails.Verification
python3 tests/run_all.py→ PASS: all 60 suites passed (group: all).The remaining half of that recovered worktree — instruction-surface prompt-audit cleanup and a compact freeform witness — is a separate concern and does not touch these files. It follows in its own PR.
🤖 Generated with Claude Code