Skip to content

fix(prompts): ask for a progress note, not for written-out reasoning - #219

Open
glayyiyi wants to merge 1 commit into
google:mainfrom
glayyiyi:fix/flash-progress-note-prompt
Open

glayyiyi wants to merge 1 commit into
google:mainfrom
glayyiyi:fix/flash-progress-note-prompt

Conversation

@glayyiyi

Copy link
Copy Markdown

Summary

With Claude Opus 5.5 on Amazon Bedrock, some Flash turns come back with no content and these stop details:

stop_reason: refusal
stop_details: {type: refusal, category: reasoning_extraction}

The turn has 0 output tokens. The run then has nothing to act on.

Two pieces of the Flash prompt ask the model to write out its thinking:

  • the cognitive-protocol line in flash_runner.md: "think through observation, planning and action … your reply carries one short paragraph of connected prose reflecting those three phases";
  • REASONING_REMINDER: "Your previous turn carried no visible reasoning text (internal thinking is not shown) …".

This PR rewords both as a progress note: what the screen shows, which milestone it is on, and the next action. The step log receives the same information, and nothing else changes.

Evidence

I captured the refused requests through a logging proxy and replayed them unchanged against the same endpoint.

request change Opus 5.5 Sonnet 5.5 Haiku 5.5
turn 1 (system prompt only) none refusal 3/3 tool call 2/2 tool call 2/2
turn 1 reword the protocol line tool call 4/4
turn 5 (with the reminder) reword the protocol line only refusal 2/2
turn 5 reword the reminder only refusal 2/2
turn 5 reword both (this PR) tool call 3/3

The refusal depends on the request content. On some screens the original wording does go through, so it does not happen on every run.

With this change, plus #217, the two cases I use for on-device acceptance pass 4/4 rounds on Opus 5.5, with no refusals. One case is applying a theme in a custom-drawn store app; the other is Settings → Dark theme.

Tests

The existing Flash runner and reminder tests pass. They compare against the REASONING_REMINDER constant, not its text. make test, make lint and make typecheck pass.

Claude Opus 5.5 on Bedrock refuses the Flash turn (stop_reason refusal,
category reasoning_extraction, 0 output tokens) when the system prompt
asks it to write prose reflecting its thinking and when the follow-up
reminder mentions that internal thinking is not shown. Replayed against
captured requests: the original wording is refused 3/3 on Opus 5.5 while
Sonnet 5.5 and Haiku 5.5 answer; with both lines reworded as a progress
note (screen, milestone, next action) Opus answers with a tool call 3/3.
Same information reaches the step log.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant