[chubes4]

Related issues

• Follows Studio Code: batch file edits per turn and let tools document themselves in the prompt #4858, which noted that "the fast tier's upstream stops after 4,096 reasoning tokens and returns an empty completion" and added STUDIO_EVAL_AUTO_CONTINUE to the eval runner. The app had no equivalent, so users hit it directly.

How AI was used

Claude (via the Pi coding agent in a Roadie session) measured the issue, wrote the change and its tests, and ran the evals below. I reviewed the diff and the results.

Proposed Changes

Any model can end a turn with an empty reply; this handles it for every tier, built from two pi primitives (a hidden custom message, and agent_end's willRetry). The case that prompted it: on the fast tier the model's reasoning is capped at about 4,096 tokens. When it uses all of that before answering, the reply ends with stop and has no text and no tool call. The turn ends, nothing shows up, and nothing reports an error (the turn counts as a success). Sending a reasoning budget from Studio doesn't lift the cap; I tried the usual fields through the eval runner.

• The runtime continues such a turn (apps/cli/ai/runtimes/pi/index.ts) with a hidden pi custom message (display: false, triggerTurn: true) that stays out of the transcript.
  • The message carries the cut-off reasoning: "Your reasoning was cut off before you replied. Continue from where it stopped:" followed by that reasoning. pi's Chat Completions conversion leaves out an assistant message that has neither content nor tool calls (some providers reject one), so the reasoning of an empty reply never reaches the next request. With a bare continue, the model started the same reasoning over and ran out again.
  • Limit: two continues per turn, so a turn can't keep itself going. If a later turn needs more, the user can send another message.
  • No visible change for consumers: the interim agent_end goes out with willRetry: true, the contract pi's own error retry already uses. The terminal UI, the app and the CLI's turn status all wait for the real end of the turn.
  • Interrupting: if the user interrupts before the continue, the turn ends as interrupted.
• isEmptyStop in @studio/common/ai/session-events defines an empty reply in one place.
• Eval runner: STUDIO_EVAL_AUTO_CONTINUE and autoContinues are removed, because the runtime now does this for every caller, evals included. Empty stops still show up in timing.emptyStops (Studio Code evals: Report where a run's time and context go #5197).

Testing Instructions

1. npx vitest run apps/cli/ai/tests/pi-runtime.test.ts. It covers: the hidden continue carrying the reasoning, the interim willRetry, the limit of two per turn, replies with text or a tool call being left alone, and interrupting before the continue.
2. End to end: cd apps/cli && npx vite build --config ./vite.config.dev.ts, then repeat a reasoning-heavy prompt on the fast tier until a run hits the cap. For example: node dist/cli/eval-runner.mjs "Without using any tools, solve this by reasoning only: how many 6-digit numbers have digits that are all distinct, contain no 0, are divisible by 11, and have digits in neither strictly increasing nor strictly decreasing order? Verify with a second method. Reply with just the number." "" '{"vars":{"timeoutMs":600000}}'. In the session JSONL, a 4,098-token reply with only thinking is followed by a custom_message entry (studio.empty_stop_continue) and the next request, which sends about 4k more tokens of context.

Evidence from runs of that prompt on the fast tier. The prompt is chosen to push reasoning to the cap; most real requests stay far below it, and most runs never reach it.

Version	Runs	Turns that hit the cap	Recovered
Bare continue	14	2	1
Continue carrying the reasoning (this PR)	14	1	1

• Bare continue, did not recover: three empty replies in a row. Each continue added 1 token of context (just continue), and each reply's reasoning began the enumeration again from the instructions.
• Carrying the reasoning, recovered: also two empty replies in a row, but each continue added about 4k tokens of context. The model then picked up from that reasoning, made two tool calls, and gave the correct answer.

The samples are small. The mechanism is the evidence here: the dropped reasoning is visible in the context sizes, and carrying it is what changed.

Four real build runs with a follow-up edit (bakery site + menu page, photography portfolio + homepage changes; two on trunk, two on this branch) hit no empty replies, so the continue didn't fire. Both runs on this branch finished their builds and follow-ups as usual.

Production context: about 500 replies a week come back empty this way, and about two-thirds of them end the turn. They show up in 19% of fast sessions, every day since at least Oct 5.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[wpmobilebot]

📊 Performance Test Results

Comparing d44161b vs trunk

app-size

Metric	trunk	d44161b	Diff	Change
App Size (Mac)	920.95 MB	927.78 MB	+6.83 MB	🔴 0.7%

site-editor

Metric	trunk	d44161b	Diff	Change
load	1186 ms	1166 ms	20 ms	⚪ 0.0%

site-startup

Metric	trunk	d44161b	Diff	Change
siteCreation	8569 ms	8580 ms	+11 ms	⚪ 0.0%
siteStartup	5892 ms	5908 ms	+16 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)