Repository navigation
Studio Code evals: Report token usage and support follow-up messages - #5193
Merged
Merged
Conversation
1 task done
chubes4
marked this pull request as ready for review
October 20, 2026 01:01
…reason and provider error
Collaborator
📊 Performance Test ResultsComparing 7228893 vs trunk app-size
site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
…epending it to every message (#5192) ## Related issues - Follows #5170, #5189, #5190: moving Studio Code context from prose into structure, one small step at a time. - First step toward composing site facts the way the system prompt already composes tool snippets: each fact is rendered by the code that owns it, from Studio's own data. ## How AI was used Claude (via the Pi coding agent in a Roadie session) traced how the active site reaches the model, wrote the change and tests, and ran the eval below. I reviewed the diff and the eval traces. ## Proposed Changes The active site was stated by prepending a `[Active site: …]` line to every user message. So the same fact piled up in the conversation history once per turn, and it was mixed into what the user typed. This PR states it once, in an `## Active site` section of the system prompt instead: - `buildSystemPrompt` takes `activeSite` and renders the section. The text is the same as before, minus the brackets: name, path, running/stopped, and the DESIGN.md path when there is one; or ID, URL and "WordPress.com" for a remote site. - The Pi runtime already rebuilds the session, and so the system prompt, on every turn. It now passes the `activeSite` it was already receiving into `buildSystemPrompt`, so the section always reflects the current site and running state, including after compaction. - `ai` and the eval runner stop prepending the line. The user message is exactly what the user typed. - The three skills and the one prompt line that referred to "the active-site line" now say "the Active site section". Trade-off: when the active site or its running state changes, the system prompt changes, so that one turn misses the prompt cache. Before, each turn added the line to the history instead. ## Testing Instructions Unit tests: `npx vitest run apps/cli/ai apps/cli/commands/ai` (245 pass). - `system-prompt.test.ts`: the section for a local site, a WordPress.com site, and a site with a DESIGN.md. It sits above the user's global instructions and is absent without an active site. - `commands/ai/tests/index.test.ts`: the dispatched message is the typed text, and the runtime gets the site with the daemon's running state. Eval: four cases that depend on the agent knowing its active site, 2 runs each, this branch against trunk (`76f5c92bf`), using the `fast` model: | Case | What passes | Branch | Trunk | |---|---|---|---| | Tagline | Sets `blogdescription` and `blog_public` on the active site | 2/2 | 2/2 | | Plugins | Lists every installed plugin of the active site | 2/2 | 2/2 | | Admin bar (core file) | Core checksums still verify, and the change lives in `wp-content/` | 1/2 | 1/2 | | Design system | Reads the seeded DESIGN.md and the new page uses its required CTA text | 2/2 | 2/2 | Neither admin-bar failure involves the active site: - On the branch, the agent left core alone and set the height through Additional CSS (`wp_update_custom_css_post`). That's valid, but the check only looks at files. - On trunk, the model emitted a malformed tool call and the turn ended. In all 16 runs the agent worked in the correct site directory. The result is parity: the same facts, no longer repeated in the history. Multi-message eval, using #5193's token usage and `followUps`: a seeded site and eight messages (tagline, theme, pages, create a draft page, plugins, permalinks, site title), 2 runs per arm. To match the app, the trunk arm adds the site line to every message. | | Trunk (line in every message) | This PR (system-prompt section) | |---|---|---| | Final state correct (tagline, title, draft page) | 2/2 | 2/2 | | Site lines saved in the session | 8 per run | 0 | | Copies of the line re-sent across the run's requests | 73–78 | 0 (the section is sent once per request) | | Total tokens per run | 218k, 244k | 225k, 214k | The line is about 25 tokens, so this saves roughly 1.5k tokens per eight-message run, under 1% and well inside the run-to-run noise. The gain is in the transcript (it holds what the user typed, and no outdated copies of the site's state), not in the token bill. Manual: 1. `studio ai` with a site selected, then ask "what site are you working on?" The agent answers from the system prompt, and your message in the history has no prefix. 2. Switch site with `/site`, ask again: the answer reflects the new site. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 20, 2026
## Related issues - Follows #5193 (token usage and follow-ups in the eval runner). That gave each run's total cost; this shows where a run's time and context go. ## How AI was used Claude (via the Pi coding agent in a Roadie session) wrote the change and its tests, and ran the eval below. I reviewed the diff and the results. ## Proposed Changes Additions to the eval runner's result (`apps/cli/ai/eval-runner.ts`), with the summary logic in a new `apps/cli/ai/eval-timing.ts`: - **`requests`:** one record per model request: - start, end and duration, measured from the timestamp pi sets just before it sends the request; - context tokens sent (input + cache read + cache write) and output tokens; - `stopReason`, plus `errorMessage` when the request failed; - whether the reply carried text, and how many tool calls. - **`timing`:** a summary of the run: - model time; - tool wall time, counting parallel calls once; - number of requests and peak context; - `emptyStops`: replies that end with `stop` but carry no text and no tool call, so the turn ends and the user sees nothing; - per tool: calls, total and max time, and output size, slowest first. - **`outputChars`** on each `toolEvents` entry: the length of the result's text, which is what the tool adds to the context. Before, a run only had `turnDurationsMs`, which mixes model time with the previous tools' time. Context growth, failed requests that were retried, and silent empty stops were visible only by reading the session JSONL by hand. Existing fields are unchanged. The README documents the new fields. ## Testing Instructions 1. `npx vitest run apps/cli/ai/tests/eval-timing.test.ts` 2. `cd apps/cli && npx vite build --config ./vite.config.dev.ts` 3. Run any building case, for example `node dist/cli/eval-runner.mjs "Build me a website for my neighborhood bakery." "" '{"vars":{"timeoutMs":1200000}}'` 4. In the result file, expect `timing` (model time, tool wall time, peak context, per-tool entries) and one `requests` entry per assistant message. Example from one real run, a single-message bakery build on the default `fast` tier that took 181s: | | | |---|---| | Model time | 149s over 31 requests | | Tool wall time | 27s | | Context | 12k → 101k tokens | | Slowest tools | `inspect_design` 10.4s / 5 calls (108k chars of output), `site_start` 5.8s, `validate_blocks` 4.9s | | Largest outputs | `inspect_design` 108k chars, `Skill` 79k, `validate_blocks` 22k | | Hidden failure | request 2 hit `Request timed out.` after 10.5s and was retried | ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used
Claude (via the Pi coding agent in a Roadie session) wrote the change, updated the README, and used it to measure #5192. I reviewed the diff and the results.
Proposed Changes
Two additions to
apps/cli/ai/eval-runner.ts, so prompt changes can be compared on cost and across a conversation, not just on one request's outcome:usagein the result: input, output, cache read, cache write and total tokens, summed from every assistant message. Promptfoo assertions can report it next to pass/fail.followUpsvar: a list of further user messages ({ "prompt": "…" }objects, since promptfoo expands an array of strings into one test case per element), each sent in the same session once the previous one finishes (including anySTUDIO_EVAL_AUTO_CONTINUEresumes). The timeout covers the whole conversation.The transcript, every run: the session JSONL is always written (to
<tmpdir>/studio-eval-sessions, orSTUDIO_EVAL_SESSION_DIR), and the result reports its path assessionFile. Before, it was kept only when that variable was set.stopReasonanderrorMessagefrom the last assistant message. A provider failure ends the turn without throwing, so the result used to saysuccess: falsewitherror: null.These supersede #3376's diagnostics. Its transcript, final text and last-message usage are covered by the above and by
textSegments/timedOut, and its "no useful output" heuristic is a judgment that belongs in each case's assertions.The README documents all of it.
Testing Instructions
cd apps/cli && npx vite build --config ./vite.config.dev.tsnode dist/cli/eval-runner.mjs "What's this site's tagline?" "" '{"vars":{"followUps":[{"prompt":"Which theme is active?"}]}}'usageobject with non-zero counts,stopReason: "stop", both answers intextSegments, and asessionFilethat exists and holds both messages.I ran it as the measurement for #5192: a seeded site and eight follow-up messages per run (results on that PR).
Pre-merge Checklist