[chubes4]

Related issues

• Follows Studio Code evals: Report token usage and support follow-up messages #5193 (token usage and follow-ups in the eval runner). That gave each run's total cost; this shows where a run's time and context go.

How AI was used

Claude (via the Pi coding agent in a Roadie session) wrote the change and its tests, and ran the eval below. I reviewed the diff and the results.

Proposed Changes

Additions to the eval runner's result (apps/cli/ai/eval-runner.ts), with the summary logic in a new apps/cli/ai/eval-timing.ts:

• requests: one record per model request:
  • start, end and duration, measured from the timestamp pi sets just before it sends the request;
  • context tokens sent (input + cache read + cache write) and output tokens;
  • stopReason, plus errorMessage when the request failed;
  • whether the reply carried text, and how many tool calls.
• timing: a summary of the run:
  • model time;
  • tool wall time, counting parallel calls once;
  • number of requests and peak context;
  • emptyStops: replies that end with stop but carry no text and no tool call, so the turn ends and the user sees nothing;
  • per tool: calls, total and max time, and output size, slowest first.
• outputChars on each toolEvents entry: the length of the result's text, which is what the tool adds to the context.

Before, a run only had turnDurationsMs, which mixes model time with the previous tools' time. Context growth, failed requests that were retried, and silent empty stops were visible only by reading the session JSONL by hand. Existing fields are unchanged.

The README documents the new fields.

Testing Instructions

1. npx vitest run apps/cli/ai/tests/eval-timing.test.ts
2. cd apps/cli && npx vite build --config ./vite.config.dev.ts
3. Run any building case, for example node dist/cli/eval-runner.mjs "Build me a website for my neighborhood bakery." "" '{"vars":{"timeoutMs":1200000}}'
4. In the result file, expect timing (model time, tool wall time, peak context, per-tool entries) and one requests entry per assistant message.

Example from one real run, a single-message bakery build on the default fast tier that took 181s:

Model time	149s over 31 requests
Tool wall time	27s
Context	12k → 101k tokens
Slowest tools	inspect_design 10.4s / 5 calls (108k chars of output), site_start 5.8s, validate_blocks 4.9s
Largest outputs	inspect_design 108k chars, Skill 79k, validate_blocks 22k
Hidden failure	request 2 hit Request timed out. after 10.5s and was retried

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[wpmobilebot]

📊 Performance Test Results

Comparing 1bbd1aa vs trunk

app-size

Metric	trunk	1bbd1aa	Diff	Change
App Size (Mac)	927.54 MB	927.54 MB	0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	1bbd1aa	Diff	Change
load	1137 ms	1192 ms	+55 ms	🔴 4.8%

site-startup

Metric	trunk	1bbd1aa	Diff	Change
siteCreation	8144 ms	8583 ms	+439 ms	🔴 5.4%
siteStartup	5900 ms	5914 ms	+14 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)