[chubes4]

Related issues

• Related to Spike: Studio Code reads the WordPress source instead of being taught WordPress #5146

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) wrote the outcome capture, the checks, their unit tests and the isolation, and ran the suite on this branch and on the #5146 spike. I reviewed the diff and the results. The checks in outcome.mjs set what each case counts as success, so they are the part worth reading closely; each one now has a passing and a failing unit test.

Proposed Changes

• Judge outcomes, not routes. The suite asserted which tools the agent called and in what order. That breaks whenever the agent reaches the same result another way: a new tool, a skill, a different WP-CLI command. It also cannot compare two agent designs fairly. Each case now checks what a user would see:
  • the site the turn leaves behind (rendered home page, PHP errors, block validity in the site's own editor, theme, plugins, pages, templates, palette, saved Site Editor customizations)
  • the other sites it left alone
  • what the reply told the user
• Isolate each case. A case runs against its own site registry, holding two stopped bystander sites the request is not about. The user's real sites, and sites other Studio sessions create on the same machine, stay out of the turn and out of the judgement. A case fails when it changes a bystander.
• Give each case its own sites folder. Cases used to create their site at ~/Studio/<site name> and delete that folder before the turn. Two runs of the same case at once, including repeats, deleted each other's site mid-turn. The runner now loads through a small entry (eval-runner-entry.ts) that sets a temporary sites root per case before anything reads it. site-paths.ts honours it under E2E, the same way well-known-paths.ts honours E2E_CLI_CONFIG_PATH. Eval sites no longer land in ~/Studio. Ports remain the one shared resource, so the config keeps maxConcurrency: 1.
• The runner captures the state after the turn (eval-outcome.ts). The checks live in outcome.mjs, with parameters in each case's vars. There is still no LLM grader.
• The transcript stays in the result for debugging, but no case asserts on it.
• Every check has a test. scripts/eval/outcome.test.mjs runs each check in outcome.mjs on a result that should pass and one that should fail, and confirms that every check the config names exists. Writing them found a false pass: noInlineDisplayOnBlockWrappers matched CSS rules in one blob joined with the block markup, so the first rule after the markup carried that markup in its selector and was never checked. It now reads each stylesheet on its own (theme .css files, custom CSS, <style> blocks and theme.json css).
• The push case asks the user for real. With Studio Code evals: Let the agent ask the user, and run on a WordPress.com site #5230 the runner can script the user's answers. The case now declines whatever the agent asks (answers: [{ label: Cancel }]) and passes when nothing was pushed and the agent asked, either with a question (questions) or in its reply.

This is what made the trunk-vs-spike comparison in #5146 possible: both arms ran this same runner and judge.

Testing Instructions

• npm run eval -- --filter-pattern "identifies as|leaves a working new site": both pass on this branch, built from trunk.
• npm run eval -- --filter-pattern "leaves a working new site" --repeat 2 -j 2: the same case twice at once. Both pass, each with its site in its own folder.
• npx vitest run apps/cli/lib/tests/site-paths.test.ts scripts/eval/outcome.test.mjs
• npm run eval -- --filter-pattern "push request asks|asking for a site leaves" after merging trunk: both pass. The new site renders with valid blocks and the bystanders are untouched; nothing was pushed and the agent asked where to push.
• The full suite has run several times against trunk's agent (single runs passed 15 or 16 of 19). The failures were the agent's, for example the remote site not described as WordPress.com in differentiate-preview-vs-remote.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[wpmobilebot]

📊 Performance Test Results

Comparing 29c1f63 vs trunk

app-size

Metric	trunk	29c1f63	Diff	Change
App Size (Mac)	922.00 MB	922.00 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	29c1f63	Diff	Change
load	1202 ms	1208 ms	+6 ms	⚪ 0.0%

site-startup

Metric	trunk	29c1f63	Diff	Change
siteCreation	8577 ms	8572 ms	5 ms	⚪ 0.0%
siteStartup	5896 ms	5890 ms	6 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)