Repository navigation
Studio Code evals: Let the agent ask the user, and run on a WordPress.com site - #5230
Merged
Merged
Conversation
chubes4
changed the title
1 task done
Collaborator
📊 Performance Test ResultsComparing 0163412 vs trunk app-size
site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
1 task
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
…e site instead of a rule to load it (#5231) ## Related issues - Continues the series simplifying the Studio Code system prompt (#5189, #5190, #5192): moves a rule from prose into structure. - The eval uses `seed.remoteActiveSite` from #5230. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, ran the eval below against a WordPress.com site I own, and cleaned up the posts it created. I reviewed the diff and the results. ## Proposed Changes The system prompt carried a general rule telling the agent to load the `edit-manage-remote-site` skill first for any WordPress.com site. That rule stood in for a fact Studio already knows: since #5192, the active site is stated in the prompt, including whether it is a WordPress.com site. - **When a WordPress.com site is active, its runbook comes with it.** The Active site section now includes the `edit-manage-remote-site` runbook. The agent starts out knowing that the site changes only through `wpcom_request` and how to use that tool, so it no longer spends a step loading the skill first. - **The rule is removed from the general rules.** For a WordPress.com site the user names while a local site is active, the skill list already covers it: the `Skill` tool and `studio_instructions` list `edit-manage-remote-site` with its description, "Edit and manage one of the user's live remote sites…". - Prompts for local sites and for no active site are unchanged, apart from one rule fewer. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 256 pass. The new test checks that the runbook is in the prompt only when a WordPress.com site is active. - **Eval:** 24 runs, 12 per arm, on Franklin Works, a Business-plan WordPress.com site. Trunk and this branch both include #5230. - **read:** WordPress.com site active, "Which theme is active on this site, and does its plan let me install plugins?" - **write:** WordPress.com site active, "Add a draft post titled … with one short paragraph about field notes." - **named:** a local site active, the same draft request naming "my WordPress.com site Franklin Works (chubesmattic.wpcomstaging.com)". | Case | Arm | Correct outcome | Loaded the skill | Median tokens | |---|---|---|---|---| | read | trunk | 4/4 | 4/4 | 74k | | read | branch | 4/4 | 0/4 | 66k | | write | trunk | 4/4 | 4/4 | 49k | | write | branch | 4/4 | 1/4 | 38k | | named | trunk | 4/4 | 4/4 | 83k | | named | branch | 4/4 | 4/4 | 102k | - **Correct outcome:** - read: the answer names Assembler and says the Business plan allows plugins. - write and named: exactly one draft with that title on Franklin Works. - In every run, the agent changed the site only through `wpcom_request`; no local tool was called. - **named, branch:** the agent loaded the skill in 4 of 4 runs without the rule. One run took 752k tokens, because `GET !/me/sites` with `plan` in `fields` returns 2.3 MB, and `wpcom_request` passes it through whole. Trunk has the same path. Without that run, the branch's median is 84k. - All 16 drafts were trashed afterwards. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
## Related issues - Related to #5146 ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the outcome capture, the checks, their unit tests and the isolation, and ran the suite on this branch and on the #5146 spike. I reviewed the diff and the results. The checks in `outcome.mjs` set what each case counts as success, so they are the part worth reading closely; each one now has a passing and a failing unit test. ## Proposed Changes - **Judge outcomes, not routes.** The suite asserted which tools the agent called and in what order. That breaks whenever the agent reaches the same result another way: a new tool, a skill, a different WP-CLI command. It also cannot compare two agent designs fairly. Each case now checks what a user would see: - the site the turn leaves behind (rendered home page, PHP errors, block validity in the site's own editor, theme, plugins, pages, templates, palette, saved Site Editor customizations) - the other sites it left alone - what the reply told the user - **Isolate each case.** A case runs against its own site registry, holding two stopped bystander sites the request is not about. The user's real sites, and sites other Studio sessions create on the same machine, stay out of the turn and out of the judgement. A case fails when it changes a bystander. - **Give each case its own sites folder.** Cases used to create their site at `~/Studio/<site name>` and delete that folder before the turn. Two runs of the same case at once, including repeats, deleted each other's site mid-turn. The runner now loads through a small entry (`eval-runner-entry.ts`) that sets a temporary sites root per case before anything reads it. `site-paths.ts` honours it under `E2E`, the same way `well-known-paths.ts` honours `E2E_CLI_CONFIG_PATH`. Eval sites no longer land in `~/Studio`. Ports remain the one shared resource, so the config keeps `maxConcurrency: 1`. - The runner captures the state after the turn (`eval-outcome.ts`). The checks live in `outcome.mjs`, with parameters in each case's vars. There is still no LLM grader. - The transcript stays in the result for debugging, but no case asserts on it. - **Every check has a test.** `scripts/eval/outcome.test.mjs` runs each check in `outcome.mjs` on a result that should pass and one that should fail, and confirms that every check the config names exists. Writing them found a false pass: `noInlineDisplayOnBlockWrappers` matched CSS rules in one blob joined with the block markup, so the first rule after the markup carried that markup in its selector and was never checked. It now reads each stylesheet on its own (theme `.css` files, custom CSS, `<style>` blocks and theme.json `css`). - **The push case asks the user for real.** With #5230 the runner can script the user's answers. The case now declines whatever the agent asks (`answers: [{ label: Cancel }]`) and passes when nothing was pushed and the agent asked, either with a question (`questions`) or in its reply. This is what made the trunk-vs-spike comparison in #5146 possible: both arms ran this same runner and judge. ## Testing Instructions - `npm run eval -- --filter-pattern "identifies as|leaves a working new site"`: both pass on this branch, built from trunk. - `npm run eval -- --filter-pattern "leaves a working new site" --repeat 2 -j 2`: the same case twice at once. Both pass, each with its site in its own folder. - `npx vitest run apps/cli/lib/tests/site-paths.test.ts scripts/eval/outcome.test.mjs` - `npm run eval -- --filter-pattern "push request asks|asking for a site leaves"` after merging trunk: both pass. The new site renders with valid blocks and the bystanders are untouched; nothing was pushed and the agent asked where to push. - The full suite has run several times against trunk's agent (single runs passed 15 or 16 of 19). The failures were the agent's, for example the remote site not described as WordPress.com in `differentiate-preview-vs-remote`. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Claude (via the Pi coding agent in a Roadie session) found the gap while evaluating a change to
site_push, wrote both commits and the README text, and ran the checks below. I reviewed the diff and the results.Proposed Changes
The eval runner gave the agent no way to ask the user anything. The app and
studio aialways passonAskUsertorunStudioAgentTurn, so the agent hasAskUserQuestionand tools can confirm with the user. The runner passed nothing, so every eval so far measured an agent that cannot ask. Anything that depends on a question (confirming a target, picking a design, clarifying a request) was evaluated on a different agent from the one users get. The README already listed aquestionsresult field that the runner never returned.answersvar scripts the user's replies. It is a list of{ "label": "…" }objects, not strings, because promptfoo turns a list of strings into one test per element. With it, the turn runs with an ask-the-user handler: each question takes the next label in order, and once the labels run out the user cancels. Without it, behaviour is unchanged.questions. Each entry holds the question the agent asked, its option labels and the scripted answer. The README line describing it is now true.seed.activeSite.connectedWpcomSitestakes the same fields asseed.connectedWpcomSiteswithoutlocalSiteId, which is the new site's id and only known once the site exists. This lets push and pull flows be evaluated on the active site.seed.remoteActiveSite(name,url,wpcomSiteId) makes an existing site active for the turn, as picking it in the app or CLI does, so the Live chat path can be evaluated. Nothing is created or deleted, so the case owns cleaning up what the agent changes on that site.Testing Instructions
npx vitest run apps/cli/ai apps/cli/commands/ai: 255 pass.Build the runner (
cd apps/cli && npx vite build --config ./vite.config.dev.ts), then run the same prompt with and withoutanswers:answers: the agent callsAskUserQuestion.questionsis[{ "question": "Which color do you prefer?", "options": ["Red", "Blue"], "answer": "Blue" }], and the reply is "You prefer the color blue."answers(same build): no tool call, and the reply starts "I don't have anAskUserQuestiontool available in this session".Used for a 22-run push eval: one connected WordPress.com site, seeded with
activeSite.connectedWpcomSites, and "Push my changes to WordPress.com.". On trunk the agent asked before pushing in 4 of 4 runs, and each question and answer appears inquestions.Pre-merge Checklist