Repository navigation
Studio Code: Bring the remote-site runbook with a WordPress.com active site instead of a rule to load it - #5231
Merged
Merged
Conversation
…e site instead of a rule to load it
chubes4
marked this pull request as ready for review
October 21, 2026 02:05
Collaborator
📊 Performance Test ResultsComparing b03c761 vs trunk app-size
site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 21, 2026
## Related issues - Follows #5231, which brings the remote-site runbook when a WordPress.com site is active. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, and ran the eval below on trunk and on this branch. I reviewed the diff and the transcripts. ## Proposed Changes `site_list` listed only local sites. When the user names one of their WordPress.com sites and no WordPress.com site is active, the agent looked there, found nothing, and concluded the site doesn't exist. Nothing pointed it to WordPress.com, except a line in the remote-site runbook, which is only loaded once the agent already knows the site is remote. - `site_list` now returns `{ local, wordpressCom }`. `wordpressCom` holds the sites the stored login reaches, with the ID `wpcom_request` takes as `siteId`. They come from the same `getWpComSites()` the `/site` picker uses: about 14 KB for 160 sites, versus 2.4 MB for `GET !/me/sites` with `plan`. - Signed out, it says `signedIn: false`. If the lookup fails, it reports the error and still lists the local sites. - Its annotation is now `REMOTE_READ`, like `site_connected_remote_sites`. That tool is still read-only, so hosts run it without asking. - The remote-site runbook points to `site_list` for other sites instead of `site_connected_remote_sites` or `GET !/me/sites`. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/tools.test.ts`: it lists local and WordPress.com sites together, reports a signed-out user, and keeps the local sites when WordPress.com can't be reached. - Eval: two prompts naming a WordPress.com site (a Business plan Atomic site) while no WordPress.com site is active, 4 runs each per arm, with trunk and this branch as two providers in one promptfoo config. | Prompt | Trunk | This branch | |---|---|---| | "Which theme is my Franklin Works site using?" (no active site) | 0/4: asked which site was meant; one run grepped `~/Studio` with Bash | 4/4: `site_list`, then one `wpcom_request` | | "Add a draft post titled 'Site list eval' to my Franklin Works site." (a local site active) | 0/4: offered the local site or a new one; one run used Bash | 4/4: a draft on the right site, without touching the local site | No run on either arm called `GET !/me/sites`. Median tokens went from 37k to 52k (theme) and from 32k to 53k (draft), because the branch finishes the task where trunk stopped to ask. The drafts were trashed afterwards. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
This was referenced Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
seed.remoteActiveSitefrom Studio Code evals: Let the agent ask the user, and run on a WordPress.com site #5230.How AI was used in this PR
Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, ran the eval below against a WordPress.com site I own, and cleaned up the posts it created. I reviewed the diff and the results.
Proposed Changes
The system prompt carried a general rule telling the agent to load the
edit-manage-remote-siteskill first for any WordPress.com site. That rule stood in for a fact Studio already knows: since #5192, the active site is stated in the prompt, including whether it is a WordPress.com site.edit-manage-remote-siterunbook. The agent starts out knowing that the site changes only throughwpcom_requestand how to use that tool, so it no longer spends a step loading the skill first.Skilltool andstudio_instructionslistedit-manage-remote-sitewith its description, "Edit and manage one of the user's live remote sites…".Testing Instructions
npx vitest run apps/cli/ai apps/cli/commands/ai: 256 pass. The new test checks that the runbook is in the prompt only when a WordPress.com site is active.Eval: 24 runs, 12 per arm, on Franklin Works, a Business-plan WordPress.com site. Trunk and this branch both include Studio Code evals: Let the agent ask the user, and run on a WordPress.com site #5230.
wpcom_request; no local tool was called.GET !/me/siteswithplaninfieldsreturns 2.3 MB, andwpcom_requestpasses it through whole. Trunk has the same path. Without that run, the branch's median is 84k.Pre-merge Checklist