Studio Code: State the active site in the system prompt instead of prepending it to every message by chubes4 · Pull Request #5192 · Automattic/studio

chubes4 · 2026-10-20T00:26:30Z

Related issues

Follows Studio Code: Give the agent its WordPress.com site's plan features instead of describing plan tiers #5170, Studio Code: Drop the prompt rules that repeat the file tools' own guidance #5189, Studio Code: Drop the plan guardrail that repeats the hosting-plans-helper skill #5190: moving Studio Code context from prose into structure, one small step at a time.
First step toward composing site facts the way the system prompt already composes tool snippets: each fact is rendered by the code that owns it, from Studio's own data.

How AI was used

Claude (via the Pi coding agent in a Roadie session) traced how the active site reaches the model, wrote the change and tests, and ran the eval below. I reviewed the diff and the eval traces.

Proposed Changes

The active site was stated by prepending a [Active site: …] line to every user message. So the same fact piled up in the conversation history once per turn, and it was mixed into what the user typed.

This PR states it once, in an ## Active site section of the system prompt instead:

buildSystemPrompt takes activeSite and renders the section. The text is the same as before, minus the brackets: name, path, running/stopped, and the DESIGN.md path when there is one; or ID, URL and "WordPress.com" for a remote site.
The Pi runtime already rebuilds the session, and so the system prompt, on every turn. It now passes the activeSite it was already receiving into buildSystemPrompt, so the section always reflects the current site and running state, including after compaction.
ai and the eval runner stop prepending the line. The user message is exactly what the user typed.
The three skills and the one prompt line that referred to "the active-site line" now say "the Active site section".

Trade-off: when the active site or its running state changes, the system prompt changes, so that one turn misses the prompt cache. Before, each turn added the line to the history instead.

Testing Instructions

Unit tests: npx vitest run apps/cli/ai apps/cli/commands/ai (245 pass).

system-prompt.test.ts: the section for a local site, a WordPress.com site, and a site with a DESIGN.md. It sits above the user's global instructions and is absent without an active site.
commands/ai/tests/index.test.ts: the dispatched message is the typed text, and the runtime gets the site with the daemon's running state.

Eval: four cases that depend on the agent knowing its active site, 2 runs each, this branch against trunk (76f5c92bf), using the fast model:

Case	What passes	Branch	Trunk
Tagline	Sets `blogdescription` and `blog_public` on the active site	2/2	2/2
Plugins	Lists every installed plugin of the active site	2/2	2/2
Admin bar (core file)	Core checksums still verify, and the change lives in `wp-content/`	1/2	1/2
Design system	Reads the seeded DESIGN.md and the new page uses its required CTA text	2/2	2/2

Neither admin-bar failure involves the active site:

On the branch, the agent left core alone and set the height through Additional CSS (wp_update_custom_css_post). That's valid, but the check only looks at files.
On trunk, the model emitted a malformed tool call and the turn ended.

In all 16 runs the agent worked in the correct site directory. The result is parity: the same facts, no longer repeated in the history.

Multi-message eval, using #5193's token usage and followUps: a seeded site and eight messages (tagline, theme, pages, create a draft page, plugins, permalinks, site title), 2 runs per arm. To match the app, the trunk arm adds the site line to every message.

	Trunk (line in every message)	This PR (system-prompt section)
Final state correct (tagline, title, draft page)	2/2	2/2
Site lines saved in the session	8 per run	0
Copies of the line re-sent across the run's requests	73–78	0 (the section is sent once per request)
Total tokens per run	218k, 244k	225k, 214k

The line is about 25 tokens, so this saves roughly 1.5k tokens per eight-message run, under 1% and well inside the run-to-run noise. The gain is in the transcript (it holds what the user typed, and no outdated copies of the site's state), not in the token bill.

Manual:

studio ai with a site selected, then ask "what site are you working on?" The agent answers from the system prompt, and your message in the history has no prefix.
Switch site with /site, ask again: the answer reflects the new site.

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

…epending it to every message

…5193) ## Related issues - Supports the series simplifying the Studio Code system prompt (#5189, #5190, #5192): each change needs an eval, and some changes only show up in token cost or across several messages. ## How AI was used Claude (via the Pi coding agent in a Roadie session) wrote the change, updated the README, and used it to measure #5192. I reviewed the diff and the results. ## Proposed Changes Two additions to `apps/cli/ai/eval-runner.ts`, so prompt changes can be compared on cost and across a conversation, not just on one request's outcome: - **`usage` in the result:** input, output, cache read, cache write and total tokens, summed from every assistant message. Promptfoo assertions can report it next to pass/fail. - **`followUps` var:** a list of further user messages (`{ "prompt": "…" }` objects, since promptfoo expands an array of strings into one test case per element), each sent in the same session once the previous one finishes (including any `STUDIO_EVAL_AUTO_CONTINUE` resumes). The timeout covers the whole conversation. - **The transcript, every run:** the session JSONL is always written (to `<tmpdir>/studio-eval-sessions`, or `STUDIO_EVAL_SESSION_DIR`), and the result reports its path as `sessionFile`. Before, it was kept only when that variable was set. - **`stopReason` and `errorMessage`** from the last assistant message. A provider failure ends the turn without throwing, so the result used to say `success: false` with `error: null`. These supersede #3376's diagnostics. Its transcript, final text and last-message usage are covered by the above and by `textSegments`/`timedOut`, and its "no useful output" heuristic is a judgment that belongs in each case's assertions. The README documents all of it. ## Testing Instructions 1. `cd apps/cli && npx vite build --config ./vite.config.dev.ts` 2. `node dist/cli/eval-runner.mjs "What's this site's tagline?" "" '{"vars":{"followUps":[{"prompt":"Which theme is active?"}]}}'` 3. In the result file, expect a `usage` object with non-zero counts, `stopReason: "stop"`, both answers in `textSegments`, and a `sessionFile` that exists and holds both messages. I ran it as the measurement for #5192: a seeded site and eight follow-up messages per run (results on that PR). ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…ite-in-system-prompt

…states it

wpmobilebot · 2026-10-20T02:20:53Z

📊 Performance Test Results

Comparing b3fbde0 vs trunk

app-size

Metric	trunk	`b3fbde0`	Diff	Change
App Size (Mac)	927.54 MB	927.54 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	`b3fbde0`	Diff	Change
load	1166 ms	1161 ms	5 ms	⚪ 0.0%

site-startup

Metric	trunk	`b3fbde0`	Diff	Change
siteCreation	8561 ms	8599 ms	+38 ms	⚪ 0.0%
siteStartup	5900 ms	5896 ms	4 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

… installed skills (#5195) ## Related issues - Follows #5192, #5190 and #5189: drop instructions that repeat something the agent already gets. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) traced how skills reach a site, found the stale list, wrote the change, and drafted this description. I reviewed the change and the reasoning. ## Proposed Changes `STUDIO.md`'s "Available Skills" section lists six skills as installed in every site. Only the skills a user picks in **Settings → Skills** are installed (`installAiInstructionsToSite` copies just `selectedSkills`), and they can be added or removed later. So the list often names skills the site doesn't have, and it's the same for every site. Agents don't need it: Claude Code finds skills in `.claude/skills/` and Codex in `.agents/skills/`, each from the skill's own name and description, and they see exactly what's installed. This removes the section: - **The list** goes. Agents discover the installed skills themselves. - **The `studio wp` prefix note** goes. It repeats the file's first section, "WP-CLI in WordPress Studio". - **The `wp-project-triage` note** stays, moved under "Studio-Specific Notes", since nothing else says it. Net: 16 lines removed, 1 added. `STUDIO.md` is a managed file, so existing sites get the change the next time they start (`updateManagedInstructionFiles`). ## Testing Instructions 1. In Settings → Skills, install only **Studio CLI**, then create a site. 2. Open the site's `STUDIO.md`: no "Available Skills" section, and the triage note sits under "Studio-Specific Notes". 3. `ls <site>/.claude/skills <site>/.agents/skills` shows only `studio-cli`, which is what agents discover. On trunk, `STUDIO.md` also names the five WordPress skills that aren't there. `npx vitest run packages/common/lib/tests/agent-skills.test.ts` passes (17). No eval: this file isn't loaded by Studio Code. It only reaches external agents opened in a site folder. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?

….com site (#5230) ## Related issues - Supports the series simplifying the Studio Code system prompt (#5189, #5190, #5192): each change gets an eval, and changes that move a rule from prose into a tool often move it into a question to the user. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found the gap while evaluating a change to `site_push`, wrote both commits and the README text, and ran the checks below. I reviewed the diff and the results. ## Proposed Changes **The eval runner gave the agent no way to ask the user anything.** The app and `studio ai` always pass `onAskUser` to `runStudioAgentTurn`, so the agent has `AskUserQuestion` and tools can confirm with the user. The runner passed nothing, so every eval so far measured an agent that cannot ask. Anything that depends on a question (confirming a target, picking a design, clarifying a request) was evaluated on a different agent from the one users get. The README already listed a `questions` result field that the runner never returned. - **An `answers` var scripts the user's replies.** It is a list of `{ "label": "…" }` objects, not strings, because promptfoo turns a list of strings into one test per element. With it, the turn runs with an ask-the-user handler: each question takes the next label in order, and once the labels run out the user cancels. Without it, behaviour is unchanged. - **The result reports `questions`.** Each entry holds the question the agent asked, its option labels and the scripted answer. The README line describing it is now true. - **The seeded active site can have connected WordPress.com sites.** `seed.activeSite.connectedWpcomSites` takes the same fields as `seed.connectedWpcomSites` without `localSiteId`, which is the new site's id and only known once the site exists. This lets push and pull flows be evaluated on the active site. - **One of the user's WordPress.com sites can be the active site.** `seed.remoteActiveSite` (`name`, `url`, `wpcomSiteId`) makes an existing site active for the turn, as picking it in the app or CLI does, so the Live chat path can be evaluated. Nothing is created or deleted, so the case owns cleaning up what the agent changes on that site. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 255 pass. - Build the runner (`cd apps/cli && npx vite build --config ./vite.config.dev.ts`), then run the same prompt with and without `answers`: ```sh node apps/cli/dist/cli/eval-runner.mjs x "" '{"vars":{"prompt":"Use AskUserQuestion to ask me which color I prefer, with options Red and Blue. Then tell me my answer in one sentence.","answers":[{"label":"Blue"}]}}' ``` - With `answers`: the agent calls `AskUserQuestion`. `questions` is `[{ "question": "Which color do you prefer?", "options": ["Red", "Blue"], "answer": "Blue" }]`, and the reply is "You prefer the color blue." - Without `answers` (same build): no tool call, and the reply starts "I don't have an `AskUserQuestion` tool available in this session". - Used for a 22-run push eval: one connected WordPress.com site, seeded with `activeSite.connectedWpcomSites`, and "Push my changes to WordPress.com.". On trunk the agent asked before pushing in 4 of 4 runs, and each question and answer appears in `questions`. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…and #5230

…e site instead of a rule to load it (#5231) ## Related issues - Continues the series simplifying the Studio Code system prompt (#5189, #5190, #5192): moves a rule from prose into structure. - The eval uses `seed.remoteActiveSite` from #5230. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, ran the eval below against a WordPress.com site I own, and cleaned up the posts it created. I reviewed the diff and the results. ## Proposed Changes The system prompt carried a general rule telling the agent to load the `edit-manage-remote-site` skill first for any WordPress.com site. That rule stood in for a fact Studio already knows: since #5192, the active site is stated in the prompt, including whether it is a WordPress.com site. - **When a WordPress.com site is active, its runbook comes with it.** The Active site section now includes the `edit-manage-remote-site` runbook. The agent starts out knowing that the site changes only through `wpcom_request` and how to use that tool, so it no longer spends a step loading the skill first. - **The rule is removed from the general rules.** For a WordPress.com site the user names while a local site is active, the skill list already covers it: the `Skill` tool and `studio_instructions` list `edit-manage-remote-site` with its description, "Edit and manage one of the user's live remote sites…". - Prompts for local sites and for no active site are unchanged, apart from one rule fewer. ## Testing Instructions - `npx vitest run apps/cli/ai apps/cli/commands/ai`: 256 pass. The new test checks that the runbook is in the prompt only when a WordPress.com site is active. - **Eval:** 24 runs, 12 per arm, on Franklin Works, a Business-plan WordPress.com site. Trunk and this branch both include #5230. - **read:** WordPress.com site active, "Which theme is active on this site, and does its plan let me install plugins?" - **write:** WordPress.com site active, "Add a draft post titled … with one short paragraph about field notes." - **named:** a local site active, the same draft request naming "my WordPress.com site Franklin Works (chubesmattic.wpcomstaging.com)". | Case | Arm | Correct outcome | Loaded the skill | Median tokens | |---|---|---|---|---| | read | trunk | 4/4 | 4/4 | 74k | | read | branch | 4/4 | 0/4 | 66k | | write | trunk | 4/4 | 4/4 | 49k | | write | branch | 4/4 | 1/4 | 38k | | named | trunk | 4/4 | 4/4 | 83k | | named | branch | 4/4 | 4/4 | 102k | - **Correct outcome:** - read: the answer names Assembler and says the Business plan allows plugins. - write and named: exactly one draft with that title on Franklin Works. - In every run, the agent changed the site only through `wpcom_request`; no local tool was called. - **named, branch:** the agent loaded the skill in 4 of 4 runs without the rule. One run took 752k tokens, because `GET !/me/sites` with `plan` in `fields` returns 2.3 MB, and `wpcom_request` passes it through whole. Trunk has the same path. Without that run, the branch's median is 84k. - All 16 drafts were trashed afterwards. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

Studio Code: State the active site in the system prompt instead of pr…

1bdd11c

…epending it to every message

github-actions Bot assigned chubes4 Oct 20, 2026

Eval README: activeSite seeds the system prompt's Active site section

52c6c83

chubes4 mentioned this pull request Oct 20, 2026

Studio Code evals: Report token usage and support follow-up messages #5193

Merged

1 task done

chubes4 added 2 commits October 19, 2026 21:43

Merge remote-tracking branch 'origin/trunk' into studio-code/active-s…

55c3d1d

…ite-in-system-prompt

Merge trunk; follow-ups need no site line now that the system prompt …

b3fbde0

…states it

chubes4 marked this pull request as ready for review October 20, 2026 02:00

chubes4 merged commit a9832e9 into trunk Oct 20, 2026
12 checks passed

chubes4 deleted the studio-code/active-site-in-system-prompt branch October 20, 2026 02:32

chubes4 mentioned this pull request Oct 20, 2026

STUDIO.md: Drop the skills list that agents already discover from the installed skills #5195

Merged

1 task

This was referenced Oct 21, 2026

Studio Code evals: Let the agent ask the user, and run on a WordPress.com site #5230

Merged

Studio Code: Bring the remote-site runbook with a WordPress.com active site instead of a rule to load it #5231

Merged

chubes4 added a commit that referenced this pull request Oct 21, 2026

Merge trunk: the active site and scripted answers now come from #5192 …

ae47af2

…and #5230

This was referenced Oct 21, 2026

Studio Code: Keep the agent's identity in SOUL.md #5257

Closed

Studio Code: Load pi extensions from Studio's agent directory #5260

Closed

Navigation Menu

Repository navigation

Studio Code: State the active site in the system prompt instead of prepending it to every message - #5192

Studio Code: State the active site in the system prompt instead of prepending it to every message#5192
chubes4 merged 4 commits into
trunkfrom
studio-code/active-site-in-system-prompt

chubes4 commented Oct 20, 2026 •

edited

Loading

Uh oh!

wpmobilebot commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

chubes4 commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues

How AI was used

Proposed Changes

Testing Instructions

Pre-merge Checklist

Uh oh!

wpmobilebot commented Oct 20, 2026

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chubes4 commented Oct 20, 2026 •

edited

Loading