Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts by chubes4 · Pull Request #5301 · Automattic/studio

chubes4 · 2026-10-22T21:19:18Z

Related issues

Builds on Studio Code: Apply theme template edits to Site Editor customizations instead of silently missing them #5092 (STU-2155, Wojtek). When a block theme's template, template part, or Global Styles have a copy saved in the Site Editor, WordPress uses the saved copy instead of the theme's file. Studio Code: Apply theme template edits to Site Editor customizations instead of silently missing them #5092 taught the agent this with one sentence in edit-manage-site (moved there from validate_blocks in review). This PR states the saved copies as facts read from the site, and deletes the sentence.
Builds on Studio Code: Show a core block's declared settings in tool results that show the block #5307, which shows each core block's declared settings in the tool results that show the block. With that in trunk, this PR no longer points the agent at block.json.

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, and ran the evaluation below. I reviewed the diff and the eval transcripts.

Proposed Changes

The Active site section now lists facts about a local site, read from its own files and database (site-facts.ts). For the STU-2155 header case it reads:

"Eval Template Override" at /…/eval-template-override (running)
WordPress 7.1.3.
Saved in the Site Editor, which WordPress uses instead of the theme's files: template part header (post 5).

A site with more saved copies gets one grouped line per kind, about 6 tokens per copy:

Saved in the Site Editor, which WordPress uses instead of the theme's files: templates index (post 12), single (13); template part header (post 11). Only in the Site Editor: template page-about (post 14); template part promo (post 15). Global Styles (post 16), applied over theme.json, set color, typography, 2 block styles, and color settings.

WordPress version: read from the site's wp-includes/version.php.
Copies saved in the Site Editor: the latest published wp_template and wp_template_part post per slug for the active theme, read read-only from the site's SQLite database with Node's built-in node:sqlite. Each copy is grouped by whether it replaces a file in the theme or its parent theme (checked on disk, in the folders WordPress reads them from) or exists only in the Site Editor.
Global Styles: the wp_global_styles post, with the areas it sets and a count of per-block styles. The details are in the post.
A site without a readable database gets no line.
Each line appears only when it applies. The prompt is rebuilt every turn, so a copy saved mid-session shows up on the next turn. WordPress.com sites get no facts.
The edit-manage-site sentence that told the agent to look for saved copies with post list is deleted.

Testing Instructions

npx vitest run apps/cli/ai passes (332). The tests use a real SQLite database and theme files:
- templates, template parts, and Global Styles are found and grouped, including copies a parent theme's file backs and copies only in the Site Editor;
- the latest copy per slug is used;
- other themes, drafts, and empty Global Styles are ignored;
- a site without a database gets no line;
- the version fact and the Active site rendering are covered.
I checked against a real seeded site that the facts reach the prompt: the agent quoted the section above.
Eval: Wojtek's two cases, judged by their own assertions. The baseline is current trunk (aabc42a3e, with the sentence and Studio Code: Show a core block's declared settings in tool results that show the block #5307), from the trunk-plus-Studio Code: Show a core block's declared settings in tool results that show the block #5307 arm of Studio Code: Show a core block's declared settings in tool results that show the block #5307's eval, which ran the same code. The branch is this PR merged with that trunk.

	Trunk (with the sentence)	This branch (without it)
Template part (`template-part-db-override`)	6/6, 223k tokens, 17 calls	6/6, 167k tokens, 13 calls
Global styles (`global-styles-db-override`)	6/6, 189k tokens	4/4, 87k tokens in 3 runs, 1.4M in one

That's parity. Every branch run edited the saved copy (post 5) and set submenuVisibility there, and the user's tagline was kept in all six. The agent quoted the fact in its reasoning ("the description said 'Saved in the Site Editor … template part header (post 5).' So post 5 is authoritative").
One global-styles run passed after 57 tool calls, most of them checking why the saved background didn't render at first. The saved copy itself was found from the fact.
Earlier rounds of this PR, before Studio Code: Read the file a wp_cli --post_content path names #5305 and Studio Code: Show a core block's declared settings in tool results that show the block #5307 were in trunk, failed two of six template-part runs on the navigation's click setting and one on a --post_content=<path> save. Studio Code: Show a core block's declared settings in tool results that show the block #5307 and Studio Code: Read the file a wp_cli --post_content path names #5305 fix those, and neither failure appears here.

Pre-merge Checklist

Have you checked for TypeScript, React or other console errors?

…urce, and Site Editor copies as facts

…or sentence it replaces

…rize Global Styles by area

wpmobilebot · 2026-10-22T23:50:28Z

📊 Performance Test Results

Comparing b5e5d7e vs trunk

app-size

Metric	trunk	`b5e5d7e`	Diff	Change
App Size (Mac)	940.68 MB	940.68 MB	+0.00 MB	⚪ 0.0%

site-editor

Metric	trunk	`b5e5d7e`	Diff	Change
load	1251 ms	1230 ms	21 ms	⚪ 0.0%

site-startup

Metric	trunk	`b5e5d7e`	Diff	Change
siteCreation	7510 ms	7538 ms	+28 ms	⚪ 0.0%
siteStartup	4370 ms	4383 ms	+13 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

## Related issues - Found in the STU-2155 header eval (#5092, #5301). An agent ran `post update 5 --post_content=<site>/tmp/header-new.html`. That saved the path itself as the template part's content, which wiped the header the site renders, and the agent reported success. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, and ran the checks below. I reviewed the diff and the tool output. ## Proposed Changes WP-CLI's `--post_content` takes the content itself; `post create <file>` and `post update <id> <file>` read it from a file. When `post create` or `post update` gets a `--post_content` value that is the path of a file in the site folder (absolute or relative to it), `wp_cli` now runs the command with that file as the positional argument, so the post gets the file's content. The result says so: > Read the post content from tmp/y.html, the file `--post_content` named. > Success: Updated post 6. - Unquoted content runs to the end of the command, so flags after the path (`--porcelain`) are split back out and kept. - Content that only looks like a path is saved as written, as before: a missing file, a folder, a file outside the site folder, a path inside text, or anything with markup. - Only `post create` and `post update` change. Every other command runs as written. - Nothing new becomes possible: the rewritten command is one the agent could already run. WP-CLI's `post update` reads the first non-numeric argument as the file, so several IDs still work. - `--post_content=$(cat file)` is still refused: shell syntax doesn't run, and that's a different mistake. ## Testing Instructions - `npx vitest run apps/cli/lib/tests/wp-cli-post-content.test.ts apps/cli/ai` passes (289). New tests cover an absolute path, a relative path with flags around it, five values that look like paths but aren't files, files outside the site folder, and other commands. - Real site. The agent wrote `tmp/x.html` and `tmp/y.html`, then ran: - `post create --post_type=page --post_title=Probe --post_status=publish --post_content=<absolute path of x.html> --porcelain`: page 6 created. - `post update 6 --post_content=tmp/y.html`: page 6 updated. - Page 6's content was then `y.html`'s block markup. Without this change, the first command saves `<path> --porcelain` as the content. - No telemetry changes: the note goes to the tool result only. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…ite-facts

…at show the block (#5307) ## Related issues - Builds on #5300 (attribute check) and #5301 (Active site facts). Found while evaluating #5301. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) read the failing eval transcripts, wrote the change and its tests, and ran the evals below. I reviewed the code and the transcripts. ## Proposed Changes When a tool result shows block markup, Studio Code adds one line for each core block type in it, listing the settings that block type declares in the site's own `wp-includes/blocks/<block>/block.json`. For example: > `core/navigation settings, declared in …/wp-includes/blocks/navigation/block.json: ref (number), …, showSubmenuIcon (boolean, default true), submenuVisibility ("hover"|"click"|"always", default "hover"), overlayMenu (string, default "mobile"), …` - Each block type is described once per turn. Attributes read from the block's HTML (`source`) are content, so they are left out, along with internal `__unstable`/`__experimental` ones. - It applies to every Studio tool result, so `wp_cli post get`, `Read` and `Bash cat` all get it. - It reads files on disk only. Plugin blocks, remote sites and blocks without a `block.json` are skipped. **Why.** In the template-part eval ("Make the header navigation open its submenus on click instead of on hover"), the failing runs reason "there's no built-in 'open on click' toggle … this requires custom JavaScript", then write a script and CSS into the theme, at 1–1.7M tokens per run. The navigation block declares `submenuVisibility: "click"`. The model doesn't remember it and has no reason to open `block.json`. Stating the declared settings where the block appears means the model reads them while it is looking at the block. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/block-settings.test.ts`: describes the core blocks in a header from their `block.json`, describes each one once, and leaves out content attributes, closing comments and blocks without a `block.json`. - **Eval:** `template-part-db-override`, 6 runs per arm, run against this branch and current trunk (`09253bcef`) at the same time: | | Pass | JS workaround | Mean tokens | Mean tool calls | Mean time | |---|---|---|---|---|---| | This branch | 6/6 | 0 | 223k | 17 | 53s | | Trunk | 4/5 | 1 | 675k | 32 | 94s | One trunk run stalled at the 420s timeout after 91k tokens without reaching an answer. It is left out above as an infrastructure stall. Counted as a failure, trunk is 4/6. - Every branch run's session shows the `core/navigation settings` line in the `post get 5` or `Read parts/header.html` result, and every branch run set `submenuVisibility` in the saved header. No branch run wrote JavaScript or read core's `navigation/view.js`. On trunk, one run wrote a script and another read `view.js` three times before finding the setting. - **Regression check:** the three other cases that start on a local site, 6 runs per arm: | Case | This branch | Trunk | |---|---|---| | `global-styles-db-override` | 6/6, 189k tokens | 5/6, 187k tokens | | `design-system-annotation` | 6/6, 260k tokens | 5/6, 289k tokens | | `active-site-reused` | 6/6, 68k tokens | 6/6, 66k tokens | Both trunk failures are unrelated to this change: one is a port collision during inspection, the other discarded the saved styles. The settings line appeared in only 1 of these 18 runs (it passed), because these tasks rarely read block markup, so this check shows no harm rather than a gain. - **Size:** replaying 263 earlier sessions that read block markup, the added lines come to a median of ~240 tokens per session (p90 ~460, max ~1.8k). - Six runs per arm can't establish a pass-rate gain, so the claim is parity or better on accuracy. The fall in tokens and tool calls is large and consistent across runs. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?

…ite-facts

…shown with the blocks

youknowriad · 2026-10-23T10:12:35Z

I don't understand this PR to be honest. the facts about a given sites are infinite and can change over time. While I understand the WordPress version, all the rest feel like useless information in the context most of the time.

chubes4 · 2026-10-23T11:11:44Z

It's a step towards removing skill routing from the system prompt

youknowriad · 2026-10-23T11:35:12Z

How so?

chubes4 · 2026-10-23T12:32:26Z

It's leading towards the goal of reducing non-deterministic behavior and simplifying our system prompts to make the agent more fluid and accurate.

When running evals for #5265, I found that the agent often skipped skills, or would pick edit-manage-site over the more focused skills. My plan is to remove edit-manage-site to unblock this. That means moving more of those skill responsibilities into structure. This should cause the agent to naturally reach for the more focused skills.

You are right that this can be useless context most of the time though. I think a better idea is to move the saved Site Editor copy notice into the tool results similarly to how #5307 handles core block context.

youknowriad · 2026-10-23T12:54:26Z

FYI edit-manage-site is a for the plugins / mcp though. It's a recent addition specifically targeted towards these plugins that expose high level skills.

…all touches the theme file they replace (#5343) ## Related issues - Follows #5301, addressing the review there: the saved Site Editor copies were listed in every system prompt, though they matter only when the agent touches a theme file they replace. - Builds on #5333's shared result notes, so the notice reaches Studio Code and `studio mcp` the same way. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the change and the test and ran the evals below. I reviewed the diff and the results. ## Proposed Changes When someone saves a template, template part or Global Styles in the Site Editor, WordPress renders that saved copy instead of the theme file. An agent that edits `parts/header.html` then changes nothing the visitor sees. #5092 told the agent this in `edit-manage-site`; #5301 moved it into Active site facts, which listed every saved copy in every prompt. - The notice moves into tool results, stated when a call or its result names a theme file a saved copy replaces, for example: > WordPress renders the template part header from its copy saved in the Site Editor (post 12), not from wp-content/themes/twentytwentyfive/parts/header.html. - `cli/ai/saved-copies.ts` reads the copies from the site's SQLite database when a theme file is named, so copies saved earlier in the turn count. Each copy is stated once per tracker. - It is registered in `cli/ai/result-notes.ts` (#5333). The notes now see the call's arguments with its result (a Write names the file in its arguments), on both surfaces. - Active site facts keep the WordPress version; saved copies leave them. Limits: - MCP clients read and write theme files with their own tools, which never pass through Studio, so for them the notice fires only when a Studio tool names the file. - Like every result note, a notice stated early in a turn can be lost if the context is compacted later in that turn. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/saved-copies.test.ts`: a real SQLite database with a saved template part and Global Styles; notices for the theme files they replace, none for other files, and each copy once; run with `wp_` tables and with another prefix, as imported sites can have. - Eval (`/tmp` override kit, 6 runs per case, Studio's default model): a site whose header part and Global Styles were saved in the Site Editor; the task edits the header, then the styles. Scored from the rendered site. | Case | #5301 (facts in prompt) | This PR | |---|---|---| | Template part saved in the Site Editor | 6/6 | 6/6 | | Global Styles saved in the Site Editor | 4/4 | 6/6 | The notice appeared in all 12 transcripts. An earlier run of this change scored 5/6 on Global Styles; the miss was broken JSON the agent wrote to a file and saved with `post update`, which the inline `--post_content` JSON check (#5310) does not cover. - Eval on site edits with no saved copies (8 cases from the routing eval: contact form, carousel, newsletter, heading font, testimonials, header search block, hero, spacing; each run stops at its first write). Before #5301, `edit-manage-site` told the agent to check for saved copies first; #5092's review asked what that costs. | | Sessions that queried saved copies | Queries | Tool calls before the first write | |---|---|---|---| | Before #5301 (Oct 22 runs) | 38/126 | 43 | 14.3 | | This PR | 0/32 | 0 | 14.1 | The queries are gone, but calls before the first write are unchanged: the agent spends them on other reading. Tokens (median 134k before, 136k now) and wall time (median 38.6s on trunk, 36.4s now) are at parity too. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?

Studio Code: State the active site's WordPress version, block type so…

c5ee3fc

…urce, and Site Editor copies as facts

github-actions Bot assigned chubes4 Oct 22, 2026

chubes4 added 2 commits October 22, 2026 17:28

Keep only the fact, drop the file-tool note, and delete the Site Edit…

9485f25

…or sentence it replaces

Group the saved copies, mark those only in the Site Editor, and summa…

a5d9673

…rize Global Styles by area

chubes4 marked this pull request as ready for review October 22, 2026 23:33

chubes4 mentioned this pull request Oct 22, 2026

Studio Code: Read the file a wp_cli --post_content path names #5305

Merged

1 task done

Merge remote-tracking branch 'origin/trunk' into studio-code/active-s…

8b010ac

…ite-facts

chubes4 mentioned this pull request Oct 23, 2026

Studio Code: Show a core block's declared settings in tool results that show the block #5307

Merged

1 task done

chubes4 added 3 commits October 22, 2026 22:27

Merge remote-tracking branch 'origin/trunk' into studio-code/active-s…

a6e46b6

…ite-facts

Studio Code: Drop the block.json pointer from Active site facts, now …

f419a7a

…shown with the blocks

Studio Code: Use WordPress 7.1.3 in the Active site facts version test

b5e5d7e

chubes4 merged commit e32085b into trunk Oct 23, 2026
13 checks passed

chubes4 deleted the studio-code/active-site-facts branch October 23, 2026 04:47

youknowriad mentioned this pull request Oct 23, 2026

Studio Code: Refuse a file path given as wp_cli --post_content instead of rewriting the command #5327

Merged

1 task done

chubes4 mentioned this pull request Oct 23, 2026

Studio Code: State saved Site Editor copies in tool results, when a call touches the theme file they replace #5343

Merged

1 task

youknowriad mentioned this pull request Oct 23, 2026

Studio Code: Revert active site facts and saved Site Editor copy notes #5361

Merged

1 task

Navigation Menu

Repository navigation

Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts - #5301

Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts#5301
chubes4 merged 7 commits into
trunkfrom
studio-code/active-site-facts

chubes4 commented Oct 22, 2026 •

edited

Loading

Uh oh!

wpmobilebot commented Oct 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

youknowriad commented Oct 23, 2026

Uh oh!

chubes4 commented Oct 23, 2026

Uh oh!

youknowriad commented Oct 23, 2026

Uh oh!

chubes4 commented Oct 23, 2026

Uh oh!

youknowriad commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

chubes4 commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues

How AI was used in this PR

Proposed Changes

Testing Instructions

Pre-merge Checklist

Uh oh!

wpmobilebot commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

Uh oh!

youknowriad commented Oct 23, 2026

Uh oh!

chubes4 commented Oct 23, 2026

Uh oh!

youknowriad commented Oct 23, 2026

Uh oh!

chubes4 commented Oct 23, 2026

Uh oh!

youknowriad commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chubes4 commented Oct 22, 2026 •

edited

Loading

wpmobilebot commented Oct 22, 2026 •

edited

Loading