[chubes4]

Related issues

• Follows Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts #5301, addressing the review there: the saved Site Editor copies were listed in every system prompt, though they matter only when the agent touches a theme file they replace.
• Builds on Studio MCP: Add the declared settings of core blocks to tool results #5333's shared result notes, so the notice reaches Studio Code and studio mcp the same way.

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) wrote the change and the test and ran the evals below. I reviewed the diff and the results.

Proposed Changes

When someone saves a template, template part or Global Styles in the Site Editor, WordPress renders that saved copy instead of the theme file. An agent that edits parts/header.html then changes nothing the visitor sees. #5092 told the agent this in edit-manage-site; #5301 moved it into Active site facts, which listed every saved copy in every prompt.

• The notice moves into tool results, stated when a call or its result names a theme file a saved copy replaces, for example:

  WordPress renders the template part header from its copy saved in the Site Editor (post 12), not from wp-content/themes/twentytwentyfive/parts/header.html.

• cli/ai/saved-copies.ts reads the copies from the site's SQLite database when a theme file is named, so copies saved earlier in the turn count. Each copy is stated once per tracker.
• It is registered in cli/ai/result-notes.ts (Studio MCP: Add the declared settings of core blocks to tool results #5333). The notes now see the call's arguments with its result (a Write names the file in its arguments), on both surfaces.
• Active site facts keep the WordPress version; saved copies leave them.

Limits:

• MCP clients read and write theme files with their own tools, which never pass through Studio, so for them the notice fires only when a Studio tool names the file.
• Like every result note, a notice stated early in a turn can be lost if the context is compacted later in that turn.

Testing Instructions

• npx vitest run apps/cli/ai/tests/saved-copies.test.ts: a real SQLite database with a saved template part and Global Styles; notices for the theme files they replace, none for other files, and each copy once; run with wp_ tables and with another prefix, as imported sites can have.

• Eval (/tmp override kit, 6 runs per case, Studio's default model): a site whose header part and Global Styles were saved in the Site Editor; the task edits the header, then the styles. Scored from the rendered site.

  Case	Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts #5301 (facts in prompt)	This PR
  Template part saved in the Site Editor	6/6	6/6
  Global Styles saved in the Site Editor	4/4	6/6

  The notice appeared in all 12 transcripts. An earlier run of this change scored 5/6 on Global Styles; the miss was broken JSON the agent wrote to a file and saved with post update, which the inline --post_content JSON check (Studio Code: Refuse wp_cli --post_content JSON that does not parse #5310) does not cover.

• Eval on site edits with no saved copies (8 cases from the routing eval: contact form, carousel, newsletter, heading font, testimonials, header search block, hero, spacing; each run stops at its first write). Before Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts #5301, edit-manage-site told the agent to check for saved copies first; Studio Code: Apply theme template edits to Site Editor customizations instead of silently missing them #5092's review asked what that costs.

  	Sessions that queried saved copies	Queries	Tool calls before the first write
  Before Studio Code: State the active site's WordPress version, block types, and Site Editor copies as facts #5301 (Oct 22 runs)	38/126	43	14.3
  This PR	0/32	0	14.1

  The queries are gone, but calls before the first write are unchanged: the agent spends them on other reading. Tokens (median 134k before, 136k now) and wall time (median 38.6s on trunk, 36.4s now) are at parity too.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[youknowriad]

Thanks, this answers my concern about the prompt, but to be honest, I don't see how this is worth it. your numbers show the old sentence cost nothing measurable (14.3 vs 14.1 calls, same tokens and time) and passed the same evals, so I’d rather keep that one line in edit-manage-site than maintain a SQLite reader and path matching that covers files named exactly. It feels over engineered and too specific.

[chubes4]

For me it's about the philosophy behind the changes. The opportunity to remove a line from the prompt and replace it with code is always worth it, because it is now impossible to have the kind of error that caused the line to be added to the prompt in the first place.

The line in the skill was also extremely specific and didn't need to be there. It also could be ignored by the agent. Some of my earlier evals did show the agent ignoring it.

Also, pushing on this revealed multiple bugs like the ones with --post_content that came up during evals.

By removing prose from our skills and prompts, we can reserve real estate for custom workflows and other things that can't be handled structurally.

[youknowriad]

The opportunity to remove a line from the prompt and replace it with code is always worth it, because it is now impossible to have the kind of error that caused the line to be added to the prompt in the first place.

I'm not sure I'm on board with this policy personally. I think it's always a balance and if removing a code means shipping some code that is too specific to address one edge case, I don't think it's worth it at the moment.

It's possible that I'm wrong but I don't want us in a month to end up with a harness with hundreds of specific fixes like this.

[chubes4]

I'm taking small bites towards an architecture like #5146. These low-risk changes make it easier to ship bigger things later, such as removing validate_blocks in favor of automated validation owned by Studio.

I don't see these as one-off changes, I am creating primitives that share WordPress state with the agent. Over time, this will lead to Studio Code being more efficient, faster, and more reliable. I'm confident that this direction is worth it.

[youknowriad]

In that case, I think it's better to build the whole architecture change in a dedicated branch to be able to compare it without ur current architecture, unless you think that PR is already better than trunk in which case I will have to test it properly.

I need to be convinced that the new architecture is better, I'm not at the moment. We have a lot of generic tools like rest api, wp cli, ... and automating validation in the tools is not something that I think can work reliably for instance, especially with third party blocks (woo, jetpack...). I want to be convinced otherwise though so that's why I think it's better to build the whole vision separately instead of building it bit by bit so we have two mixed visions that fight each other.

[chubes4]

I don't think #5146 is better than trunk in its current state. Fair enough, I'll build out everything I think needs to be done on that branch and let you know when it's ready.

I expected it would take some convincing, which is part of why I started small.