Repository navigation
Conversation
chubes4
force-pushed
the
spike/read-core-source-agent
branch
4 times, most recently
from
October 16, 2026 15:56
23ed4cf to
62b23a8
Compare
This was referenced Oct 16, 2026
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
## Related issues - Related to #5146, where the agent eval surfaced it ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found this while comparing Studio Code agent runs, traced it to the code, wrote the change and its test, and ran the verification below. Opened as a draft for human review. ## Proposed Changes - Gutenberg logs block validation problems as console-style format strings, for example `Expected attribute \`%s\` of value \`%s\`, saw \`%s\`.`, followed by the values. `validate_blocks` joined these pieces as they came, so the agent read the template with its placeholders still in it and the values tacked on afterwards. - Placeholders are now filled in from the values that follow them, so each issue reads as one sentence. Before: ``` Expected attribute `%s` of value `%s`, saw `%s`. class wp-block-heading has-text-color wp-block-heading ``` After: ``` Expected attribute `class` of value `wp-block-heading has-text-color`, saw `wp-block-heading`. ``` ## Testing Instructions - With a site running, call `validate_blocks` on `<!-- wp:heading {"style":{"color":{"text":"#faf8f5"}}} --><h2 class="wp-block-heading" style="color:#faf8f5">Hi</h2><!-- /wp:heading -->`. The issue reads like the "After" output above. This was checked against a running site's editor. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
This was referenced Oct 16, 2026
Merged
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
## Related issues - Related to #5146, where the agent eval surfaced it ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) found this while comparing Studio Code agent runs, traced it to the code, wrote the change and its test, and ran the verification below. Opened as a draft for human review. ## Proposed Changes - When a connected WordPress.com site is no longer in the user's site list, `withWpcomDetails` marks it `deleted`, but it also blanks out its name and URL. Both the Studio app's connected-sites list and the agent's `site_connected_remote_sites` tool then show an unnamed site, so the user can't tell which connection is broken. - The site now keeps the name and URL it was connected under, along with its `deleted` status. ## Testing Instructions - `npx vitest run packages/common/lib/sync`: the test now expects a gone site to keep its stored name and URL. - Manually: connect a site, then remove it from the account, or edit its ID in `shared.json` to one that doesn't exist. Its row keeps the original name and URL. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
Closed
1 task
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
…fusals already carry (#5162) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) went through the system prompt rule by rule looking for rules a tool can enforce, then wrote the change and its tests and ran the checks below. Opened as a draft for human review. ## Proposed Changes - The Working cadence section listed the Write, Edit and Bash payload limits and warned against heredocs and Python scripts. The tools already enforce all of this: an oversized call is refused, and the refusal says how to split the work (a skeleton plus Edit calls) and not to fall back to heredocs or scripts. - That prompt text is removed, cutting the prompt by about 340 characters. The ~8KB-per-Edit target the agent actually uses is in the `block-content` skill's recipes, where the long files are written. - `STUDIO_EDIT_CALL_TARGET_BYTES` was only used by that text, so it is removed too. ## Testing Instructions - `npx vitest run apps/cli/ai` passes. The pi runtime tests still check that oversized Write, Edit and Bash calls are refused. Targeted eval (one prompt asking for a ~24 KB plugin file, judged on the resulting site): with this change merged alongside #5163–#5165, the first oversized `Write` was refused on every run, and the agent built the file with smaller edits, as on trunk. Pass rate matched trunk (1/2 on each arm; the failing run on both arms is the same assertion issue, unrelated to this change). ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
chubes4
added a commit
that referenced
this pull request
Oct 16, 2026
…#5165) ## Related issues - Related to #5146: one of a series of small changes that move system prompt rules into tool structure ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) went through the system prompt rule by rule looking for rules a tool can enforce, then wrote the change and its tests and ran the checks below. Opened as a draft for human review. ## Proposed Changes - In a terminal session, the prompt carried a paragraph saying the user may not see screenshots, so the agent should not reply as if they had ("Here's your site!"). - `take_screenshot` now states that fact in its result, at the moment it matters: "The user does not see this capture: the terminal shows them only the saved file path." The desktop app (which shows the capture) and MCP clients (which get image links to show) don't get the line. - The paragraph is removed from the prompt, cutting the terminal prompt by about 420 characters. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/tools.test.ts -t "saved file path"` checks which sessions get the line. Targeted eval ("Take a screenshot of this site's homepage and show it to me", run in the terminal, judged on the final reply): with this change merged alongside #5162–#5164, the agent took the screenshot and pointed the user to the saved file on 2/2 runs, and never claimed the image was shown. This matches trunk (2/2) without the prompt rule. ## Pre-merge Checklist - [ ] Have you checked for TypeScript, React or other console errors?
1 task
…of re-teaching WordPress in skills Spike. The system prompt gains a 'WordPress source is the reference' section; block-content keeps only Studio mechanics (validate_blocks gate, core/html policy, skeleton-first recipes); visual-polish keeps its diagnose/fix/re-check method and drops the WordPress layout/button explainers; visual-design, build-site and edit-manage-site drop their WordPress how-it-works bullets. The eval runner records token usage and cost per run.
…s, with cost and core-read summary
…instructions Delete the block-content and visual-polish skills; the validate_blocks gate, core/html policy, and one-batch polish loop move into the system prompt and build-site as a few lines. Cut plugin-recommendations and imagery to the decisions and gotchas the source cannot show, the system prompt's cadence and skill routing to an index, and the validate_blocks and inspect_design descriptions to their contracts. The design flow (pick_design, DESIGN.md, Site Spec) is unchanged.
…prompt Write and Edit now refuse WordPress core files and installed third-party themes (pointing at scaffold_theme parentTheme), and report when a Site Editor copy of the template or part, or saved Global Styles, overrides the theme file just written. Theme CSS gets checked for inline-level display on block classNames and box styles on the button wrapper, citing the core code that explains each. validate_blocks rewrites bare #anchor links in template parts and shared templates to /#anchor. The system prompt drops the rules these now enforce.
Writing a template, part, or tmp/ page file queues it in a block validation gate. Validation runs in the background against the site's block editor, and wp_cli, take_screenshot, and inspect_design wait for it. The gate saves each file through the editor when that keeps its text, classes, and attributes, so the front end renders what the block attributes say even for blocks valid only through a deprecated save. It reports only what the model must change, and refuses a wp_cli command that would publish a file with open problems. Theme and plugin commands invalidate earlier results. The validate_blocks tool and every prompt and skill line about it are gone. The eval accepts the gate's reports as validation evidence.
…ools The system prompt now states identity and environment only. What it used to instruct moves into structure: - Skills declare their tools in frontmatter; those tools join the model's tool list when the skill loads, and stay for the conversation. - The harness reloads the Studio preview after a turn that changed the site; refresh_browser is removed. - Bash refuses WP-CLI and names wp_cli. - take_screenshot reports when the user cannot see the capture. - Hosting and domain policy moves to the hosting-plans-helper skill. The tool list, skill routing, cadence, and size limits are dropped from the prompt: tool definitions, the Skill index, and the payload guards already carry them. promptSnippet had no consumer left and is removed.
…quest writes on them
The active site line for a WordPress.com site now states what its plan
includes and lacks and the user's role there, read from plan.features.active
and capabilities on /sites/{id}. wpcom_request refuses writes the site does
not allow (Global Styles, custom CSS, plugin installs and management, theme
uploads, admin changes by non-administrators) and names the route.
The plan check and per-tier prose leave the edit-manage-remote-site skill.
Pushes, pulls, site and preview deletion, backup imports that replace a site, WordPress.com DELETE requests, and WP-CLI commands that delete data past the trash now wait for the user's yes in Studio's own question UI. A declined call returns what the user answered; a session that cannot ask does not run it. Questions are asked one at a time. The confirmation rules leave site_push, site_pull, and the edit-manage-remote-site skill.
…sult records them
Run 22c failed with every header link going nowhere: the theme's part
wrote `<!-- wp:navigation-link {"label":"Home","type":"page","id":11} /-->`
with no url, and core prints a navigation link's href only from its url
(wp-includes/blocks/navigation-link.php), so the four links rendered as
<a> with no href. The agent's own curl check grepped labels and hrefs
separately and called it fine.
When the editor's link picker links a page or term, it binds the link's url
to it (useEntityBinding in @wordpress/block-library): kind post-type or
taxonomy, and metadata.bindings.url core/post-data or core/term-data
"link", which core fills with get_permalink() / the term link at render
(wp-includes/block-bindings/post-data.php; navigation-link url is a
supported binding attribute in block-bindings.php). The block editor pass
that stores palette colors now stores this too, for files and posts alike,
and the note says so. A link with a url, a binding, or kind custom stays as
written; navigation is a dynamic block, so the markup stays valid.
Live: the editor pass binds page and category links, in submenus too, and
leaves the rest; on a real site do_blocks() renders the unbound link as
<a> with no href and the bound one with href="<permalink>". Mutations: 6/6
caught.
Run 22's contact form cost about 2M tokens in each passing session. 22a read Jetpack's source for 19 turns (1.91M) to work out the form's block markup; 22b wrote it by hand, missed the closing comments, and spent three fix scripts on 8 invalid blocks (1.7M). The editor that validates every write already has the answer: the site's own block editor, with the blocks of the site's active plugins, builds a block the way the inserter does, from its default inserter variation's attributes and inner blocks (createBlock and createBlocksFromInnerBlocksTemplate in @wordpress/blocks). block_markup returns that markup for a block name (or a named variation), with the block's inserter variations, the blocks that go inside it (their parent or ancestor, or its allowedBlocks), and the other blocks in its namespace, so a misremembered name finds the real one. It is a Studio tool, so MCP serves it to external agents too. plugin-recommendations teaches it in place of the WP_Block_Type_Registry listing and block.json reading, and the core/html form refusal names it. The validator's editor tab is cached per site and registers the blocks of the plugins active when it loaded, so a plugin activated since was missing from it: its blocks parsed as core/missing, which validation skips. The tab now loads again, once, when content names a block it lacks; both the validator and block_markup go through it. Live on a site with Jetpack: with Jetpack deactivated the editor loads, Jetpack is activated, and block_markup then returns jetpack/contact-form's Contact Form variation (name, email, message, submit; 11 blocks, all valid); saved to a page on Twenty Twenty-Five it renders a POST form with those inputs and a submit button. Mutations: 14/14 caught.
block_markup asked the agent to remember a call. The answer belongs in the notes it already gets at the moment its markup goes wrong, and those notes already show "the block saves" for a block whose attributes the editor can read. Two cases had nothing correct to copy from, and they are the two that cost run 22 about 2M tokens each: - Markup that does not parse into the blocks it names. 22b wrote Jetpack's form with no closing comments; the note listed core/freeform and the group around it, with the group's save of a garbled parse. - A block that is not registered. The parser reads it as core/missing, which validation skipped, so a misspelled jetpack/contact-forms passed as valid; the "not registered" branch could never run. A note that the editor cannot fix now ends with what the site's block editor inserts for each block the failing markup names, where that insert brings inner blocks of its own (the default inserter variation, built with createBlock and createBlocksFromInnerBlocksTemplate), leaving out a block another insert already holds, three at most. On a site with Jetpack that is jetpack/contact-form's form, whole; core's blocks keep their templates in their edit components, so they add nothing. The core/html form refusal ends with the same for the block its policy names. An unregistered block is reported, with the blocks its namespace does register. block_markup and its skill line go; insertedTemplates() is the primitive, and the editor tab still loads again for a block registered since it loaded. Live on a site with Jetpack, the gate gives: for 22b's broken form, the mismatches and then the editor's contact form; for jetpack/contact-forms, "not registered ... The registered jetpack/ blocks are jetpack/contact-form, ..."; for a core/html form, the refusal and the contact form. Without Jetpack, the tests register blocks with inner-block variations in the real editor. Mutations: 16/16 caught.
3e9ba2b committed only the tool's deletion; these are its changes, as its message describes. block_markup asked the agent to remember a call. The answer belongs in the notes it already gets at the moment its markup goes wrong, and those notes already show "the block saves" for a block whose attributes the editor can read. Two cases had nothing correct to copy from, and they are the two that cost run 22 about 2M tokens each: - Markup that does not parse into the blocks it names. 22b wrote Jetpack's form with no closing comments; the note listed core/freeform and the group around it, with the group's save of a garbled parse. - A block that is not registered. The parser reads it as core/missing, which validation skipped, so a misspelled jetpack/contact-forms passed as valid; the "not registered" branch could never run. A note that the editor cannot fix now ends with what the site's block editor inserts for each block the failing markup names, where that insert brings inner blocks of its own (the default inserter variation, built with createBlock and createBlocksFromInnerBlocksTemplate), leaving out a block another insert already holds, three at most. On a site with Jetpack that is jetpack/contact-form's form, whole; core's blocks keep their templates in their edit components, so they add nothing. The core/html form refusal ends with the same for the block its policy names. An unregistered block is reported, with the blocks its namespace does register. block_markup and its skill line go; insertedTemplates() is the primitive, and the editor tab still loads again for a block registered since it loaded. Live on a site with Jetpack, the gate gives: for 22b's broken form, the mismatches and then the editor's contact form; for jetpack/contact-forms, "not registered ... The registered jetpack/ blocks are jetpack/contact-form, ..."; for a core/html form, the refusal and the contact form. Without Jetpack, the tests register blocks with inner-block variations in the real editor. Mutations: 16/16 caught.
Run 23's failing session (23b) resolved every note: its last checks said the Home page and templates/page.html were valid. Studio then saved the editor's version of each to store its palette colors, and the judge found six and two Classic blocks in them. The agent had marked its sections with comments between blocks (<!-- ===== HERO ===== -->). The editor parses text outside any block as a Classic block and wraps it in a paragraph (normalizeRawBlock in @wordpress/blocks runs autop on it), so the save turned each comment into <p><!-- ... --></p>, which is HTML outside any block. The pass's own check only asked whether the blocks parse as valid. 9 of 38 spike sessions across runs 1-23 wrote such comments. Two changes, at the one place every editor save goes through (storeAsTheEditorDoes, for files and posts, before and after a fix): - dropCommentsBetweenBlocks() drops a Classic block that holds nothing but comments, before the other passes. A comment renders nothing, so the page does not change, and the note says what was dropped and why. Comments inside a block are left to the block. - A save stands only when the full validator finds no more problems in it than in what it replaces, whatever pass changed it; otherwise the content stays as written. describeEditorSave() now builds the note for both the file and the post paths, which listed the same passes twice. Replaying 23b's own Home page and page.html through the gate on a live site with the session's palette: 78/78 and 16/16 valid, 6 and 2 comments dropped, 32 and 6 colors stored, 0 invalid blocks in what is saved (before: 6 and 2 core/freeform). Mutations: 12/12 caught.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+765
to
+766
| ! String( block.attributes?.content ?? '' ) | ||
| .replace( /<!--[\s\S]*?-->/g, '' ) |
Eval cases ran one at a time because two concurrent cases could give their sites the same port: the runner reserves the ports of the sites it can see when it starts, and a case cannot see the port another assigns after that. Yoga x3 took 11-17 minutes run back to back. Each runner process now claims a range of 100 ports before the port finder loads and starts its sites there (STUDIO_BASE_PORT, which the port finder already reads; a value the user set stays). A range is held by listening on a port of its own below the ranges, which one process at a time can do and which the OS releases when the process ends, so nothing is left to clean up after a killed case. A first version held ranges with lock folders and pids; its own concurrency test caught a reclaim race in it. The eval config now runs four cases at a time. Tests spawn real processes that claim at once (distinct ranges), end (range given back), and find a range's port taken by something else (passed over). Mutations: 4/4 caught.
Run 24a wrote jetpack/contact-form with Jetpack active but its Forms module off; the note said the plugin that registers the blocks is not active, and the agent spent four calls finding the module. The note now states what the editor shows: no blocks of that namespace are registered in the site's block editor.
…ster A block a plugin registers from block.json is registered in the editor by its editorScript. When that script throws while the editor loads, the block never reaches the editor, and the note said only that it is not registered there. In run 25 (custom-block-plugin) the agent's index.js read `wp.blocks` with no index.asset.php naming its dependencies, so WordPress registered the script with none (register_block_script_handle, wp-includes/blocks.php: `$script_asset['dependencies'] ?? array()`) and it ran before `wp` existed. The agent found the block registered in PHP, took the note for stale and spent 14 calls arguing with it; the judge failed the case. The editor page now keeps what it throws (Playwright's pageerror), as the message and the script line that threw it, and the not-registered note adds it: "The site's block editor threw: Cannot read properties of undefined (reading 'blocks') (http://…/opening-hours/block/index.js line 2)." It states only what the editor showed. Replayed with run 25's plugin on a live site: the note above, exactly. Tests: a live case writes a block.json plugin whose editorScript reads wp with no asset file (gated on STUDIO_BLOCK_EDITOR_PATH beside STUDIO_BLOCK_EDITOR_URL). Mutations 6/6 caught. The "loads the editor again" test gets the 60s its neighbours have: two editor loads exceed the 5s default on a site running Jetpack, before this change too.
… from When the editor makes a block again from its comment attributes, an inline style written on its markup that no attribute holds is dropped, the fix loses something, and the block stays as written with a note asking the agent to move the style into the comment. In run 25 (multi-page-site-menu, 7.94M tokens) about 3.5M went to six rounds of rewriting such styles, e.g. `margin-top:var(--wp--preset--spacing--50)` on core/columns, because the note does not say which attribute holds it. The fix pass now stores them itself. Core's style engine knows which style attribute path saves each CSS property; getCSSRules in @wordpress/style-engine is handed a style whose every value names its own path, and the rules it returns map marginTop to spacing.margin.top, backgroundColor to color.background, borderTopLeftRadius to border.radius.topLeft and so on, with no list in Studio. For each element of the block's own markup (inner blocks excluded, from the default parser's innerHTML), the styles the remade block would drop are stored at those paths: a preset as var:preset|… the way the editor stores one, a value as written (of a shorthand too, so background:#efe6d8 stores #efe6d8). The block is kept only when every element it saves holds the styles written on it; otherwise it stays as before, and the fix's loss check still decides what Studio saves. A shorthand's longhands it leaves at initial are what an unset longhand is, so they no longer count as dropped styles. Replayed on the 26 page revisions run 25's yoga session wrote: blocks the fix would lose something from went 69 -> 43; six revisions went to none. The rest are list text and a button whose comment's font size preset and written size disagree, which stays the agent's choice. Tests (live): a spacing preset, a color with radius and padding, a background shorthand, a style on the element the block saves it on, and only a block's own styles beside an inner block's are stored; a style no attribute saves, one the block saves on another element, and one its attributes already set otherwise stay as written. Mutations 11/11 caught.
…a store judged on what WooCommerce sells
Prompts named their sites "Eval …" and told the agent "make every choice
yourself: nobody can answer questions during this run". No user writes
either, and both tell the model it is being tested and to skip questions a
user would want asked. The judge needed the exact name only to find the
site; each case already runs in folders of its own and the capture records
which sites the turn created.
- The site judged is the seeded active site the request is about, or else
the one site the turn created (more than one fails). vars.siteName still
picks one by name. asksCurrentOrNew reads the seeded site's name from the
case instead of matching "Eval Active Site".
- Every prompt reads as a user ask with a real name (Juniper & Oak, Lotus
Flow Yoga, Corner Bakehouse, …), and seeded sites carry real names.
- The scripted user (eval-answers.ts) answers by question, not by order:
each answer fits questions whose text matches its `when` and that can
take it (typed where the question allows it, otherwise one of its option
labels). defaultTest gives every case a user who answers "Whatever you
think is best." and No to starting a classic theme. In run 25 the yoga
case's single ordered "No", meant for the classic-theme question, answered
"Install Jetpack for the contact form?" instead. Every case can now ask, as
in the app; push and site-pick cases keep their own answers.
New case abilities-plugin: "Write a plugin for my book club's site that
keeps our reading list … I want AI agents to be able to list the books and
add new ones through the WordPress Abilities API." The seed records the
abilities the site has; after the turn the inspection runs each new one
through core's REST route (/wp-abilities/v1/abilities/<name>/run, method
from its annotations as the run controller requires), with input made from
its input schema: each that writes as a visitor and as an administrator,
then the read-only ones. abilitiesWork passes when every new ability is in
REST, a visitor is refused, one adds the book and a read-only one lists it.
Through the runner on a seeded site: doing nothing fails ("no ability was
added"); installing a reference plugin passes. Fixtures are the inspection's
real output for that plugin and five variants (visitor allowed, no
read-only annotation, not in REST, input refused, book not listed).
woocommerce-shop-setup: storeHasProducts now also needs the shop page
published and vars.minProducts (2: a dog and a cat product) that
WooCommerce reports purchasable and visible in the catalog
(WC_Product::is_purchasable(), is_visible()), read from wc_get_products()
in the capture. Test data is what WooCommerce reported on a live site for
a priced, an unpriced and a catalog-hidden product.
Mutations caught: scripted user 5/5, site pick 4/4, abilitiesWork 7/7,
storeHasProducts 4/4.
In run 26 the abilities-plugin agent registered its book post type with capabilities edit_post => edit_posts, read_post => read and delete_post => delete_posts, and map_meta_cap. WordPress then checks edit_posts, read and delete_posts as the meta capabilities in every check (_post_type_meta_capabilities() in wp-includes/post.php), and with no post map_meta_cap() maps them to do_not_allow: the site's administrator lost them everywhere. Its "add book" ability, guarded by edit_posts, refused the administrator. The agent called it "a known WP-CLI eval limitation", started the site and set up an application password to test over REST, then read map_meta_cap() and has_cap() until it found its own post type: about 25 calls and 1.5M tokens. When the PHP of a site's active plugins or themes changes, or which are active, Studio asks WordPress which capabilities the administrator role grants that user_can() denies the first administrator, once with the site's code loaded and once with --skip-plugins --skip-themes. Those denied only with the code loaded are the problem, so what core itself denies (manage_links with no links, unfiltered_upload) is not reported. The problem names what map_meta_cap() maps each to and, when a post type took the capability over as a meta capability, that post type, with capability_type as core names a type's capabilities (edit_page, edit_pages). It stays open until the code changes and WordPress grants them again. Nothing is listed in Studio: the capabilities are the role's, the cause WordPress's own. Live on a fresh site: a post type mapping primitive capabilities, a map_meta_cap filter denying manage_options, and capability_type 'book' (nothing reported); WordPress is asked only when the active code changed, not for an inactive theme's file. Mutations 9/9 caught. Through the eval runner, copying run 26's original plugin into Chapter & Verse ends the turn with the problem in its checks.
The six outcome cases of the open eval PR (eval-user-reported-failures), with their judges and capture: woo-mini-cart, faq-details-block, style-variation-for-dark-look and page-in-native-blocks (a feature WordPress has natively, asked the way users ask), debug-plugin-fatal (a plugin update fatals every page) and debug-migrated-images (images still point at the old domain after a move). The capture adds each image on the home page with the status the site answers it with, and the code the turn added, changed or removed on the seeded site. Their seeded sites carry real names (Hearth & Clay, Brightwater Dental, Lumen Design Studio, Oakline Carpentry, Daybreak Coffee, Flour & Ember), as the other cases now do. Through the runner, a turn that does nothing fails each case's own check: no Mini-Cart in the header parts, no Details blocks, no style variation, no About page, the fatal error, the images from another site.
The end-of-turn checks now compare each site with itself as Studio first saw it in the session, before the first call that could change it, and raise only what is new. A site the user brings with a problem of its own is no longer the turn's to fix. In run 27 (global-styles-db-override) the agent changed the background in the saved Global Styles, as asked, in six calls. The end of the turn then raised that the seeded theme's index.html renders no page content, which the site had before the turn. To fix it, the agent made a child theme and activated it, which left the user's saved Site Editor styles behind with the old theme, and the case failed on them. The baseline is taken in SiteChanges.snapshot, which every call that can change a site already runs, so it covers file tools, shells and MCP alike. Tests: a problem the site had is not raised, one the turn made is, and a later call's snapshot does not move the baseline. Mutations caught 3/3.
AskUserQuestion now ends the turn (Pi's AgentToolResult.terminate) when any question comes back unanswered, as it does when the user presses Esc in studio ai's picker. Its result says which questions were closed and the answers given, so the next message the user sends reads against it. defineTool carries a tool's terminate through to Pi. The eval's scripted user dismisses a question no answer fits, leaving it out of the answers as the picker does, and records it as "(dismissed)". Before, it typed "Cancel", which the app never sends. In run 27 (active-site-different-name) the agent asked where to build "Joe's Bakery", read the typed "Cancel" as a go-ahead and built on the active site. Tests: the answers when all are given, the end of the turn when one is missing or blank, and the scripted user's dismissals. Mutations caught 5/5.
The header-menu check now counts each link to a section of the home page
("/#features") as a target, as it does a link to a page, so a header of
several section links outside core/navigation is raised as a menu built
some other way. One link is still a call to action, and the bare home
link (the brand) still counts for nothing.
In run 27 (one-page-nav-anchor-links) the agent built the "header menu
that jumps to each section" as three paragraphs linking to /#features,
/#about and /#contact. The check left out every link whose path is "/",
so it said nothing, and the case failed on a header with no navigation.
Tests: several section links with no navigation are raised. Mutations
caught 3/3.
The skill now says Jetpack registers its blocks only while its "blocks" module is active (Jetpack_Gutenberg::should_load), which a site never connected to WordPress.com starts without, and lists that module for Jetpack's blocks. It replaces the line that said a missing block's own module is usually inactive, which sent the agent looking for a "slideshow" module. In run 27 (jetpack-catchall-slideshow) the agent installed Jetpack and wrote a jetpack/slideshow block, and Studio's note said no jetpack/ blocks are registered. It looked for a slideshow module, activated "shortcodes" and saved a [gallery type="slideshow"] shortcode instead. On a copy of that site, Jetpack 16.3.1 in offline mode registers no blocks; after "wp jetpack module activate blocks" it registers jetpack/slideshow and the rest.
The capture listed the registered block styles before it read the theme's merged data, and reading it is what registers the block styles of a theme's styles/blocks/*.json files (WP_Theme_JSON_Resolver::get_theme_data calls wp_register_block_style_variations_from_theme_json_partials). So a theme that gave its cards a block style that way, as Twenty Twenty-Five gives its own, failed repeated-treatment-uses-block-style on "registered variations []". It now reads the merged data first. Checked through WP-CLI on Twenty Twenty-Five with a styles/blocks/card.json for core/group: before reading the theme's data the registry has no core/group styles, after it has card and section-1 to section-5.
The skill's "The look as theme.json" section now says a treatment several blocks of one type share, such as cards, is a block style: a styles/blocks/<slug>.json file with blockTypes, as Twenty Twenty-Five's are, which WordPress registers and the block's Styles panel offers, and which a block takes as "className":"is-style-<slug>". In run 27 (repeated-treatment-uses-block-style) the agent gave the three cards a "card" class styled in style.css, and nothing the spike gives the agent named WordPress's form for a shared look. Trunk says it in its system prompt and block-content skill; this is the one line of it the spike's skills keep, in the skill that writes the theme. A check was tried instead (classes several blocks share that the theme's CSS styles) and dropped: on run 27's sites it raised 7 and 17 classes, most of them layout classes that are no shared look.
WordPress reads theme.json and each styles/*.json file as
( new WP_Theme_JSON( $decoded ) )->get_raw_data(), which migrates the file
to the current schema and removes what the schema has no place for, with
no word to anyone. A write of one of those files now gets a note listing
the keys it removed, from WP_Theme_JSON's own steps run on the site
(migrate, unwrap the shared block style variations, sanitize), after
resolving the theme's data as WordPress does first, so the theme's block
styles are registered. A file without "version" gets its own note, since
WP_Theme_JSON_Schema::migrate keeps only { "version": 3 } of it, and a
file that is not JSON says what json_last_error_msg() says.
In run 27 (style-variation-for-dark-look) the agent wrote styles/dark.json
without "version", saw the variation registered with only its title, and
spent 17 calls and about 0.3M tokens taking WP_Theme_JSON apart to find
why. Run on the 40 theme JSON files the agents wrote in runs 1-27, the
check keeps Twenty Twenty-Five's 30 whole and 18 of the 40, and names
what the rest lose: "color": "#1b2a4a" where the schema has
{ "text": ... } (headings and links that render without the colour they
were given), fontWeight beside typography rather than in it, h1 inside
heading, textUnderlineOffset, a "google" key on font families, and
styles for WooCommerce blocks on a site without WooCommerce. "$schema",
which WordPress removes too, is for the editor writing the file and is
left out.
About 0.23s of WP-CLI per write of a theme JSON file. Tests run on a real
WordPress under STUDIO_TEST_WORDPRESS; mutations caught 10/10.
Studio's own confirmations (site_create on another site's request, a destructive call, an MCP server's approval) read a question the user closed as a "No": "Not run: the user answered "No" to: Create a new site "Joe's Bakery"? This request is about "Corner Crumb"." In run 28 (active-site-different-name) the agent took that No as the user choosing the active site, said "your answer says to build into that site", and started Corner Crumb to build Joe's Bakery in it. A confirmation the user closes now does what AskUserQuestion's does since a413e0b: the call does not run, the result says the user closed it without answering, and the turn ends, so what comes next is the user's. The confirmer returns a Refusal ({ text, endsTurn }) and the two places that ask, Studio's tool wrapper and the MCP tools, return it as a terminating result instead of throwing it. AskUserQuestion and the confirmer share the wording, closedWithoutAnswering(). Tests: confirmation, MCP tools and the Pi runtime's wrapper; mutations caught 6/6.
Block validation compares the markup written with what the block saves, so a block whose save is empty passes it and still shows nothing on the page. In run 28's jetpack-catchall-slideshow case the agent wrote Jetpack's slideshow self-closing, with only its "ids". Jetpack reads the slides from the block's saved HTML (its block.json "images" attribute has "source": "query") and its render() returns that HTML, so the page rendered no slideshow while validation passed. The agent guessed the block needed a WordPress.com connection and switched to another plugin, which failed the case. After a call changes a post or page, Studio now renders its content as the front end does, through the_content, whose do_blocks() renders each block with render_block() (wp-includes/blocks.php), and raises a problem naming each block whose output is empty. Where the block was written without saved HTML, the problem names the attributes its block.json reads from that HTML. The inner blocks of a dynamic block are left out, because the dynamic block renders its own output from them, as Jetpack's form renders its fields. A site Studio cannot render for says nothing. On a copy of the run-28 site with Jetpack active, the agent's exact markup gives jetpack/slideshow, written without saved HTML, its "images" read from it. Across the 112 published posts and pages of the last 30 eval sites it names no block, after the dynamic-block rule removed Jetpack form fields and WooCommerce's product template, about 0.3s per site.
…ts of their kind In run 28's repeated-treatment-uses-block-style case the agent set "fontSize":"0.8rem" on 5 blocks of its front-page template, beside 7rem padding on 4 and 5, though the visual-design skill already says to drive sizes from the theme's presets. A block takes a preset by its slug, which WordPress renders as the preset's CSS variable (block-supports/typography.php, the style engine), so the editor offers it and Global Styles changes it on every block at once; the same raw value on many blocks is one treatment kept in many places. When a theme template, part or pattern, or any file opening with a block, is written, Studio now names the font size, font family, color and spacing values set raw on three or more of its blocks, and lists the site's presets of each kind as wp_get_global_settings() resolves them, with how a block takes one. Presets, CSS variables and zero values are left out. On the run-28 template the note names the three values and the presets WordPress resolves, in 0.25s. Those font sizes are core's defaults, because core keeps its default font sizes over a theme's of the same slug unless the theme sets defaultFontSizes to false (prevent_override in class-wp-theme-json.php), so the theme's own small 0.875rem was never used. Across 799 template files agents wrote on earlier eval sites the note fires on 14, each a value written by hand on many blocks.
…to spike-refresh-v2
Trunk's #5407 test loads block-content, which the spike removed, so the merge left it failing; it now loads rank-me-up.
Run 28 showed prose holding no rule: the visual-design skill said to drive every size from the theme's presets, and the agent wrote "fontSize":"0.8rem" on five blocks. So each MUST, NEVER, "do not" and CRITICAL left in the skills and tool descriptions the agent reads went one of three ways. Rules a tool already holds are deleted. Studio's block validation turns core/html into editable blocks, so plugin-recommendations no longer forbids a hand-built core/html or form. site_push and site_pull refuse a remote site that is not syncable and say what it needs (sync-site-picker.ts), so site_connected_remote_sites no longer says not to try. need_for_speed and rank_me_up appear only once their skill is loaded (skill-tool-scope.ts), so "CALL ONLY ON EXPLICIT USER REQUEST" is gone. Rules a tool can hold now do. generate_images leaves the site's name out of each image's subject, composition and site context, because the image model paints a name it is given in as a fake wordmark; its description had said NEVER to include it. AskUserQuestion drops an "Other" option passed in, because Studio adds the free-form one itself; its description had said do NOT add one. The rest says what to do: the visual-design rules for building what the brief names or the user picked, a reference site's feel, the four options' grounds, square full-bleed corners and the sneak peek's frame; the preview and remote site descriptions; and the robots.txt finding in rank-me-up. Product policy in the system prompt's identity is left as written.
In run 29 (repeated-treatment-uses-block-style) Ember Roasters' three cards took a "feature-card" class styled in the theme's CSS, and the case failed with no block style registered. Run 28 passed the same case with the same skill line, so the line in visual-design that named block styles held in one run of two. It is replaced by a note. On a write of a theme's stylesheet or block markup, Studio now counts the classes the blocks of the theme's templates, parts, patterns and published posts carry, by block type, and reads the properties each class is given by a rule of its own in the theme's stylesheets. A class on three or more blocks of one type, whose rule sets a property WordPress's theme.json styles hold outside spacing and dimensions, is a look those blocks share. The note names each such class, its block type and those properties, says that WordPress's form for it is a block style in styles/blocks/<slug>.json, which WP_Theme_JSON_Resolver::get_style_variations() registers, and lists where each of its properties goes in the block style's styles, by WP_Theme_JSON::PROPERTIES_METADATA, which themePresets now reads from WordPress with the presets. Each class is named once a session per block type, and all of them in one note per write. The run 27 check that was dropped for noise counted every class the CSS styled. Leaving out classes that only place a block (grid-column, padding, width) is what keeps this one to looks: on the 55 eval themes it names 211 classes, such as eyebrows, cards and labels; run 29's Ember theme gets `.feature-card` and `.eyebrow`, and run 28's, which used block styles, gets nothing. Each note took 0.5 to 2 seconds.
WordPress merges a theme's presets into core's, and while a default preset set is on (settings.typography.defaultFontSizes, spacing.defaultSpacingSizes, color.defaultPalette and the rest of PRESETS_METADATA's prevent_override settings), it drops each theme preset whose slug is one of core's (WP_Theme_JSON::merge() and filter_slugs() in wp-includes/class-wp-theme-json.php). A theme that names its font sizes small, medium and large gets core's 13px, 20px and 36px, and nothing in the theme.json it wrote says so. Run 28's Ember Roasters set small to 0.875rem and got 13px. themePresets now also resolves which of the theme's presets the merge dropped, and the value core's preset of that slug renders, taken as get_settings_values_by_slug() takes it (value_key, or value_func for font sizes). On a write of the active theme's theme.json, Studio names them, the setting that drops them, and the two ways to keep them: the setting set to false, as Twenty Twenty-Five does, or a slug of the theme's own. On the eval sites, 51 of the 148 whose active theme the agent built had presets replaced: font sizes on 38 and spacing sizes on about 47, besides a palette white on 2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experimental spike — not for merge as-is
Studio Code teaches WordPress to the model in prose. The system prompt and skills explain constrained-layout selectors, root padding, which template renders the front page, Site Editor overrides, preset slugs, which writes need confirmation, which WordPress.com plans allow what. Each rule was added to fix a real failure. Together they are expensive context. They also drift from the WordPress actually running on the site, and the model can ignore them.
This spike asks whether Studio Code works as a generic WordPress agent instead:
studio_instructionswithout a skill) it is 4,624 vs 1,006.block-contentandvisual-polishare gone entirely.The branch is built out as a whole and compared with trunk on cost, time, and results, rather than landed piece by piece.
What moved from prose into tools
validate_blocks):post create/post updatecontent are validated with the block editor running on the site. This happens before any tool puts that content on the site or renders it.core/htmlpolicy, notes the bare#anchorlinks in shared parts (left as written, so#topkeeps working), and rejects post content that opens as JSON but does not parse.wp_cli: refuses shell-chained commands (&&,;,|), since it runs one WP-CLI command without a shell.wp_cli, apart from read-only subcommands), or creating one while a site is active, wait for the user.tmpfiles get the same write notes as Write and Edit. The post content and theme checks skip read-only tools and look only at the request's sites.refresh_browser).~/.studio/mcp.json, as the servers describe them (mcp-tools.ts). A server tool named like one of Studio's own, or like an earlier server's, is left out, and a server that doesn't answer is skipped for five minutes. WordPress.com's own gates (plan, role, the free-site upsell) and its approval decide what runs. When the server asks for approval, Studio asks the user and sendsuser_confirmed, which the model cannot set. This replaceswpcom_request,wpcom-capabilities.tsand theedit-manage-remote-siterunbook. The Studio plugin lists the same WordPress.com server for Claude Code and Codex.Eval
The eval judges what a user would see, not the route the agent took, so agents that use different tools compare fairly. This branch runs trunk's suite unchanged: rendered pages, block validity, theme files, Site Editor overrides, other sites left untouched. Each case runs against its own site registry (#5167).
Status (Oct 23)
Merged with trunk; no commits behind. Where trunk landed a mechanism in its own form, this branch uses trunk's: the core-file refusal, saved-copy result notes, Active site facts, the outcome judge, the
core/htmlpolicy, and the unparseable-JSON refusal. Trunk's fixes are kept: the empty-stop retry, leaked tool-call markup,wp_cliquoting, and the host option to turn offwpcom_request(now removed with the tool).Next, prose that still gates behavior, moved into structure. Each change reaches Studio Code and
studio mcpthrough the same tool or fact:edit-manage-site's Site Editor sentence andplugin-recommendations' Coming soon step are deleted.site_createwhile the request already has a site, once per request;build-site's site-selection rules are one sentence.site-database.ts), each owned by the module that knows that part of WordPress and listed once insite-facts.ts. They now include the active theme and parent, which of them WordPress updates, and the active plugins. Whether WordPress updates a theme comes from its own last update check (theupdate_themestransient), which also decides the write refusal, so themes the user develops stay editable.The plugin's hooks run Studio's write checks on Claude Code's and Codex's own file tools.Removed: Claude Code and Codex use Studio throughstudio mcp, which reports on its tools' results, and the eval runs Studio Code.WordPress.com plan facts state what the API returns, without a hardcoded feature list: the Active site line states the plan and the user's role, andReplaced by the next item.wpcom_requestrefuses a write the plan lacks a feature for, naming the feature as the platform does and the plans this site is offered that include it (/sites/:id/plans).~/.studio/mcp.json({ url, headers }or{ command, args, env }).wpcom_request, the plan facts above and the remote-site runbook are deleted (−320 lines). Verified live on a free test site: a realstudio codeturn listed template parts throughwpcom_search→wpcom_execute; a draft-post write asked the user once, sent nothing when declined, and ran when approved, with the model's ownuser_confirmedstripped.scaffold_themeis deleted (−598 lines) and the agent no longer writesDESIGN.md. A look option is the theme.json it would become, rendered as the same design board, and the picked one is saved as a file to copy into the theme. After any theme.json write, Studio downloads from Google Fonts the files of each font family declared without them, in the weights its styles use, and says so; a family Google Fonts lacks is reported. The look-and-layout picker is unchanged, and so are the app's Design system tab and its annotations.inspect_designand the screenshot-and-fix loop are deleted (−477 lines). In a build eval its eight calls found WordPress structure: a full-width hero that couldn't reach the page edges (the template's post content sat in a constrained group, so the agent hand-rolledwidth:100vw; margin-left:calc(50% - 50vw)), and a palette slugtextwhose.has-text-colorclass overrode every block's own text color. A write now notes a full-width block that a constrained block around it holds to the content width, naming the block to give align full; theme CSS that stretches to the viewport width; and a palette slug whose class collides with the block editor's color markers. Replayed on that eval's files, each note fires.take_screenshotstays, to show the user a page, and Studio gives a theme without a thumbnail its design board asscreenshot.png. Theedit-manage-siteskill is deleted: editing and managing a site is what Studio is for, andbuild-siteends at publishing the content.site-notes.ts), by the action that triggers it:beforeWrite(the refusal),write(block validation, the pattern cache, fonts, the theme thumbnail, and what WordPress will render differently from the file), andresult(core block settings, and site facts that are new or no longer true). Studio Code fires all three (a shell command's writes included), andstudio mcpfiresresult. It replaces three hand-wired pipelines; a new note is one more source, and site facts are itsresultsource that speaks once and then only on change.build-siteis its five steps, without the Site Spec ritual:pick_design's description takes the rules for calling it and each draw ends with what follows;generate_images' parameter descriptions takeimagery's spec tips. A write note for images shown from another host replaces "never use a web URL" (wp_filter_content_tagsinwp-includes/media.php).annotatesays where an element is rendered instead of teaching database overrides (the saved copies the notes flag);open_annotation_browsertakes the site and page.liberatekeeps its commands and loop, without repeating whatstudio create --fromprints.hosting-plans-helperremoved: WordPress.com's MCP server serves the plan catalogue aswpcom/plans-list.build-siteandvisual-designgates: the Site Spec as a tool call,pick_designchoosing a varied set, sneak peek checks inpresent_design_options, an end-of-turn render check (measured facts such as overflow and unloaded fonts).Peeled off for trunk
Fixes the spike exposed that also apply to trunk: #5160, #5161 and #5167 (all merged).
Testing
npx vitest run apps/cli/ai scripts/eval packages/common packages/design-md: 1,767 passed (mcp-tools.test.tsruns a real stdio MCP server; the thumbnail test renders a real board). The 16 e2e and startup tests that need a freshdistfail the same way before and after.tscpasses, and the CLI builds (vite build).npx vitest run apps/cli/ai packages/common --tagsFilter='!e2e'1,280 passed;tscpasses. They include the Windows failures: the lossless-fix command now names its file with forward slashes (Git Bash reads backslashes as escapes), and the site test fixture closes its database.AI disclosure: written by Claude (Anthropic) running in the Pi coding agent via Roadie, directed by @chubes4. Claude wrote the code, the eval harness changes, and this description, and checked WordPress behavior against the source on disk.