Skip to content

Studio Code: Check block content where it is saved instead of asking the agent to validate it - #5241

Closed
chubes4 wants to merge 3 commits into
trunkfrom
studio-code/validate-on-write
Closed

chubes4 wants to merge 3 commits into
trunkfrom
studio-code/validate-on-write

Conversation

@chubes4

@chubes4 chubes4 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Related issues

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, and ran the evaluation below. I reviewed the diff and the eval runs.

Proposed Changes

Block content had to pass validate_blocks before reaching a site, and the agent was told so in a general rule ("MUST pass validate_blocks… Never apply unvalidated block content") and in five skills. The check is deterministic; only remembering to call it was not. Now the content is checked where it is saved:

  • wp_cli post create and post update check block content in the site's block editor before saving it, inline (--post_content=) or from a file. Re-serializations the editor makes without loss are kept and shown as a diff; content the editor rejects is not saved, and the result lists why.
  • Studio Code's Write and Edit run the same check on a Studio site's theme templates/*.html and parts/*.html, which are live as soon as they are written.
  • Studio Code no longer has a validate_blocks tool. The general rule and the skill steps that told the agent to call it are gone; the block-content skill states how content is checked instead.
  • The page-content recipe saves its scratch file with wp_cli post update <id> tmp/page-<slug>.html instead of a wp_cli eval import, so the save goes through the check.
  • MCP clients keep validate_blocks, now only for theme files they write with their own file tools (Studio cannot see those writes). Their post content goes through wp_cli and is checked too.
  • block-check.ts holds the one check all three use.

Testing Instructions

  • npx vitest run apps/cli/ai: new tests cover content the editor rejects (nothing saved), lossless fixes kept inline and in files, commands without block content left unchecked, and templates/parts checked only on Studio sites.
  • Eval: two suite cases that write block content, trunk (f142df336) and this branch, 3 runs each, judged by the suite's outcome checks (siteWorks includes zero invalid blocks in the site's own editor):
Trunk This branch
Cases passed 5/6 6/6
Invalid blocks left on the site 0/6 0/6
validate_blocks calls 10 (38 s) 0
post create/update time, median 464 ms 408 ms (includes the check)
Tool calls, median 21.5 25
Tokens, median (total) 459k (3.1M) 640k (4.9M)

Trunk's one failure was the hero case's style check (.eyebrow given an inline display), not block validity. In one branch run the check refused a post update whose paragraph held a nested <p>; the agent fixed the markup and saved it. The branch's token total comes mostly from two long runs: one spent its extra calls on a 404 after the agent saved the page with --post_status=published, the other on that refusal and on an invalid _wp_page_template value. With three runs per case the token difference is within run-to-run variance (trunk's own runs on the hero case range 525k–906k).

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors?

…the agent to validate it

wp_cli post create and post update check block content in the site's block
editor before saving it, inline or from a file: lossless re-serializations are
kept and shown as a diff, and content the editor rejects is not saved. Studio
Code's Write and Edit run the same check on a Studio site's theme templates and
parts, which are live as soon as they are written. Studio Code no longer has a
validate_blocks tool, and the prompt rule and skill steps that told it to call
one are gone. MCP clients keep validate_blocks for theme files written with
their own tools; wp_cli checks their post content too. The page-content recipe
saves its scratch file with post update <id> <file> instead of wp_cli eval.
@chubes4

chubes4 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

Closing to focus on #5146

@chubes4 chubes4 closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant