Skip to content

Studio Code: Refuse wp_cli --post_content JSON that does not parse - #5310

Merged
chubes4 merged 1 commit into
trunkfrom
studio-code/post-content-json-check
Oct 23, 2026
Merged

chubes4 merged 1 commit into
trunkfrom
studio-code/post-content-json-check

Conversation

@chubes4

@chubes4 chubes4 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Related issues

How AI was used

Claude (via the Pi coding agent in a Roadie session) found the failure in eval transcripts, wrote the change and tests, and ran the eval below. I reviewed the diff and the results.

Proposed Changes

When wp_cli post create|update --post_content=… content starts with { but doesn't parse as JSON, the tool saves nothing and returns the parse error:

Nothing was saved: `--post_content` starts with `{` but is not valid JSON (Expected ',' or '}' after property value in JSON at position 174 (line 1 column 175)). WordPress ignores JSON post content that does not parse.

Block markup and HTML never start with {, so content that does is JSON, such as a Global Styles post. When WordPress can't parse a Global Styles post, it ignores the whole post. WP-CLI still saves it and prints Success: Updated post 5., so the agent has no sign anything went wrong.

That happened in an eval run of global-styles-db-override. The agent's JSON was missing its last }. Over the next 45 tool calls it checked the CSS, cleared transients, read the theme JSON resolver's source, and hex-dumped the post before finding the syntax error. The run used 1.4M tokens; the other runs used about 90k.

The check is in getUnsupportedWpCliPostContentMessage, next to the existing $(cat …) refusal. It doesn't depend on post types or the database. It refuses instead of repairing because there's no single correct fix for broken JSON.

Testing Instructions

  • npx vitest run apps/cli/lib/tests/wp-cli-post-content.test.ts covers the exact command from the failed run. It is refused with the parse error, and the same command with the brace restored is accepted.

  • Replay on a real site: the failed run's command, sent unchanged through wp_cli, was refused with the message above. The Global Styles post kept its previous valid content (#f6efe4, line height 1.7).

  • Eval: global-styles-db-override, 6 runs on this branch. All 6 passed.

    Run Result Tokens Calls What happened
    0 Pass 209k 16 Refused once, then saved valid JSON on the next call
    1 Pass 110k 10
    2 Pass 209k 14 Refused once, then saved valid JSON on the next call
    3 Pass 203k 15
    4 Pass 1029k 42 Saved with wp in Bash, which this check doesn't cover. Its JSON was missing the last }, and it took the same debugging spiral
    5 Pass 200k 13

    In both refused runs, the agent's JSON was valid but wp_cli split the quoting wrong. Run 0 wrapped the whole argument in single quotes, so a trailing ' ended up in the content. Run 2 escaped the inner double quotes with backslashes, so the content was cut to {\. On trunk, both would have been saved and reported as successful.

  • npx vitest run apps/cli/ai apps/cli/lib/tests/wp-cli-post-content.test.ts passes. Lint passes, and the typecheck shows only the existing errors in commands/export.ts and commands/push.ts.

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors?

@wpmobilebot

Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing f028cd9 vs trunk

app-size

Metric trunk f028cd9 Diff Change
App Size (Mac) 940.68 MB 940.68 MB +0.00 MB ⚪ 0.0%

site-editor

Metric trunk f028cd9 Diff Change
load 1250 ms 1228 ms 22 ms ⚪ 0.0%

site-startup

Metric trunk f028cd9 Diff Change
siteCreation 7537 ms 7543 ms +6 ms ⚪ 0.0%
siteStartup 4376 ms 4366 ms 10 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

@chubes4
chubes4 merged commit 1711172 into trunk Oct 23, 2026
13 checks passed
@chubes4
chubes4 deleted the studio-code/post-content-json-check branch October 23, 2026 03:33
chubes4 added a commit that referenced this pull request Oct 23, 2026
…5312)

## Related issues

- Pairs with #5310, which refuses `--post_content` JSON that doesn't
parse. In the #5310 eval, both refusals were valid JSON that `wp_cli`
had broken while splitting the command. This PR splits those commands
correctly, so they save as written. It is independent of #5310 and can
merge in either order.

## How AI was used

Claude (via the Pi coding agent in a Roadie session) found the two
quoting forms in eval transcripts, wrote the change and tests, and ran
the checks below. I reviewed the diff and the results.

## Proposed Changes

`wp_cli` now reads a quoted `--post_content` value the way a shell
would, in the two forms agents used where it didn't before:

- **Backslash-escaped quotes inside double quotes:**
`--post_content="{\"styles\":…}"`. The value used to stop at the first
`\"`, so `{\` was saved as the post content. Inside double quotes, a
backslash now escapes `"`, `\`, `$` and backticks. Single-quoted values
stay literal.
- **Quotes around the whole argument:** `'--post_content={"styles":…}'`.
The closing `'` used to become part of the content. A quote just before
`--post_content=` now wraps the whole value.

In both cases WP-CLI saved the broken content and printed `Success:
Updated post 5.`. Global Styles JSON that doesn't parse is ignored by
WordPress, so the background change never showed and the agent started
debugging.

**What else changes:** in 118 inline `--post_content` writes from 535
local agent transcripts, these two forms appear only in the two eval
runs above. No double-quoted value contains any other backslash, so
content written in the forms agents already use splits as before.

## Testing Instructions

- `npx vitest run apps/cli/ai/tests/tools.test.ts` covers both forms
with a trailing `--porcelain`. Both tests fail on trunk, and the
existing quoting tests still pass.
- **Replay on a real site:** the two commands from the #5310 eval, run
unchanged through `wp_cli` on the `global-styles-db-override` site, now
save the intended JSON (`post get 5 --field=post_content` returns it).
The front end renders the new `#1e2a44` background, and the user's line
height `1.7` is kept.
- `npx vitest run apps/cli/ai
apps/cli/lib/tests/wp-cli-post-content.test.ts` passes. Lint passes, and
the typecheck shows only the existing errors in `commands/export.ts` and
`commands/push.ts`.

## Pre-merge Checklist

- [x] Have you checked for TypeScript, React or other console errors?
chubes4 added a commit that referenced this pull request Oct 23, 2026
…all touches the theme file they replace (#5343)

## Related issues

- Follows #5301, addressing the review there: the saved Site Editor
copies were listed in every system prompt, though they matter only when
the agent touches a theme file they replace.
- Builds on #5333's shared result notes, so the notice reaches Studio
Code and `studio mcp` the same way.

## How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) wrote the change
and the test and ran the evals below. I reviewed the diff and the
results.

## Proposed Changes

When someone saves a template, template part or Global Styles in the
Site Editor, WordPress renders that saved copy instead of the theme
file. An agent that edits `parts/header.html` then changes nothing the
visitor sees. #5092 told the agent this in `edit-manage-site`; #5301
moved it into Active site facts, which listed every saved copy in every
prompt.

- The notice moves into tool results, stated when a call or its result
names a theme file a saved copy replaces, for example:
> WordPress renders the template part header from its copy saved in the
Site Editor (post 12), not from
wp-content/themes/twentytwentyfive/parts/header.html.
- `cli/ai/saved-copies.ts` reads the copies from the site's SQLite
database when a theme file is named, so copies saved earlier in the turn
count. Each copy is stated once per tracker.
- It is registered in `cli/ai/result-notes.ts` (#5333). The notes now
see the call's arguments with its result (a Write names the file in its
arguments), on both surfaces.
- Active site facts keep the WordPress version; saved copies leave them.

Limits:

- MCP clients read and write theme files with their own tools, which
never pass through Studio, so for them the notice fires only when a
Studio tool names the file.
- Like every result note, a notice stated early in a turn can be lost if
the context is compacted later in that turn.

## Testing Instructions

- `npx vitest run apps/cli/ai/tests/saved-copies.test.ts`: a real SQLite
database with a saved template part and Global Styles; notices for the
theme files they replace, none for other files, and each copy once; run
with `wp_` tables and with another prefix, as imported sites can have.
- Eval (`/tmp` override kit, 6 runs per case, Studio's default model): a
site whose header part and Global Styles were saved in the Site Editor;
the task edits the header, then the styles. Scored from the rendered
site.

  | Case | #5301 (facts in prompt) | This PR |
  |---|---|---|
  | Template part saved in the Site Editor | 6/6 | 6/6 |
  | Global Styles saved in the Site Editor | 4/4 | 6/6 |

The notice appeared in all 12 transcripts. An earlier run of this change
scored 5/6 on Global Styles; the miss was broken JSON the agent wrote to
a file and saved with `post update`, which the inline `--post_content`
JSON check (#5310) does not cover.

- Eval on site edits with no saved copies (8 cases from the routing
eval: contact form, carousel, newsletter, heading font, testimonials,
header search block, hero, spacing; each run stops at its first write).
Before #5301, `edit-manage-site` told the agent to check for saved
copies first; #5092's review asked what that costs.

| | Sessions that queried saved copies | Queries | Tool calls before the
first write |
  |---|---|---|---|
  | Before #5301 (Oct 22 runs) | 38/126 | 43 | 14.3 |
  | This PR | 0/32 | 0 | 14.1 |

The queries are gone, but calls before the first write are unchanged:
the agent spends them on other reading. Tokens (median 134k before, 136k
now) and wall time (median 38.6s on trunk, 36.4s now) are at parity too.

## Pre-merge Checklist

- [ ] Have you checked for TypeScript, React or other console errors?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants