Skip to content

Studio Code: Refuse writes to a Studio site's WordPress core instead of forbidding them in the prompt - #5164

Merged
chubes4 merged 6 commits into
trunkfrom
studio-code/refuse-core-file-writes
Oct 21, 2026
Merged

chubes4 merged 6 commits into
trunkfrom
studio-code/refuse-core-file-writes

Conversation

@chubes4

@chubes4 chubes4 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Related issues

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) went through the system prompt rule by rule looking for rules a tool can enforce, then wrote the change and its tests and ran the checks below.

Proposed Changes

The prompt said "Do NOT modify WordPress core files. Only work within wp-content/". This replaces that rule with a check in the Write and Edit tools.

  • Write and Edit refuse a Studio site's core files: wp-includes/, wp-admin/, and the root core PHP files such as wp-settings.php. A WordPress update replaces these, so a change there is lost. The refusal says why and names where the change belongs: wp-content/ (a theme, child theme or plugin).
  • Only Studio sites are guarded. The check finds the WordPress install around the file (the nearest folder with both wp-includes and wp-content) and refuses only if Studio manages that folder (findSiteByFolder). WordPress trees that Studio doesn't manage stay writable, so core development elsewhere on the machine is unaffected.
  • wp-config.php stays writable. It is the site's own file, and setting a constant there is legitimate.
  • The rule leaves the prompt entirely. Studio Code's tools enforce it. External agents connected over MCP use their own file tools and get no core-file line: they have no reason to edit core unless asked.

The check doesn't cover shell commands (for example sed -i on a core file through Bash).

Testing Instructions

  • npx vitest run apps/cli/ai/tests/file-write-refusal.test.ts. It covers:
    • refused core paths in a Studio site, relative and absolute, with wp-content/ named in the refusal;
    • allowed paths, including wp-config.php, wp-content/ files, and core paths in a WordPress tree Studio doesn't manage;
    • the default lookup through Studio's site list;
    • neither prompt (Studio Code or external) mentions core files.
  • npx vitest run apps/cli/ai: 258 tests pass.
  • Eval, core-file case ("The admin bar on this site is too short. Make it 40px tall. Its styles are in wp-includes/css/admin-bar.css."), judged on whether core stays intact (wp core verify-checksums) and the change lands in wp-content/:
Arm Core left intact Change made in wp-content/
No rule and no check 0/4 (every run edited core) 0/4
Trunk (the prose rule) 2/2 1/2 (one run left core alone but never made the change)
This branch (the check) 4/4 (each core edit refused once) 4/4

Without either, the agent edits core whenever it is pointed there, and the change is lost at the next WordPress update. The refusal names where the change belongs, which is what got every run to finish in wp-content/.

  • After merging trunk: npx vitest run apps/cli/ai, 272 pass.

Pre-merge Checklist

  • Have you checked for TypeScript, React or other console errors?

chubes4 added a commit that referenced this pull request Oct 16, 2026
…#5165)

## Related issues

- Related to #5146: one of a series of small changes that move system
prompt rules into tool structure

## How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) went through the
system prompt rule by rule looking for rules a tool can enforce, then
wrote the change and its tests and ran the checks below. Opened as a
draft for human review.

## Proposed Changes

- In a terminal session, the prompt carried a paragraph saying the user
may not see screenshots, so the agent should not reply as if they had
("Here's your site!").
- `take_screenshot` now states that fact in its result, at the moment it
matters: "The user does not see this capture: the terminal shows them
only the saved file path." The desktop app (which shows the capture) and
MCP clients (which get image links to show) don't get the line.
- The paragraph is removed from the prompt, cutting the terminal prompt
by about 420 characters.

## Testing Instructions

- `npx vitest run apps/cli/ai/tests/tools.test.ts -t "saved file path"`
checks which sessions get the line.

Targeted eval ("Take a screenshot of this site's homepage and show it to
me", run in the terminal, judged on the final reply): with this change
merged alongside #5162–#5164, the agent took the screenshot and pointed
the user to the saved file on 2/2 runs, and never claimed the image was
shown. This matches trunk (2/2) without the prompt rule.

## Pre-merge Checklist

- [ ] Have you checked for TypeScript, React or other console errors?
@chubes4
chubes4 force-pushed the studio-code/refuse-core-file-writes branch from f652cf1 to dc14756 Compare October 17, 2026 04:05
@chubes4
chubes4 marked this pull request as ready for review October 17, 2026 04:08
@wpmobilebot

wpmobilebot commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing d5ba4bb vs trunk

app-size

Metric trunk d5ba4bb Diff Change
App Size (Mac) 922.00 MB 922.01 MB +0.00 MB ⚪ 0.0%

site-editor

Metric trunk d5ba4bb Diff Change
load 1213 ms 1242 ms +29 ms ⚪ 0.0%

site-startup

Metric trunk d5ba4bb Diff Change
siteCreation 8595 ms 8582 ms 13 ms ⚪ 0.0%
siteStartup 5891 ms 5892 ms +1 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

…ore-file-writes

# Conflicts:
#	apps/cli/ai/system-prompt.ts
@chubes4 chubes4 changed the title Studio Code: Refuse writes to WordPress core files instead of forbidding them in the prompt Studio Code: Refuse writes to a Studio site's WordPress core instead of forbidding them in the prompt Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants