[chubes4]

Related issues

• Measures the routing question on Studio Code: List the skills from their own descriptions instead of hand-written routing #5265: which routing rules we'd risk losing when the skills' own descriptions replace the hand-written routing.

How AI was used in this PR

Claude (via the Pi coding agent in a Roadie session) wrote the runner change, the check, the cases and their tests, and ran them. I reviewed the diff and the runs.

Proposed Changes

• stopAt in the eval runner. It takes a list of rules, each a tool name with optional regexes for the tool's input fields. The run ends at the first tool call that matches a rule, and the result's stoppedAt names that call. The matcher is generic data (apps/cli/ai/eval-stop.ts), so it covers any tool, including new or plugin-provided ones, without changes to the runner.
• skillsLoadedFirst in outcome.mjs: the skills named in a case's skills var were loaded before the run's stop call. This is the one check judged on the route, not the outcome, because which runbook the agent read is the thing under test. The README says so.
• 15 routing-* cases, each stopped at the agent's first call that does the work: Write, Edit, generate_images, or a wp_cli create/update/install. The new-site cases also stop at site_create and scaffold_theme. They cover new sites, plugin features (forms, carousels, newsletters, polls, events), typography and redesigns, sections and template parts, imagery, a layout complaint on a seeded broken homepage, and a plan question. A run costs about 100k tokens and 30 seconds, instead of a full build.

Testing Instructions

• npx vitest run apps/cli/ai/tests/eval-stop.test.ts scripts/eval/outcome.test.mjs

• npm run eval -- --filter-pattern "^routing:" runs the routing cases.

• Baseline: trunk (f60de7f0f) passes 69/75, from five runs of each case. Median 116k tokens and about a minute per run, against several minutes and 0.5–3M tokens for a full build case.

  Cases	Trunk
  New site (make, contact form, slideshow)	15/15
  Plugin features (form, carousel, newsletter, poll)	20/20
  Events page (plugin-recommendations)	1/5
  Typography (visual-design)	3/5
  Redesign, section, template part, hero image, layout complaint, plan question	30/30

  The two weak cases are known gaps on trunk, kept so a fix shows up in the suite.

• On Studio Code: List the skills from their own descriptions instead of hand-written routing #5265, these cases showed it wasn't ready: 62/75, mostly runs that loaded neither entry skill (build-site, edit-manage-site). It also found a skill still pointing at the prompt section Studio Code: List the skills from their own descriptions instead of hand-written routing #5265 removes.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

[wpmobilebot]

📊 Performance Test Results

Comparing f132f9a vs trunk

app-size

Metric	trunk	f132f9a	Diff	Change
App Size (Mac)	940.62 MB	945.21 MB	+4.58 MB	🔴 0.5%

site-editor

Metric	trunk	f132f9a	Diff	Change
load	1236 ms	1276 ms	+40 ms	⚪ 0.0%

site-startup

Metric	trunk	f132f9a	Diff	Change
siteCreation	7556 ms	7554 ms	2 ms	⚪ 0.0%
siteStartup	4389 ms	4374 ms	15 ms	⚪ 0.0%

---

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

[youknowriad]

The routing cases fail on trunk with (vars.skills || []).filter is not a function. promptfoo expands an array var into one test per element, so skills reaches skillsLoadedFirst as a string, and [build-site, plugin-recommendations] runs as two separate tests. The unit test passes the array in directly, so it doesn't catch this.

Using objects the way followUps and answers do should fix it: skills: [{ name: hosting-plans-helper }], mapped to names in the check. The agent itself was fine in my run, it loaded hosting-plans-helper first.