Repository navigation
Conversation
…and-written routing Each skill's description now names the work it covers and when to load it, and the prompt's Skills section is generated from those descriptions, the same way the tool list is generated from the tools. The hand-written Skill routing paragraphs are gone, and the Skill tool no longer repeats the list. MCP clients get the same list through studio_instructions.
Contributor
|
That sounds like a good move. Any potential routing rules that we'd risk losing here? |
Contributor
Author
|
I don't think we will lose anything, this just simplifies and saves context. I will stress test it with more evals and make sure the routing still works as expected before merge. |
1 task done
chubes4
added a commit
that referenced
this pull request
Oct 22, 2026
…#5287) ## Related issues - Measures the routing question on #5265: which routing rules we'd risk losing when the skills' own descriptions replace the hand-written routing. ## How AI was used in this PR Claude (via the Pi coding agent in a Roadie session) wrote the runner change, the check, the cases and their tests, and ran them. I reviewed the diff and the runs. ## Proposed Changes - **`stopAt` in the eval runner.** It takes a list of rules, each a tool name with optional regexes for the tool's input fields. The run ends at the first tool call that matches a rule, and the result's `stoppedAt` names that call. The matcher is generic data (`apps/cli/ai/eval-stop.ts`), so it covers any tool, including new or plugin-provided ones, without changes to the runner. - **`skillsLoadedFirst` in `outcome.mjs`:** the skills named in a case's `skills` var were loaded before the run's stop call. This is the one check judged on the route, not the outcome, because which runbook the agent read is the thing under test. The README says so. - **15 `routing-*` cases**, each stopped at the agent's first call that does the work: `Write`, `Edit`, `generate_images`, or a `wp_cli` create/update/install. The new-site cases also stop at `site_create` and `scaffold_theme`. They cover new sites, plugin features (forms, carousels, newsletters, polls, events), typography and redesigns, sections and template parts, imagery, a layout complaint on a seeded broken homepage, and a plan question. A run costs about 100k tokens and 30 seconds, instead of a full build. ## Testing Instructions - `npx vitest run apps/cli/ai/tests/eval-stop.test.ts scripts/eval/outcome.test.mjs` - `npm run eval -- --filter-pattern "^routing:"` runs the routing cases. - Baseline: trunk (`f60de7f0f`) passes 69/75, from five runs of each case. Median 116k tokens and about a minute per run, against several minutes and 0.5–3M tokens for a full build case. | Cases | Trunk | |---|---| | New site (make, contact form, slideshow) | 15/15 | | Plugin features (form, carousel, newsletter, poll) | 20/20 | | Events page (`plugin-recommendations`) | 1/5 | | Typography (`visual-design`) | 3/5 | | Redesign, section, template part, hero image, layout complaint, plan question | 30/30 | The two weak cases are known gaps on trunk, kept so a fix shows up in the suite. - On #5265, these cases showed it wasn't ready: 62/75, mostly runs that loaded neither entry skill (`build-site`, `edit-manage-site`). It also found a skill still pointing at the prompt section #5265 removes. ## Pre-merge Checklist - [x] Have you checked for TypeScript, React or other console errors?
1 task done
Contributor
Author
|
Closing to prove the full architecture in #5146 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
promptSnippet), Studio Code: Refuse writes to a Studio site's WordPress core instead of forbidding them in the prompt #5164, and Studio Code: Reload the preview in the app when the agent finishes, instead of asking the agent to #5171.How AI was used in this PR
Claude (via the Pi coding agent in a Roadie session) wrote the change and the tests, and ran the evaluation below. I reviewed the diff and the eval runs.
Proposed Changes
The prompt's "Skill routing" section was about 2.9k characters of hand-written paragraphs saying which skill to load for which request, separate from the skills it routed to. The Skill tool's description also repeated every skill's description. Now each skill states its own trigger:
nameanddescription, the same way the tool list is generated from the tools. One line states what skills are: "Skills are runbooks for the work their descriptions name. Load a skill before starting that work."build-site,visual-design,block-content,imagery,visual-polish, andplugin-recommendations. Their remaining rules (make images before the markup, never use web image URLs, run every build step) were already in the skill bodies.studio_instructions. Thebuild-sitestub inplugins/wp-studiofollows its new description.Size, counting the prompt plus the Skill tool description: 9,128 → 6,969 characters for Studio Code (−24%). For MCP clients the instructions grow from 4,624 to 6,507 characters, because they now list every skill's description instead of the routing paragraphs.
Testing Instructions
npx vitest run apps/cli/ai: the prompt lists every skill with its description in both runtimes, and nothing outside the Skills section names a skill.ab00abf70) and this branch, judged by the suite's outcome checks. Skills loaded are counted from theSkilltool calls.plugin-recommendations→ Jetpack)plugin-recommendations→ WooCommerce)plugin-recommendationsloaded on slideshow and shopThe first branch round lost a trigger. On "Create a new site… and add a slideshow",
build-siteloaded 1/3 times (trunk 3/3), because its description said "a whole website" while the old routing made creating any site the trigger. One run then saved a slideshow block the editor rejects.build-site's description now names a new site as its trigger, and the slideshow and palette cases were rerun on that version (the table above).build-sitethen loaded in 2/3 slideshow runs, and all runs passed. The shop runs are from the first round, wherebuild-siteloaded every time. The token medians come from three runs each and vary widely between runs.Pre-merge Checklist