[chubes4]

Related issues

• Fixes the routing cases added in Eval: Stop a run at a matching tool call, and add skill routing cases #5287, reported by @youknowriad in Eval: Stop a run at a matching tool call, and add skill routing cases #5287 (comment).

Proposed Changes

promptfoo expands a var that is an array of plain values into one test per element. The routing cases listed their skills as skills: [build-site, plugin-recommendations], so each ran as one test per skill, and skillsLoadedFirst got a string and crashed on .filter. Every routing-* case failed under npm run eval.

• skills is now a list of objects, skills: [{ name: build-site }], the same shape followUps, answers and stopAt already use, and skillsLoadedFirst reads each entry's name.
• A new test in outcome.test.mjs fails when any list var in promptfoo.config.yaml holds plain values, so this can't come back for skills or a future var. It fails on trunk's config and passes here.
• The README documents the shape.

A correction to #5287: its testing instructions said npm run eval -- --filter-pattern "^routing:" runs the routing cases, and its 69/75 baseline could not have come from that command, since every case crashed under it. The unit test passed the array to the check directly, so it never saw promptfoo's expansion.

Testing Instructions

• npx vitest run scripts/eval/outcome.test.mjs: 25 passed.
• STUDIO_EVAL_MODEL=fast npm run eval -- --filter-pattern "^routing:": 15 test cases (one per routing case), 14 passed, 8m 34s. The one failure is a real routing miss, not the harness: on routing-events the agent loaded edit-manage-site and block-content, then created the page with plain blocks without loading plugin-recommendations. routing-plan-question loaded hosting-plans-helper first, as in the run on Eval: Stop a run at a matching tool call, and add skill routing cases #5287.

Pre-merge Checklist

• Have you checked for TypeScript, React or other console errors?

---

AI disclosure: Claude (Anthropic) in the Pi coding agent, via a Roadie session directed by @chubes4, made the change, added the test and ran the evals. @chubes4 reviewed the diff and the run.