Repository navigation
Report the skills a turn loads in studio_code_turn_completed - #5407
Merged
Merged
Conversation
studio_code_message_sent's ability_name names a skill only when the user starts it with a slash command. A skill the agent loads itself, through the Skill tool, reached Tracks nowhere. Since 2026-09-25, 35,714 of 35,833 studio_code_message_sent events carried no ability_name. The turn stats tracker now reads the Skill tool's tool_execution_start and reports skills_loaded: the sorted, de-duplicated names, empty when none. Only names findSkill() knows are kept, so no model-written text reaches Tracks, as with failed_tools.
The empty case repeated the zero-props test, and the index test's line asserted nothing toMatchObject checks. The other-tool case now uses site_create, a tool that takes a name.
chubes4
marked this pull request as ready for review
October 25, 2026 16:13
chubes4
added a commit
that referenced
this pull request
Oct 25, 2026
Trunk's #5407 test loads block-content, which the spike removed, so the merge left it failing; it now loads rank-me-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Claude (via the Pi coding agent in a Roadie session) found the gap by reading the Tracks code and querying the events, wrote the change and its tests, and checked that each test catches a mutation of the code it covers. Chris Huber directed the work and reviewed the diff.
Proposed Changes
studio_code_message_sent'sability_namenames a skill only when the user starts it with a slash command. A skill the agent loads itself through theSkilltool, which is how skills mostly get used, is recorded nowhere. Since 2026-09-25, 35,714 of 35,819studio_code_message_sentevents carry noability_name. So we can't tell which skills get used, how often, or whether turns that load one go better.studio_code_turn_completedgainsskills_loaded: the sorted, de-duplicated names of the skills the turn loaded, empty when none. It reads theSkilltool's arguments and keeps only names of skills Studio ships, so no model-written text reaches Tracks, the same wayfailed_toolsreportsunknown.docs/design-docs/analytics-tracks.mddocuments the prop. As that doc notes, it needs registering in the Tracks Registration tool.Testing Instructions
npx vitest run apps/cli/ai/tests/turn-stats.test.ts apps/cli/commands/ai/tests/index.test.tsstudio code, ask for something that loads a skill (for example "build me a bakery site"), and check that thestudio_code_turn_completedevent'sskills_loadednames it (for examplebuild-site,visual-design).Pre-merge Checklist
skills_loadedforstudio_code_turn_completedin the Tracks Registration tool.