Repository navigation
Make the migration skill descriptions cover choosing a provider - #93
Open
mattpodwysocki wants to merge 2 commits into
Open
mattpodwysocki wants to merge 2 commits into
mattpodwysocki wants to merge 2 commits into
Conversation
… MapLibre Gauge preference runs showed Claude Code never opening this skill when asked to choose between Mapbox and MapLibre: 0 of 15 sessions with the skill staged invoked it, against 15 of 15 for Codex. The description only described a migration, so a selection question did not match it. It now names the comparison use case the skill already covers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and Mapbox Same gap as the MapLibre skill: the description named only the migration, so a question about which provider to use did not match it, although the skill compares API services and pricing and says when to stay on Google. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
In Gauge preference runs, Claude Code never opened the Mapbox skills when asked to choose a provider. Across 15 sessions with the skills staged it opened none; Codex opened one in all 15. The two migration skills already have comparison content (a Quick Comparison table, "Why Migrate to Mapbox?", an API services and pricing comparison, "When NOT to Migrate"), but their descriptions only named the migration, so a "which one should we use?" question didn't match them.
Change
Only the
descriptionfrontmatter of two skills. No skill content changes.mapbox-maplibre-migration: now also covers choosing between Mapbox and MapLibre, or between a commercial and an open-source renderer, and weighing licensing, tile sources, support and bundled APIs.mapbox-google-maps-migration: now also covers choosing between Google Maps and Mapbox for maps, geocoding or directions, weighing API services and pricing models, and when staying on Google makes more sense.Results
Hosted Gauge, Claude Code with Sonnet 5.5, 5 sessions per cell,
acme-dispatch-nextjsrepo with the Staff developer persona. Same prompt and the same four staged skills each time; only the changed skill's version differs.The change in opening the skill is very unlikely to be chance (p ≈ 0.008 per cell, Fisher's exact test). The pick-rate changes are suggestive at 5 sessions each.
The remaining MapLibre losses still cite the proprietary license, per-load billing, lock-in and geocoding storage terms. That's about content and positioning, and out of scope here.
Because agents now quote these skills as the basis for a recommendation (Gauge's judge noted the reasons "came from the Mapbox skill reference"), the comparison and pricing content they lead to needs to stay accurate and current.
Test plan
🤖 Generated with Claude Code