Skip to content

Add search evaluation onboarding skill - #67

Merged
vshulman merged 2 commits into
mainfrom
james-gtm-965-create-a-search-evaluation-skill-from-public-documentation
Oct 21, 2026
Merged

vshulman merged 2 commits into
mainfrom
james-gtm-965-create-a-search-evaluation-skill-from-public-documentation

Conversation

@jliounis

Copy link
Copy Markdown
Contributor

Adds parallel-search-evaluation, an external onboarding skill based on the public search evaluation guide. It guides agents through product selection, representative gold sets, controlled single-hop/multi-hop comparisons, pilot inspection, grading, failure diagnosis, and end-to-end cost/latency reporting.

The bundled artifact contract defines dataset isolation, run manifests, trial traces, and reporting fields. README examples and a project discovery link expose the skill; the existing catalog and archive builders pick it up automatically. Current API details and pricing remain linked to public documentation.

Validation:

  • All 42 repository tests pass with the released Parallel CLI installed.
  • Markdown lint, skill frontmatter validation, and TypeScript type checking pass.
  • Catalog and archive builds pass; verified the new catalog entry and both files in its archive.
  • A fresh-agent onboarding trial produced a concrete plan with mode comparisons, three repetitions, gold isolation, grading and failure policies, and complete cost accounting. No paid evaluation calls were made.

Closes GTM-965.

@jliounis
jliounis requested a review from a team October 20, 2026 19:20
@vshulman
vshulman merged commit 6c87302 into main Oct 21, 2026
1 check passed
@vshulman
vshulman deleted the james-gtm-965-create-a-search-evaluation-skill-from-public-documentation branch October 21, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants