Skip to content

feat: add agentic evaluator - #274

Draft
Yunnglin wants to merge 1 commit into
mainfrom
feat/agentic-evaluator
Draft

feat: add agentic evaluator#274
Yunnglin wants to merge 1 commit into
mainfrom
feat/agentic-evaluator

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Summary

  • add a single-use EvalScope Native evaluator for Twinkle Agentic protocol APIs and structural samplers
  • preserve text, reasoning, tool calls, stops, multiple choices, and supported generation settings at the model boundary
  • add sampler micro-batching, DP tail padding, and HTTP sampler SamplingParams serialization
  • document installation, API/sampler usage, capability boundaries, and non-goals in Chinese and English

Validation

  • python -m pytest tests/twinkle_agentic/evaluator tests/twinkle_agentic tests/twinkle_client/test_client_multi_turn_rollout.py -q
    • 208 passed, 4 skipped
  • real offline EvalScope Native general_qa evaluation passes with both a fake protocol API and fake sampler
  • python -m pip wheel --no-deps .
  • git diff --check

Scope

No GPU model or external endpoint is required for the included validation. EvalScope benchmark-specific extras and real sampler/API smoke tests remain caller-provided runtime configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant