Skip to content

Separate reusable evaluation definitions from run options - #141

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-case-sourcesfrom
refactor/eval-definitions
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-case-sourcesfrom
refactor/eval-definitions

Conversation

@Qard

Copy link
Copy Markdown
Contributor

Evaluation options previously owned the task and scorers, so running another source required rebuilding or manually wrapping the callbacks. EvalDefinition now owns a validated, shared definition; each EvalOpts supplies that definition alongside its own source and run options. Cloning a definition shares callbacks without requiring cloneable inputs or outputs.

Scorer names are captured once and rejected when empty or duplicated. The recorded evaluator uses the shared definition, while each invocation retains independent hooks, results and recording spans. Examples and documentation show constructing the definition separately from run options.

This continues stage 8 of the release refactor, stacked on #140. Eager run scheduling, managed blocking callbacks and recorder extraction remain subsequent layers.

Validation:

  • All-feature test suite: 373 unit/integration tests and 11 doctests passed.
  • Strict Clippy for all targets/features, strict rustdoc, formatting and diff checks passed.
  • Rust 1.86 with an isolated compatible dependency resolution: 35 focused eval tests, both examples and 2 eval-specific doctests passed. The default dependency-resolution MSRV issue remains tracked for release preparation; no dependency files changed.
  • Regression coverage exercises concurrent and successive recorded runs, shared callback state, independent sources/hooks/results, non-cloneable local values and scorer names changed through interior state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant