Skip to content

Isolate evaluation scheduling from recording and reporting - #142

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-definitionsfrom
refactor/eval-run-executor
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-definitionsfrom
refactor/eval-run-executor

Conversation

@Qard

Copy link
Copy Markdown
Contributor

A case whose upload failed previously lost its computed output from the partial evaluation summary. The run executor now always retains completed case outcomes and reports recording failures separately. Success/failure counts describe task and scorer computation; a recording failure still makes the overall run return an error.

The recorded evaluator delegates to a validated run plan and one bounded scheduler. Braintrust serialization/write completion and terminal presentation live in separate private adapters. The scheduler supports client-free computation without serialization bounds, retains pending source reads while driving cases, preserves source order and drains scheduled work before recorder finalization. Serialization errors are now reported instead of silently omitting fields, while computed outputs and span finalization are retained.

This continues stage 8, stacked on #141. The existing public evaluator remains the transitional recorded entry point. The eager local/recorded operation API, selected-runtime blocking callbacks, experiment-backed recording and CLI protocol adapters remain subsequent work.

Validation:

  • 377 unit/integration tests and 11 doctests passed with all features.
  • Strict Clippy for all targets/features, strict rustdoc, formatting and diff checks passed.
  • Rust 1.86 with an isolated compatible dependency resolution: 39 focused eval tests, both examples and 3 eval doctests passed. No dependency files changed; default dependency-resolution MSRV reconciliation remains release preparation.
  • New coverage exercises local execution without Tokio/client/serialization/Clone/Debug requirements, bounded concurrency and source ordering, a source waiting for case progress, recorder start/finish failure, and serialization failure with retained outputs and completed recording spans.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant