Skip to content

Separate evaluation reporting and route CLI output through run adapters - #149

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-dataset-recordingfrom
refactor/eval-reporting
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-dataset-recordingfrom
refactor/eval-reporting

Conversation

@Qard

Copy link
Copy Markdown
Contributor

Evaluation requests now own their output adapter. A public fallible EvalReporter separates terminal, JSONL and CLI SSE presentation from case execution and recording. Local/custom reporters require neither serialization nor Sync; JSONL adds serialization only when selected.

The first reporter error disables that reporter for the run while computation and recording continue. Completion returns a typed reporting stage/index and preserves the computed summary. Source and reporting errors survive together, dropped observers do not cancel reporting, and output failures do not poison client shutdown. summary_to_value now returns serialization errors instead of a null placeholder.

BtEvalRunner selects reporters and submits through the canonical local/recorded entry point. Both local environment flags avoid creating a client, list/filter modes skip execution, and JSONL/SSE replace terminal progress. SSE preserves the existing start/summary/error event format and reports write/flush failures.

Stacked on #147 (refactor/eval-dataset-recording), completing the reporting portion of the original evaluation stage. Remaining CLI completion/lifecycle consolidation belongs to release preparation. Planning notes remain local.

Validation:

  • 419 unit/integration tests and 14 doctests passed (4 existing ignored doctests).
  • Process-isolated CLI environment/stdout tests cover both local flags, list/filter behavior and clean JSONL. Reporter tests cover errors at every stage, serialization/write/flush failures, independent nonserializable local values, dropped observers and recorded-write/shutdown completion after output failure.
  • Strict all-target/all-feature Clippy, rustdoc, formatting and diff checks passed.
  • Rust 1.86: 81 focused eval tests, both eval examples and 6 eval doctests passed using an isolated compatible dependency resolution. The default dependency graph's existing MSRV mismatch remains release-preparation work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant