Repository navigation
Isolate evaluation scheduling from recording and reporting - #142
Closed
Stephen Belanger (Qard) wants to merge 1 commit into
Closed
Stephen Belanger (Qard) wants to merge 1 commit into
Stephen Belanger (Qard) wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A case whose upload failed previously lost its computed output from the partial evaluation summary. The run executor now always retains completed case outcomes and reports recording failures separately. Success/failure counts describe task and scorer computation; a recording failure still makes the overall run return an error.
The recorded evaluator delegates to a validated run plan and one bounded scheduler. Braintrust serialization/write completion and terminal presentation live in separate private adapters. The scheduler supports client-free computation without serialization bounds, retains pending source reads while driving cases, preserves source order and drains scheduled work before recorder finalization. Serialization errors are now reported instead of silently omitting fields, while computed outputs and span finalization are retained.
This continues stage 8, stacked on #141. The existing public evaluator remains the transitional recorded entry point. The eager local/recorded operation API, selected-runtime blocking callbacks, experiment-backed recording and CLI protocol adapters remain subsequent work.
Validation: