Repository navigation
Own evaluation case sources and preserve partial run failures - #140
Closed
Stephen Belanger (Qard) wants to merge 1 commit into
Closed
Stephen Belanger (Qard) wants to merge 1 commit into
Stephen Belanger (Qard) wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Evaluation sources cloned vector cases and required Sync/Clone even though each run reads its source sequentially. Read errors were printed before the evaluator returned a successful summary. Replace
eval::Datasetwith an ownedCaseSourceboundary andVecCaseSource, fallibleIterCaseSource, and pinnedStreamCaseSourceadapters. Vectors move cases out and release consumed values; sources support non-Clone/non-Sync values and streams need not be Unpin. Source errors retain application causes.EvalOpts::sourcereplaces its dataset field. Source provenance is captured before consumption asDatasetVersionRefand retained inEvalSummary::dataset, replacing the bare dataset ID. Recorded evaluator and CLI bounds no longer require deserialization, cloning, or equality from user values.Run failures now return
EvalRunErrorwith typed configuration/source/recording/execution/admission failures and a boxed partial summary when execution started. Source and recording positions are zero-based. The recorded run pump keeps a pending source-read future alive while driving active cases, enforces concurrency through its active case set, and returns results in source order. Source or recording failure stops new case reads, drains scheduled cases, and observes root recording completions before returning. Computation failures remain in their individual case results.Validation: 370 unit/integration tests and 11 doctests passed with all features, plus strict all-target Clippy, rustdoc, formatting and diff checks. Seven public regressions cover ownership/release, fallible iterators, non-Unpin/non-Sync streams, empty sources, source failure before the first case, source failure with active work/provenance/finalization, and recording failure while other cases are scheduled. These replace two older vector-source unit tests. Rust 1.86 passed 18 focused eval/source tests, both examples, and three eval doctests with an isolated compatible dependency resolution; the previously identified normal-resolution MSRV issue is unchanged.
Stacked on #139, within planned area 8. Reusable definitions/plans, eager local/recorded scheduling, managed blocking callbacks, and recorder/reporter separation remain next. The transitional recorded-case adapter still treats write failures as run-level recording failures; retaining computed outcomes independently of recording status is part of the recorder extraction. Planning documents remain excluded. Related granular layers will be grouped before landing.