Skip to content

Own evaluation case sources and preserve partial run failures - #140

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-callable-contractsfrom
refactor/eval-case-sources
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-callable-contractsfrom
refactor/eval-case-sources

Conversation

@Qard

Copy link
Copy Markdown
Contributor

Evaluation sources cloned vector cases and required Sync/Clone even though each run reads its source sequentially. Read errors were printed before the evaluator returned a successful summary. Replace eval::Dataset with an owned CaseSource boundary and VecCaseSource, fallible IterCaseSource, and pinned StreamCaseSource adapters. Vectors move cases out and release consumed values; sources support non-Clone/non-Sync values and streams need not be Unpin. Source errors retain application causes.

EvalOpts::source replaces its dataset field. Source provenance is captured before consumption as DatasetVersionRef and retained in EvalSummary::dataset, replacing the bare dataset ID. Recorded evaluator and CLI bounds no longer require deserialization, cloning, or equality from user values.

Run failures now return EvalRunError with typed configuration/source/recording/execution/admission failures and a boxed partial summary when execution started. Source and recording positions are zero-based. The recorded run pump keeps a pending source-read future alive while driving active cases, enforces concurrency through its active case set, and returns results in source order. Source or recording failure stops new case reads, drains scheduled cases, and observes root recording completions before returning. Computation failures remain in their individual case results.

Validation: 370 unit/integration tests and 11 doctests passed with all features, plus strict all-target Clippy, rustdoc, formatting and diff checks. Seven public regressions cover ownership/release, fallible iterators, non-Unpin/non-Sync streams, empty sources, source failure before the first case, source failure with active work/provenance/finalization, and recording failure while other cases are scheduled. These replace two older vector-source unit tests. Rust 1.86 passed 18 focused eval/source tests, both examples, and three eval doctests with an isolated compatible dependency resolution; the previously identified normal-resolution MSRV issue is unchanged.

Stacked on #139, within planned area 8. Reusable definitions/plans, eager local/recorded scheduling, managed blocking callbacks, and recorder/reporter separation remain next. The transitional recorded-case adapter still treats write failures as run-level recording failures; retaining computed outcomes independently of recording status is part of the recorder extraction. Planning documents remain excluded. Related granular layers will be grouped before landing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant