Skip to content

Associate recorded evaluations with dataset selections and row origins - #147

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-dataset-sourcesfrom
refactor/eval-dataset-recording
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-dataset-sourcesfrom
refactor/eval-dataset-recording

Conversation

@Qard

Copy link
Copy Markdown
Contributor

Dataset-backed evaluations now register their experiment with the prepared dataset ID and optional snapshot version. Each recorded case root carries a dataset object reference using its own row ID, transaction and optional creation time; snapshot selection and row transaction remain distinct. Missing or empty row identities omit the origin, and child spans retain independent provenance.

The executor prepares the source once, initializes recording with the captured selection, then reads cases. Source preparation failures create no experiment; experiment registration failures consume no cases and preserve the prepared dataset in an empty partial summary. Both paths release admitted operation and read-session state.

Stacked on #146 (refactor/eval-dataset-sources), continuing the original refactor's evaluation stage. Planning documents remain local.

Validation:

  • 406 unit/integration tests and 13 doctests passed (4 existing ignored doctests).
  • Added pinned/live dataset recording, missing/empty row identity, child-span provenance, setup ordering and failure cleanup coverage.
  • Strict all-target/all-feature Clippy, rustdoc, formatting and diff checks passed.
  • Rust 1.86: focused eval tests, both eval examples and 5 eval doctests passed using an isolated compatible dependency resolution. The existing default dependency graph's MSRV mismatch remains release-preparation work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant