Skip to content

Adapt versioned dataset reads into evaluation case sources - #146

Merged
Stephen Belanger (Qard) merged 1 commit into
refactor/eval-experiment-recordingfrom
refactor/eval-dataset-sources
Oct 28, 2026
Merged

Stephen Belanger (Qard) merged 1 commit into
refactor/eval-experiment-recordingfrom
refactor/eval-dataset-sources

Conversation

@Qard

Copy link
Copy Markdown
Contributor

DatasetCaseSource connects SDK datasets to the shared eval engine without requiring Clone or Serialize on user inputs and outputs. It deserializes rows as cases are requested, preserving record IDs, row transaction IDs, timestamps, metadata and tags. Optional snapshot versions are sent on every BTQL page and captured with the dataset identity in the run summary; live sources report no pinned version.

A source admits one bounded, demand-driven read session on its dataset's fixed runtime. Registration and page requests use the existing private dataset I/O boundary and pagination implementation without reserving nested response slots. The session has one queued request and one page buffer, and participates in the source client's response lifecycle. It can finish reads during shutdown when admitted beforehand; runtime loss is reported without fallback. Construction admits work immediately, so callers consume or drop an unread source before awaiting its client's shutdown.

Case sources now have a default no-op preparation hook. The executor prepares the source before capturing provenance, preserves typed preparation/page/decode errors and drains scheduled cases into partial results. Snapshot selection uses the existing BTQL top-level version request field, matching the primary JS SDK fetch implementation. Dataset provenance and case identities are retained for subsequent recording/CLI integration.

This continues stage 8, stacked on #145. Also corrected the recurring upload-test assertion to select /logs3/overflow explicitly instead of assuming the first captured request is the grant. Upload behavior is unchanged.

Validation:

  • 403 unit/integration tests and 13 doctests passed with all features.
  • Strict all-target/all-feature Clippy, strict rustdoc, formatting and diff checks passed.
  • Rust 1.86 with an isolated compatible dependency resolution: 65 focused eval tests, both examples and 5 eval doctests passed. No dependency files changed.
  • New tests cover pinned and live pagination, empty continuation pages and empty datasets, separate control/data routes and dataset credentials, non-cloneable/deserialization-only values, shutdown before polling, typed registration and decode errors, partial results after page failure, dropping unread sources, closed admission and source-runtime loss.

@Qard
Stephen Belanger (Qard) merged commit ba1d0db into refactor/eval-experiment-recording Oct 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant