Repository navigation
Adapt versioned dataset reads into evaluation case sources - #146
Merged
Stephen Belanger (Qard) merged 1 commit intoOct 28, 2026
Merged
Stephen Belanger (Qard) merged 1 commit into
Stephen Belanger (Qard) merged 1 commit into
Conversation
Stephen Belanger (Qard)
merged commit Oct 28, 2026
ba1d0db
into
refactor/eval-experiment-recording
2 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DatasetCaseSourceconnects SDK datasets to the shared eval engine without requiring Clone or Serialize on user inputs and outputs. It deserializes rows as cases are requested, preserving record IDs, row transaction IDs, timestamps, metadata and tags. Optional snapshot versions are sent on every BTQL page and captured with the dataset identity in the run summary; live sources report no pinned version.A source admits one bounded, demand-driven read session on its dataset's fixed runtime. Registration and page requests use the existing private dataset I/O boundary and pagination implementation without reserving nested response slots. The session has one queued request and one page buffer, and participates in the source client's response lifecycle. It can finish reads during shutdown when admitted beforehand; runtime loss is reported without fallback. Construction admits work immediately, so callers consume or drop an unread source before awaiting its client's shutdown.
Case sources now have a default no-op preparation hook. The executor prepares the source before capturing provenance, preserves typed preparation/page/decode errors and drains scheduled cases into partial results. Snapshot selection uses the existing BTQL top-level
versionrequest field, matching the primary JS SDK fetch implementation. Dataset provenance and case identities are retained for subsequent recording/CLI integration.This continues stage 8, stacked on #145. Also corrected the recurring upload-test assertion to select
/logs3/overflowexplicitly instead of assuming the first captured request is the grant. Upload behavior is unchanged.Validation: