Skip to content

Make CLI evaluations eager and join accepted session work - #150

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-reportingfrom
refactor/eager-cli-evaluations
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-reportingfrom
refactor/eager-cli-evaluations

Conversation

@Qard

Copy link
Copy Markdown
Contributor

CLI evaluations previously started only when their futures were polled, so dropping an observer skipped work and finish() could not join unobserved submissions. BtEvalRunner::eval and finish now return eager completion tokens. Runs execute in call order, and finish joins accepted computation, recording, output and token publication before closing SSE and releasing session ownership.

CLI and canonical evaluation entry points share the same single-response submission helpers and executor. Recorded work acquires scoped write authority at issuance, allowing already-admitted queued runs to finish after client shutdown starts. Local sessions own one runtime; supplied clients provide their fixed binding. Per-run CLI runtime overrides are rejected, and finishing does not close a supplied client.

Terminate-on-failure retains typed case/partial run outcomes and skips later queued sources. An ordered failure cutoff preserves work accepted before a later issuance rejection. Non-terminating mode keeps its existing continue-and-report-status behavior. Live ordering state retains no completed outcomes, full admission still leaves finish control capacity, and panic/runtime loss releases the ordering guard. A narrow internal finalization hook releases that guard after response publication, including cancellation races. Removed the redundant lazy run_all helper and moved adapter tests to a separate file.

Stacked on #149 (refactor/eval-reporting), continuing the original release-preparation stage. Constructor/API consolidation, dependency/MSRV resolution and packaging remain subsequent work. Planning notes remain local.

Validation:

  • 430 unit/integration tests and 14 doctests passed (4 existing ignored doctests).
  • Strict all-target/all-feature Clippy, rustdoc, formatting and diff checks passed.
  • Coverage includes discarded run/finish observers, ordering and stop cutoffs, 1,024 single-slot admissions plus finish control capacity, partial results, source-read suppression, panic/runtime loss, publication cancellation, recorded work across shutdown, shared-client reuse and SSE closure before finish publication.
  • Rust 1.86: 91 focused eval tests plus the publication-cancellation test, both eval examples and 6 eval doctests passed with an isolated compatible dependency resolution. The default dependency graph's existing MSRV mismatch remains release-preparation work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant