Skip to content

Submit local and recorded evaluation runs eagerly - #143

Closed
Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-run-executorfrom
refactor/eager-eval-runs
Closed

Stephen Belanger (Qard) wants to merge 1 commit into
refactor/eval-run-executorfrom
refactor/eager-eval-runs

Conversation

@Qard

Copy link
Copy Markdown
Contributor

Evaluation runs now submit immediately and return the same completion token for synchronous and asynchronous callers. eval::run(EvalRequest::local(options)) needs no client, credentials or serialization; it uses the requested runtime or an owned runtime. EvalRequest::recorded(options, client) inherits the client's runtime and lifecycle. Conflicting recorded runtime overrides and invalid options are rejected synchronously.

Recorded admission reserves both a response slot and a bounded operation write lease before returning. The recorder builds spans within that admitted operation, so shutdown can close public admission while accepted callbacks and recording finish their writes. Completion, panic, runtime loss and timeout release the lease; escaped spans cannot keep writing after the run. Dropping an observer leaves accepted work running.

The client-bound Evaluator delegates to this entry point. Removed SyncEvaluator and its second runtime; the synchronous example waits for the same completion token on a plain thread. This continues stage 8, stacked on #142. Managed blocking callback adapters, experiment-backed recording, dataset-source integration and remaining CLI adapters are subsequent layers.

Validation:

  • All-feature suite: 386 unit/integration tests and 12 doctests passed.
  • Strict all-target/all-feature Clippy, strict rustdoc, formatting and diff checks passed.
  • Rust 1.86 on an isolated compatible dependency resolution: 48 focused eval tests, both examples and 4 eval doctests passed. No dependency files changed.
  • New tests cover plain-thread local runs with non-serializable/non-cloneable values, explicit/owned runtime selection, runtime loss before first poll, dropped observers, immediate configuration failures, shutdown before polling, escaped spans, callback panic and shutdown timeout.
  • The first full-suite attempt hit the unchanged upload test's first-captured-request JSON assertion. Its isolated rerun and the subsequent full-suite rerun passed; no upload source or tests changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant