Skip to content

[FIX]: Evaluate probe verdicts over final traces - #149

Draft
spencrr wants to merge 5 commits into
microsoft:mainfrom
spencrr:dev/spencrr/trace-probe-cadence
Draft

[FIX]: Evaluate probe verdicts over final traces#149
spencrr wants to merge 5 commits into
microsoft:mainfrom
spencrr:dev/spencrr/trace-probe-cadence

Conversation

@spencrr

@spencrr spencrr commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description

Changes behavioral probes to evaluate the verdict once over the completed trace. Probes no longer stop on a detected verdict by default; callers can provide stop_when when an online condition should define an earlier terminal trace.

This fixes sequence and prior-condition probes that cannot be resolved correctly from partial prefixes. Final verdict evidence is stored on Result.evaluation, online stop evidence remains on turns, zero-turn runs return ERROR, and summaries use terminal rationale or evidence. Terminal evaluation stays inside the active session.

Depends on #148. Because the branches live on a fork, this PR temporarily includes lower-layer diffs and targets main; those diffs disappear as dependencies merge.

Breaking changes

Behavioral change: Result.eval_results is normally empty for probes without stop_when; use Result.evaluation for verdict evidence. Adaptive drivers without an online stop receive no evaluator feedback and may run to max_turns.

Checklist

  • pre-commit run --all-files passes
  • Tests added for one-call verdict cadence, sequence and prior-condition probes, scopes, stopping, feedback, budgets, session ordering, summaries, and error results
  • Documentation updated

Validation: 164 probe, driver, serialization, and report regressions pass; strict documentation build and static checks pass.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant