feat: standardize agent evaluation framework

[TomCC7]

Contribution path

• Stacked follow-up to feat: add frozen recording agent evaluation #3378

Problem

The frozen QA evaluator couples run orchestration, the Pi/CodePolicy runtime, and benchmark-specific scoring. That shape makes new evaluators and third-party benchmarks adopt DimOS scoring instead of keeping their native result semantics.

Solution

• Add a whole-Evaluation plugin seam with immutable run specifications, reports, artifacts, and entry-point discovery through dimos.evaluations.
• Centralize CodePolicy session setup, prompt assembly, progress, cancellation, redaction, and atomic report output in the shared runtime.
• Move frozen integer QA onto the new seam and score exact matches with OpenEvals.
• Let third-party benchmark evaluations own execution and result semantics while using the same CLI and report envelope.
• Replace the old dimos.benchmark.agent_eval path instead of retaining a compatibility layer.

How to Test

Run an evaluation:

dimos eval run dimos/benchmark/short_horizon_qa/cases/demo_go2_hongkong_office-room-count-smoke/run.json --output /tmp/dimos-eval-report.json

Validation completed:

• ./bin/pytest-fast -q --tb=short — 3,787 passed, 31 skipped
• focused evaluation tests — 53 passed, 1 deselected
• focused mypy checks — passed
• Pi extension typecheck and tests — passed
• targeted pre-commit hooks — passed

AI assistance

Codex with GPT-5 helped explore the design, implement the change, and run validation. The author reviewed the resulting architecture and code throughout the design discussion.

Checklist

• I have read and approved the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 91.52120% with 68 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/benchmark/evaluation/registry.py	68.42%	19 Missing and 5 partials ⚠️
dimos/benchmark/evaluation/runtime.py	83.76%	11 Missing and 8 partials ⚠️
dimos/benchmark/evaluation/runner.py	86.88%	6 Missing and 2 partials ⚠️
dimos/benchmark/evaluation/models.py	92.77%	3 Missing and 3 partials ⚠️
dimos/benchmark/short_horizon_qa/evaluation.py	90.00%	2 Missing and 4 partials ⚠️
dimos/benchmark/short_horizon_qa/models.py	91.48%	2 Missing and 2 partials ⚠️
dimos/benchmark/evaluation/test_registry.py	97.67%	1 Missing ⚠️

@@                    Coverage Diff                     @@
##           cc/feat/frozen-qa-eval    #3403      +/-   ##
==========================================================
- Coverage                   76.26%   75.07%   -1.19%     
==========================================================
  Files                        1211     1219       +8     
  Lines                      116918   120782    +3864     
  Branches                    10512    11043     +531     
==========================================================
+ Hits                        89164    90677    +1513     
- Misses                      24670    26991    +2321     
- Partials                     3084     3114      +30     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.66% <91.39%> (+0.09%)	⬆️
OS-ubuntu-latest	72.45% <91.39%> (+0.08%)	⬆️
Py-3.10	72.45% <91.39%> (+0.07%)	⬆️
Py-3.11	72.44% <91.39%> (+0.07%)	⬆️
Py-3.12	72.44% <91.27%> (+0.07%)	⬆️
Py-3.13	72.44% <91.39%> (+0.07%)	⬆️
Py-3.14	72.45% <91.39%> (+0.08%)	⬆️
Py-3.14t	72.44% <91.39%> (+0.08%)	⬆️
SelfHosted-Large	29.72% <45.01%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/agents/code_policy_server.py	81.94% <ø> (ø)
dimos/benchmark/evaluation/pi_process.py	77.50% <100.00%> (ø)
dimos/benchmark/evaluation/progress.py	94.87% <ø> (ø)
dimos/benchmark/evaluation/protocol.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_models.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_pi_process.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_runner.py	100.00% <100.00%> (ø)
dimos/benchmark/evaluation/test_runtime.py	100.00% <100.00%> (ø)
dimos/benchmark/short_horizon_qa/integer_answer.py	88.88% <100.00%> (ø)
...imos/benchmark/short_horizon_qa/test_evaluation.py	100.00% <100.00%> (ø)
... and 11 more

... and 45 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.