feat: add frozen recording agent evaluation - #3378
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #3378 +/- ##
==========================================
+ Coverage 76.09% 76.26% +0.17%
==========================================
Files 1189 1211 +22
Lines 115284 116918 +1634
Branches 10366 10512 +146
==========================================
+ Hits 87720 89164 +1444
- Misses 24553 24670 +117
- Partials 3011 3084 +73
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 6 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Greptile SummaryThe PR adds a synchronous frozen-recording evaluation pipeline with strict case/result contracts, replay-derived map caching, read-only Memory2 access, and a constrained Pi-to-Python execution bridge.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
C[Eval case] --> B[Resolve or prepare frozen bundle]
B --> M[Read-only Memory2 overlay]
M --> P[Persistent Python MCP session]
P --> A[Pi agent]
A --> V[Private integer validator]
V --> R[Atomic result directory]
Reviews (2): Last reviewed commit: "fix: harden frozen eval CI and caching" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
Addressed the remaining Greptile findings in
Local verification: 72 tests passed (1 deselected), plus Ruff, mypy on the changed sources, and targeted pre-commit hooks. |
github-actions
Bot
added
the
ready-to-merge
2805308 to
03d9ca7
Compare
github-actions
Bot
removed
the
ready-to-merge
github-actions
Bot
added
the
ready-to-merge
| @@ -0,0 +1,7 @@ | |||
| # Pi CodePolicy extension | |||
|
|
|||
| This package adds one `python_exec` tool to the stock Pi CLI. The tool connects | |||
There was a problem hiding this comment.
Is there a reason to use Pi for agent execution? WHy not jsut use langchain client directly its already in the repo
There was a problem hiding this comment.
It's hard for me to find a single strength for our inhouse agent other than 'its already in the repo' :(
I feel like it's too 'low-level' for the features we really care about. For example, I added gpt5.6 support from api handling level in a previous PR #2999 . I also checked if there's an easy way to support codex subscription for the inhouse agent, the answer is also no.
These features are so basic for any of the 'agent runtime' framework in the field. You should be able to just load an extension and that's it (https://awesome-pi.site/extensions/). The primary reason for pi here is it's the most light-weight one.
Nevertheless, the rest of the structure is decoupled from the actual agent being used right now, the exposed mcp can be used on any agent.
There was a problem hiding this comment.
Yeah i guess i seperate agents for simple dumb question-answer versus development agents (what you use to code, typically people Bring Their Own Harness) versus Runtime agents (agents that only execute dimOS @skills at runtime.)
this example was more for whats the easiest way to query an LLM -- probably just langchain llm.query(str) or whatever their API is. since this PR is for dumb agent QA.
Pi looks like a great harness as our internal harness for development/runtime
| _TERMINAL_INTEGER = re.compile(r"(?:^|\n)ANSWER:\s*(-?\d+)\s*\Z") | ||
|
|
||
|
|
||
| class ExactIntegerOracle(BaseEvalModel): |
There was a problem hiding this comment.
These should be generic since evaluation functions will end up an infinite list if we need to define scoring for Every single type. This is for integers, what about how we eval Vector3 or Pose or RobotState?
| @pytest.mark.self_hosted | ||
| def test_real_hongkong_recording_prepares_direct_demo_case(tmp_path: Path) -> None: | ||
| case_path = ( | ||
| Path(__file__).parent / "cases" / "demo_go2_hongkong_office-room-count-smoke" / "case.json" |
There was a problem hiding this comment.
Cant reference files like this. Will break if you pip install dimos as a library
| if self._conn is None: | ||
| assert self._path is not None | ||
| disposable, self._conn = open_disposable_sqlite_connection(self._path) | ||
| disposable, self._conn = open_disposable_sqlite_connection( |
There was a problem hiding this comment.
dont touch mem2 in an evals PR
|
|
||
| class PiAgentConfig(BaseEvalModel): | ||
| backend: Literal["pi"] = "pi" | ||
| model: Literal["gpt-5.6-luna"] = "gpt-5.6-luna" |
There was a problem hiding this comment.
why literal - if not configurable then we cant eval the underlying model
| backend: Literal["pi"] = "pi" | ||
| model: Literal["gpt-5.6-luna"] = "gpt-5.6-luna" | ||
| thinking_level: Literal["medium"] = "medium" | ||
| api_key_env: str = Field(default="OPENAI_API_KEY", min_length=1) |
There was a problem hiding this comment.
im pretty sure we dont handle env vars like this
|
|
||
|
|
||
| def _pi_paths() -> tuple[Path, Path]: | ||
| package = Path(__file__).resolve().parents[3] / "packages" / "pi-code-policy-extension" |
| f"c{int(mapper.carve_columns)}-f{mapper.frame_id}-e{mapper.emit_every}" | ||
| ) | ||
| key = re.sub(r"[^A-Za-z0-9_.-]+", "-", raw_key) | ||
| bundle = CACHE_DIR / "agent_eval" / "frozen_memory" / key |
| T = TypeVar("T") | ||
|
|
||
|
|
||
| class FrozenMemoryStore(CompositeResource): |
There was a problem hiding this comment.
This is solid - needed when the agent has full tool access during interactiveEvals and will be able to "cheat:
2 participants
Contribution path
Problem
DimOS lacked a direct way to evaluate an agent against replay-derived Memory2 state and stable case/result contracts that can later support other sources, tasks, and validators. This PR delivers the first vertical slice: one integer question over a frozen recording.
Solution
DIM-1390: eval execution framework
The final replay-first path is a synchronous CLI rather than a DimOS module:
dimos eval runexecutes exactly one case without starting a robot, simulation, replay blueprint, or module graph. It prepares a frozen source/derived Memory2 overlay, hosts an in-process MCP session controller with one persistent Jupyter kernel, and runs stock Pi 0.80.10 in a Node subprocess with onlypython_execenabled.The runner keeps the oracle outside the public case, distinguishes semantic failure from infrastructure failure, streams map/Pi/tool progress live, bounds execution time, cleans up Pi/MCP/Jupyter resources, and atomically publishes
result.jsonplus optional Pi transcript and stderr diagnostics.DIM-1392: eval data structures
The final primitive is a strict, immutable, versioned Pydantic envelope. These are the implemented field definitions; validation methods are omitted:
Compared with the ticket's proposed general-purpose dataclass,
source,task/query, andvalidator/scorerremain explicit extension points.setup,actions, and per-case timeout stay out until an execution mode uses them. Prepared replay data has a separate versionedFrozenMemoryManifest, and runtime-only agent settings live inEvalRunConfig.Supporting architecture
global_mapdata in a derived sidecar, and records source identity, mapper settings, and per-stream boundaries in a manifest. Bundles are cached and revalidated before use.mode=roplusPRAGMA query_only=ON; mutation paths reject writes.FrozenMemoryStoreoverlays source and derived streams and applies one inclusive cutoff to all of them.python_exec.0, caught infrastructure failure exits1, and preflight failure exits2. Output is built in a temporary sibling and renamed into place, never merged into a nonempty directory.0, not an authoritative room count.Future live and simulation evals
The case/result envelope, Pi runner, MCP adapter, progress events, validator seam, and artifact publisher do not depend on frozen replay. A live source can replace bundle preparation with attachment to a running DimOS instance;
LiveDimosEnvironmentalready exposes read-only memory and a connectedapphandle through the same persistentpython_execboundary.A DimSim source/runner can own scene setup, seed, blueprint lifecycle, reset, actions, and telemetry, then add tagged task and validator variants for long-horizon scoring. Live and simulation evals therefore extend source-specific lifecycle runners and model unions without replacing the agent or result pipeline introduced here.
How to Test
Install the agent dependencies and build the local Pi adapter once:
Run the direct testcase:
OPENAI_API_KEY=... uv run dimos eval run \ dimos/benchmark/short_horizon_qa/cases/demo_go2_hongkong_office-room-count-smoke/case.json \ --output=/tmp/dimos-eval-smokeRun the focused automated tests:
uv run pytest \ dimos/agents/test_code_policy_core.py \ dimos/agents/test_code_policy_server.py \ dimos/benchmark/agent_eval \ dimos/benchmark/short_horizon_qa \ dimos/cli/test_eval.py \ dimos/memory2/store/test_frozen.py npm test --prefix packages/pi-code-policy-extensionCodePolicy runs trusted, unsandboxed Python. Run only trusted evaluation agents, or place the command in an OS sandbox or container.
AI assistance
Codex with GPT-5 assisted throughout design exploration, implementation, test generation, debugging, and PR drafting. The author reviewed the architecture and implementation decisions interactively.
Checklist