feat(evals): passive/interactive agent eval framework over memory2 by spomichter · Pull Request #3411 · dimensionalOS/dimos

spomichter · 2026-08-09T04:57:29Z

Problem

Eval framework, supports both InteractiveEvals and PassiveEvals, @paul-nechifor old dimsim spatial memory eval replicated via the new framework here pretty clean:

BED = Vector3(-3.567, -1.332, 0.0)

go_to_bed = InteractiveEval(
    id="go_to_bed",
    inputs="go to the bed",
    score=lambda s: ramp((BED - s.streams.odom.last().data.position).length(), band=2.0),
    aggregate=final,          # or floor ("never left the zone") / mean / auc later
    blueprint="unitree-go2-agentic go2-memory",
    simulator="dimsim",
    scene="apartment",
)

Uses langchain openevals for VQA formatting so its compatible with all public benchmarks we'd want to run. Scoring is generic and done via lambda one-liners. Agent loop runs LLM queries with a light agent_loop in the EvalRunner -- kept minimal and using langchain client since just doing dumb QA.

Closes DIM-1392, DIM-1390

Solution

new dimos/evals/ package. one word (evals) everywhere, no benchmark vs eval split
two case types: PassiveEval (frozen mem2 recording, any replay window, non-autoregressive) and InteractiveEval (live sim/robot, actions mutate state, scored by sampling teh live recorder store)
memory2 the source of truth for all eval input — context selectors return real Streams, no copies, no parallel data structres
scoring is jsut lambdas over typed msg arithmetic (within, ramp, Vector3 subtraction etc), graded [0,1] credit not only pass/fail
we use the langchain/openevals standard for vqa formatting (inputs / reference_outputs, evaluators wrapped not subclassed) so external benchmarks map on natively and we can test against them
generic typing: PassiveEval[T] ties expected/parse/score togehter so mypy catches a mismatched case at suite definition time, not mid-run
runnable via cli (dimos evals run ...), mcp (EvalModule skills so coding agents can iterate), and pytest
--blind ablation built in — same suite with observations withheld, alredy caught two guessable MCQs
MoondreamChat adapter so evals run keyless on any gpu box

defining evals

passive one-off over a replay — the whole thing is one literal:

from dimos.evals.scorers import first_number, within, yes_no, exact
from dimos.evals.types import PassiveEval, InteractiveEval, Suite

# numeric, graded: 1.0 exact, linear to 0 at band
distance = PassiveEval(
    id="distance_10min",
    inputs="How far have you traveled in the last 10 minutes, in meters?",
    expected=142.0,
    parse=first_number,
    score=within(15.0),
    context=(lambda s: s.streams.odom.range_time(0, 600),),   # real mem2 stream
    dataset="go2_hongkong_office",
)

# vqa over a 10-image window
person = PassiveEval(
    id="person_visible",
    inputs="Is a person visible in any of these images?",
    expected="yes",
    parse=yes_no,
    score=exact,
    context=(lambda s: s.streams.color_image.range_time(58, 61).limit(10),),
    dataset="go2_short",
)

interactive — the case names its environment, score is sampled from the live mem2 store every interval_s and reduced by aggregate:

from dimos.evals.scorers import final, ramp
from dimos.msgs.geometry_msgs.Vector3 import Vector3

BED = Vector3(-3.567, -1.332, 0.0)

go_to_bed = InteractiveEval(
    id="go_to_bed",
    inputs="go to the bed",
    score=lambda s: ramp((BED - s.streams.odom.last().data.position).length(), band=2.0),
    aggregate=final,          # or floor ("never left the zone") / mean / auc later
    blueprint="unitree-go2-agentic go2-memory",
    simulator="dimsim",
    scene="apartment",
)

running

from dimos.evals.runner import EvalRunner, summarize

results = EvalRunner().run(SUITE)                    # prod model config (gpt-5.6-luna)
results = EvalRunner(blind=True).run(SUITE)          # guessing ablation
results = EvalRunner(chat_model=MoondreamChat()).run(SUITE)  # keyless local
print(summarize(results))

or dimos evals run dimos.evals.suites.go2_smoke --blind. every run writes results.jsonl + summary.json + per-case transcripts to ~/.local/state/dimos/evals/run-*/.

first real numbers

luna sighted: examples 1.00 / smoke 1.00 / vqa 0.86, blind = refusals. interactive rig scored 0.856 on a scripted go-to-bed in dimsim. the full agentic go-to-bed scores 0.0 right now — unitree-go2-agentic publishes no /odom in dimsim on current main (upstream test_dimsim_spatial_memory fails teh same way, suspect the control coordinator refactor). separate ticket coming.

Breaking Changes

None

How to Test

uv run pytest dimos/evals dimos/codebase_checks -q          # offline unit + wiring tests, no keys
uv run pytest dimos/evals/test_smoke.py -m self_hosted      # live model smoke (needs OPENAI_API_KEY + lfs data)
dimos evals run dimos.evals.suites.examples                 # 2 doc cases against go2_short
dimos evals run dimos.evals.suites.go2_smoke --blind        # guessing ablation

interactive (needs deno + display): dimos evals run dimos.evals.suites.dimsim_house --live-db recording_go2.db

Contributor License Agreement

I have read and approved the CLA

EvalCase/PassiveEval/InteractiveEval with EvalRig protocol dispatch, EvalRunner implementing the rig (model call / mcp skill / agent loop / live-store sampling), scorers as plain functions wrapping openevals, generated + hand VQA suites over go2 replays, dimsim go-to-bed interactive suite, dimos evals CLI + EvalModule MCP skills. extracts _init_model to dimos/agents/model.py for shared use.

…e-store sampling)

…ackages, no __all__)

codecov · 2026-08-09T05:04:26Z

Codecov Report

❌ Patch coverage is 80.95238% with 132 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/evals/runner.py	68.26%	65 Missing and 8 partials ⚠️
dimos/evals/suites/dimsim_house.py	40.00%	21 Missing ⚠️
dimos/evals/cli.py	33.33%	16 Missing ⚠️
dimos/evals/scorers.py	78.94%	8 Missing ⚠️
dimos/evals/types.py	91.76%	5 Missing and 2 partials ⚠️
dimos/evals/test_evals.py	97.12%	4 Missing ⚠️
dimos/evals/test_mem2_wiring.py	97.19%	1 Missing and 2 partials ⚠️

@@            Coverage Diff             @@
##             main    #3411      +/-   ##
==========================================
+ Coverage   76.05%   76.07%   +0.02%     
==========================================
  Files        1201     1212      +11     
  Lines      117016   117709     +693     
  Branches    10530    10578      +48     
==========================================
+ Hits        88996    89548     +552     
- Misses      24974    25098     +124     
- Partials     3046     3063      +17

Flag	Coverage Δ
OS-ubuntu-24.04-arm	`70.40% <78.93%> (+0.07%)`	⬆️
OS-ubuntu-latest	`72.21% <78.93%> (+0.03%)`	⬆️
Py-3.10	`72.21% <78.93%> (+0.04%)`	⬆️
Py-3.11	`72.21% <78.93%> (+0.04%)`	⬆️
Py-3.12	`72.20% <78.93%> (+0.03%)`	⬆️
Py-3.13	`72.20% <78.93%> (+0.03%)`	⬆️
Py-3.14	`72.21% <78.93%> (+0.04%)`	⬆️
Py-3.14t	`72.21% <78.93%> (+0.04%)`	⬆️
SelfHosted-Large	`29.70% <34.11%> (+0.02%)`	⬆️
SelfHosted-Linux	`35.82% <48.52%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/cli/dimos.py	`64.78% <100.00%> (+0.15%)`	⬆️
dimos/evals/suites/examples.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/go2_smoke.py	`100.00% <100.00%> (ø)`
dimos/evals/suites/go2_vqa.py	`100.00% <100.00%> (ø)`
dimos/evals/test_smoke.py	`100.00% <100.00%> (ø)`
dimos/robot/all_blueprints.py	`100.00% <ø> (ø)`
dimos/evals/test_mem2_wiring.py	`97.19% <97.19%> (ø)`
dimos/evals/test_evals.py	`97.12% <97.12%> (ø)`
dimos/evals/types.py	`91.76% <91.76%> (ø)`
dimos/evals/scorers.py	`78.94% <78.94%> (ø)`
... and 3 more

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…anges reuse mcp_client._init_model lazily instead of extracting it — keeps this PR scoped to dimos/evals (+ cli registration). extraction can be its own PR if we want it shared properly.

spomichter · 2026-08-09T08:09:03Z

@greptile review

…turbojpeg

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

TomCC7 · 2026-08-10T21:26:41Z

We need to design runtime and scorer information access more carefully. The current scheme is wrong IMO because it evaluates against Memory2 state produced by the runtime itself. To simulate real operation as faithfully as possible, the agent should have access to all information exposed by the running blueprint, including RPCs, streams, and Memory2. Scoring should instead use independent, privileged simulator state, such as ground-truth robot and object poses and object types.

…erge

paul-nechifor · 2026-08-12T01:26:06Z

@@ -0,0 +1,202 @@
+# Evals Intro


I think this should be in /docs/ as dimos evals is a command that's shipped to everyone.

mintlify · 2026-08-12T02:13:22Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
dimensional	🟢 Ready	View Preview	Aug 12, 2026, 2:14 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

paul-nechifor · 2026-08-12T03:30:45Z

+    # House convention (StoreConfig): pass an instance to inject, e.g. a fake
+    # chat model in tests. None -> built from `model` like McpClient does.
+    chat_model: Any | None = None
+    mcp_url: str = "http://localhost:9990/mcp"


mcp_port is in global_config. You should probably use that.

paul-nechifor · 2026-08-12T03:36:06Z

+    context_budget: int = 8  # max observations encoded per context Select
+    attach: bool = False  # True: drive an already-running dimos
+    launch_timeout_s: float = 1200.0  # blueprint + MCP readiness (e2e parity)
+    out_dir: Path = Path("~/.local/state/dimos/evals").expanduser()


This should use STATE_DIR / "evals" with STATE_DIR from constants.py

paul-nechifor · 2026-08-12T03:49:38Z

+            store.stop()
+        return series
+
+    def _wait_live_store(self, deadline: float) -> Store:


I don't like the way this works. It assumes that a store is on disk and also that it's a file. That's very specific to Sqlite. Other DBs don't create a single file for a DB.

I think what should happen here is that class Store should be changed to have something like Store.wait_for_creation (or a better name) and this sort of internal stuff should be performed in SqliteStore.wait_for_creation instead of in unrelated places like here.

paul-nechifor · 2026-08-12T04:01:15Z

+    def instruct(self, text: str) -> None:
+        from dimos.core.transport import pLCMTransport
+
+        transport: pLCMTransport[str] = pLCMTransport("/human_input")


You should use transport = make_transport("/human_input") since we shouldn't be strictly tied to LCM now.

paul-nechifor · 2026-08-12T04:03:00Z

+        transport.lcm.start()
+        try:
+            transport.publish(text)
+            time.sleep(0.5)  # let LCM flush before teardown


Are you sure this is an issue? I don't think I've encountered it. If it is, we should have a transport.flush() rather than a random wait as it would be a problem with transports in general, not just here.

paul-nechifor · 2026-08-12T04:05:11Z

+            transport.publish(text)
+            time.sleep(0.5)  # let LCM flush before teardown
+        finally:
+            transport.lcm.stop()


No need to call transport.lcm.start()/.stop() anymore. Just use transport.start()/.stop()

paul-nechifor · 2026-08-12T04:08:45Z

+        from dimos.evals.types import _no_setup
+
+        if case.simulator and not self.config.attach:
+            from dimos.e2e_tests.dimos_cli_call import DimosCliCall


Since dimos_cli_call is used in dimos/evals now too, it should be moved out of dimos/e2e_tests as it's no longer just for end to end testing. Maybe somewhere in dimos/cli?

True good call -- ill do a following PR just to keep this PR clean and consoludated

paul-nechifor · 2026-08-12T04:36:23Z

+            proc.global_args = ["--dimsim-scene", case.scene]
+            proc.demo_args = ["run", *case.blueprint.split()]
+            proc.start()
+            self._proc = proc


The architecture here seems a bit unclear to me.

If I understand this correctly, InteractiveEval gets an EvalRunner and sets itself up on it by calling eval_runner.setup_env(self)

I see EvalRunner as a parent object of multiple InteractiveEval objects. It's generally bad for children to control their parent object instead of the other way arround.

For example, EvalRunner.setup_env sets self._proc = DimosCliCall(). This creates a new process for every eval. When EvalRunner.stop is called, only the last self._proc is ever stopped.

This issue is obscured by the fact that an eval runs itself on the eval runner instead of the eval runner being in charge of running the evals. That is, the fact that EvalRunner._proc is leaked is hard to see because it's ultimately set from InteractiveEval.evaluate.

But also, _proc probably doesn't belong on EvalRunner. It should be on InteractiveEval. Evals should probably have a life cycle where they can be created and destroyed.

Edit: So EvalRunner.run does in fact run the evals, but also the evals know too much about the eval runner and store state on it because they can't store state on themselves.

spomichter added 5 commits August 8, 2026 19:39

feat(evals): keyless local eval model (moondream2 chat adapter)

4ea607a

test(evals): mem2 wiring integration tests (passive prompt path + liv…

f99a493

…e-store sampling)

fix(evals): wait for odom before dimsim house exploration

09d0f5d

chore(evals): drop __init__.py files per repo convention (namespace p…

a6e809a

…ackages, no __all__)

spomichter added 3 commits August 8, 2026 22:42

chore(evals): keep model construction private to evals, no agents/ ch…

48c36de

…anges reuse mcp_client._init_model lazily instead of extracting it — keeps this PR scoped to dimos/evals (+ cli registration). extraction can be its own PR if we want it shared properly.

fix blueprint test

c67a2da

docs(evals): runnable intro (memory2 doc conventions)

0ba6f92

spomichter marked this pull request as ready for review August 9, 2026 07:42

spomichter requested review from Dreamsorcerer, leshy, mustafab0 and paul-nechifor as code owners August 9, 2026 07:42

docs(evals): md-babel executable intro, stable results

c250cd1

spomichter added the PlzReview label Aug 9, 2026

fix(evals): smoke test skips jpeg cases — ros-dev container lacks lib…

0ddf8b4

…turbojpeg

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 9, 2026

spomichter mentioned this pull request Aug 9, 2026

feat(evals): pointcloud VQA suite + PointCloud2.agent_encode (evo autoresearch, 0.136 -> 0.96) #3415

Draft

1 task

Moshiii mentioned this pull request Aug 9, 2026

feat(evals): generate simulator-grounded VQA cases from RoboCasa #3416

Draft

1 task

spomichter mentioned this pull request Aug 10, 2026

feat(evals): semantic object VQA + ImageDetections3DPC.agent_encode (evo autoresearch, 0.355 -> 0.9997) #3417

Draft

1 task

TomCC7 requested a balanced review from Copilot August 10, 2026 05:30

Copilot AI reviewed Aug 10, 2026

TomCC7 mentioned this pull request Aug 10, 2026

feat(evals): add code-as-policy evaluation runtime #3434

Draft

1 task

mustafab0 previously approved these changes Aug 11, 2026

View reviewed changes

Merge branch 'main' into feat/evals-framework

2c0d7b5

spomichter dismissed mustafab0’s stale review via 2c0d7b5 August 11, 2026 22:24

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 11, 2026

chore(blueprints): regenerate all_blueprints.py after nav evaluator m…

f12f64e

…erge

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 11, 2026

paul-nechifor reviewed Aug 12, 2026

View reviewed changes

docs(evals): move evals intro to docs/usage per review

fe02470

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Aug 12, 2026

mintlify Bot deployed to staging - docs August 12, 2026 02:14 View deployment

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Aug 12, 2026

paul-nechifor reviewed Aug 12, 2026

View reviewed changes

TomCC7 mentioned this pull request Aug 12, 2026

feat: add frozen recording agent evaluation #3378

Closed

1 task

Uh oh!

Conversation

spomichter commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

defining evals

running

first real numbers

Breaking Changes

How to Test

Contributor License Agreement

Uh oh!

codecov Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

spomichter commented Aug 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

TomCC7 commented Aug 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mintlify Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paul-nechifor Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paul-nechifor Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paul-nechifor Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

spomichter commented Aug 9, 2026 •

edited

Loading

codecov Bot commented Aug 9, 2026 •

edited

Loading

mintlify Bot commented Aug 12, 2026 •

edited

Loading

paul-nechifor Aug 12, 2026 •

edited

Loading

paul-nechifor Aug 12, 2026 •

edited

Loading

paul-nechifor Aug 12, 2026 •

edited

Loading