feat(eval): integrate LIBERO-PRO code-policy benchmark

[TomCC7]

Contribution path

• Linked PR: feat(evals): add code-as-policy evaluation runtime #3434
• Stacked on the code-as-policy evaluation runtime; merge feat(evals): add code-as-policy evaluation runtime #3434 first.

Problem

DimOS needs a LIBERO-PRO evaluation whose score is comparable with the original benchmark while exercising the same unified, realtime policy interface intended for physical robots. LIBERO and its simulator dependencies must not leak into the host Python environment, and benchmark-owned lifecycle and scoring state must remain unavailable to submitted policies.

Solution

Integrate one end-to-end LIBERO-PRO single-trial evaluation on top of #3434:

• run every debug submission and the scored policy in a fresh DimOS blueprint and a fresh rootless Podman container
• expose normal robot observations and joint targets over a policy-only gRPC/protobuf interface
• expose initialization, trial lifecycle, and native scoring through a separately authenticated privileged interface
• mount revision-pinned, checksum-verified BDDL and initial-state assets into each container
• preserve the benchmark's task selection, initial state, settling steps, 20 Hz realtime execution, horizon, reward, and binary native score
• record public camera streams and joint state to Memory2 and generate a side-by-side MP4 for every trial
• keep LIBERO environment creation, stepping, rendering, and teardown on one simulator thread so EGL offscreen frames remain valid
• add the in-repo libero-pro evaluation, strict smoke-case JSON, CLI coverage, documentation, and architecture decisions

The first integration intentionally evaluates one initialization of one task. Parallel and batched execution remain out of scope for this PR.

How to Test

Build prerequisites and run the actual evaluation:

uv sync --extra agents
npm --prefix packages/pi-code-policy-extension install
npm --prefix packages/pi-code-policy-extension run build

dimos eval run \
  dimos/benchmark/libero_pro/cases/goal-task-0-single-trial/evaluation.json \
  --output /tmp/dimos-libero-pro-smoke \
  --json --quiet

Focused automated tests:

uv run pytest \
  dimos/benchmark/libero_pro \
  dimos/benchmark/evaluation/test_policy_runtime.py \
  dimos/cli/test_eval.py -q

Validated locally with Podman: one scored trial completed 300 policy ticks, emitted the native 0.0 score, and produced a 301-frame, 20 FPS MP4 with valid frames from both cameras. The focused suite passes 21 tests.

AI assistance

Codex with GPT-5.6 assisted with design discussion, implementation, diagnosis, tests, documentation, and PR drafting. The architecture and benchmark-parity decisions were reviewed interactively by the author.

Checklist

• I have read and approved the CLA.

[codecov]

❌ 4 Tests Failed:

Tests completed	Failed	Passed	Skipped
3787	4	3783	175

View the top 3 failed test(s) by shortest run time

dimos.codebase_checks.test_no_init_files::test_no_init_files

Stack Traces | 0.021s run time

def test_no_init_files():
        dimos_dir = DIMOS_PROJECT_ROOT / "dimos"
        init_files = sorted(dimos_dir.rglob("__init__.py"))
        # The root dimos/__init__.py is allowed for the porcelain lazy import.
        init_files = [f for f in init_files if f != dimos_dir / "__init__.py"]
        if init_files:
            listing = "\n".join(f"  - {f.relative_to(dimos_dir)}" for f in init_files)
>           raise AssertionError(
                f"Found __init__.py files in dimos/:\n{listing}\n\n"
                "__init__.py files are not allowed because they lead to unnecessary "
                "extraneous imports. Everything should be imported straight from the "
                "source module."
            )
E           AssertionError: Found __init__.py files in dimos/:
E             - benchmark/libero_pro/__init__.py
E             - .../libero_pro/proto/__init__.py
E           
E           __init__.py files are not allowed because they lead to unnecessary extraneous imports. Everything should be imported straight from the source module.

dimos_dir  = PosixPath('.../dimos/dimos/dimos')
init_files = [PosixPath('.../dimos/dimos/dimos/benchmark/libero_pro/__init__.py'), PosixPath('.../dimos/dimos/dimos/.../libero_pro/proto/__init__.py')]
listing    = '  - benchmark/libero_pro/__init__.py\n  - .../libero_pro/proto/__init__.py'

dimos/codebase_checks/test_no_init_files.py:25: AssertionError

dimos.benchmark.evaluation.test_policy_runtime::test_policy_result_larger_than_pipe_buffer_does_not_deadlock

Stack Traces | 2.01s run time

def test_policy_result_larger_than_pipe_buffer_does_not_deadlock() -> None:
        context = multiprocessing.get_context("spawn")
        messages, worker_messages = context.Pipe(duplex=False)
        start_event = context.Event()
        result_sending = context.Event()
        process = context.Process(
            target=_send_large_policy_error,
            args=(worker_messages, start_event, result_sending),
            daemon=True,
        )
        process.start()
        worker_messages.close()
        execution = _PolicyExecutionProcess(process, messages, start_event)
        execution.start()
>       assert result_sending.wait(timeout=2)
E       assert False
E        +  where False = wait(timeout=2)
E        +    where wait = <Event at 0xfff99dc8aab0 unset>.wait

context    = <multiprocessing.context.SpawnContext object at 0xfffa9b6c4d40>
execution  = <dimos.benchmark.evaluation.runtime._PolicyExecutionProcess object at 0xfff99e739490>
messages   = <multiprocessing.connection.Connection object at 0xfff99dca9ac0>
process    = <SpawnProcess name='SpawnProcess-27' pid=16904 parent=3019 started daemon>
result_sending = <Event at 0xfff99dc8aab0 unset>
start_event = <Event at 0xfff99dcaaa80 set>
worker_messages = <multiprocessing.connection.Connection object at 0xfff99dca85c0>

.../benchmark/evaluation/test_policy_runtime.py:177: AssertionError

dimos.codebase_checks.test_inline_heavy_imports::test_heavy_imports_are_inline

Stack Traces | 2.26s run time

def test_heavy_imports_are_inline() -> None:
        """Fail if any file imports cv2/open3d/rerun at module level."""
        hits = find_eager_heavy_imports()
        if hits:
            listing = "\n".join(
                f"  - dimos/{f}:{line}: `{module}`"
                for f, lines in sorted(hits.items())
                for line, module in lines
            )
>           raise AssertionError(
                f"Found module-level import(s) of {'/'.join(HEAVY_MODULES)}:\n{listing}\n\n"
                "These libraries load large native extensions into every process that "
                "transitively imports the module. Import them inside the function or "
                "method that uses them; imports needed only for type annotations go "
                "under `if TYPE_CHECKING:`."
            )
E           AssertionError: Found module-level import(s) of cv2/open3d/rerun:
E             - .../benchmark/libero_pro/video.py:9: `cv2`
E           
E           These libraries load large native extensions into every process that transitively imports the module. Import them inside the function or method that uses them; imports needed only for type annotations go under `if TYPE_CHECKING:`.

hits       = {'benchmark/libero_pro/video.py': [(9, 'cv2')]}
listing    = '  - .../benchmark/libero_pro/video.py:9: `cv2`'

dimos/codebase_checks/test_inline_heavy_imports.py:92: AssertionError

dimos.robot.test_all_blueprints_generation::test_all_blueprints_is_current

Stack Traces | 3.96s run time

def test_all_blueprints_is_current() -> None:
        root = DIMOS_PROJECT_ROOT / "dimos"
        all_blueprints, all_modules = _scan_for_blueprints(root)
    
        common = set(all_blueprints.keys()) & set(all_modules.keys())
        assert not common, (
            f"Names must be unique across blueprints and modules, "
            f"but these appear in both: {sorted(common)}"
        )
    
        generated_content = _generate_all_blueprints_content(all_blueprints, all_modules)
    
        file_path = root / "robot" / "all_blueprints.py"
    
        if "CI" in os.environ:
            if not file_path.exists():
                pytest.fail(f"all_blueprints.py does not exist at {file_path}")
    
            current_content = file_path.read_text()
            if current_content != generated_content:
                diff = difflib.unified_diff(
                    current_content.splitlines(keepends=True),
                    generated_content.splitlines(keepends=True),
                    fromfile="all_blueprints.py (current)",
                    tofile="all_blueprints.py (generated)",
                )
                diff_str = "".join(diff)
>               pytest.fail(
                    f"all_blueprints.py is out of date. Run "
                    f"`pytest dimos/robot/test_all_blueprints_generation.py` locally to update.\n\n"
                    f"Diff:\n{diff_str}"
                )
E               Failed: all_blueprints.py is out of date. Run `pytest dimos/robot/test_all_blueprints_generation.py` locally to update.
E               
E               Diff:
E               --- all_blueprints.py (current)
E               +++ all_blueprints.py (generated)
E               @@ -225,6 +225,9 @@
E                    "joystick-module": "dimos.robot.unitree.b1.joystick_module.JoystickModule",
E                    "keyboard-teleop": "dimos.robot.unitree.keyboard_teleop.KeyboardTeleop",
E                    "keyboard-teleop-module": "dimos.teleop.keyboard.keyboard_teleop_module.KeyboardTeleopModule",
E               +    "libero-connection": "dimos.benchmark.libero_pro.connection.LiberoConnection",
E               +    "libero-recorder": "dimos.benchmark.libero_pro.connection.LiberoRecorder",
E               +    "libero-video-recorder": "dimos.benchmark.libero_pro.video.LiberoVideoRecorder",
E                    "local-planner": "dimos.navigation.cmu_nav.modules.local_planner.local_planner.LocalPlanner",
E                    "manipulation-module": "dimos.manipulation.manipulation_module.ManipulationModule",
E                    "map": "dimos.robot.unitree.type.map.Map",

all_blueprints = {'a1z-planner-coordinator': 'dimos.robot.manipulators.a1z.blueprints.basic:a1z_planner_coordinator', 'alfred-nav': 'di...rs.a1z.blueprints.basic:coordinator_a1z', 'coordinator-basic': 'dimos.control.blueprints.basic:coordinator_basic', ...}
all_modules = {'alfred-high-level': 'dimos.robot.diy.alfred.effector_high_level.AlfredHighLevel', 'arm-command-module': 'dimos.teleo...t_extensions.ArmTeleopModule', 'b-box-navigation-module': 'dimos.navigation.bbox_navigation.BBoxNavigationModule', ...}
common     = set()
current_content = '# Copyright 2025-2026 Dimensional Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the "License");\n# you m...sted.blueprints.cloudflare.WristCamera",\n    "zed-camera": "dimos.hardware.sensors.camera.zed.camera.ZEDCamera",\n}\n'
diff       = <generator object unified_diff at 0xff337ce8dbe0>
diff_str   = '--- all_blueprints.py (current)\n+++ all_blueprints.py (generated)\n@@ -225,6 +225,9 @@\n     "joystick-module": "dim...dule": "dimos.manipulation.manipulation_module.ManipulationModule",\n     "map": "dimos.robot.unitree.type.map.Map",\n'
file_path  = PosixPath('.../dimos/robot/all_blueprints.py')
generated_content = '# Copyright 2025-2026 Dimensional Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the "License");\n# you m...sted.blueprints.cloudflare.WristCamera",\n    "zed-camera": "dimos.hardware.sensors.camera.zed.camera.ZEDCamera",\n}\n'
root       = PosixPath('.../dimos/dimos/dimos')

dimos/robot/test_all_blueprints_generation.py:76: Failed

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}