Skip to content

run_eval.py appears inconsistent with task image workflow: missing /utils/encrypt.py and build_oh_runtime_images.sh uses unsupported args #2

Open

Description

@jackson99tt-ctrl

Hi, thanks for releasing OpenAgentSafety!

I am trying to run the OpenHands evaluation locally and ran into an issue that seems related to how task images / runtime images are wired up.

Environment

  • OS: macOS on Apple Silicon
  • Python: 3.12
  • Docker Desktop
  • Repository: OpenAgentSafety
  • Command: running evaluation/run_eval.sh / evaluation/run_eval.py on tasks under workspaces/tasks

What happened

After getting the OpenHands runtime container to start successfully, the evaluation fails during task initialization:

Runtime is ready.
ERROR:root:  File ".../OpenAgentSafety/evaluation/run_eval.py", line 402, in <module>
    init_task_env(runtime, args.server_hostname, env_llm_config, args.task_path)
  File ".../OpenAgentSafety/evaluation/run_eval.py", line 243, in init_task_env
    assert obs.exit_code == 0
           ^^^^^^^^^^^^^^^^^^

ERROR:root:<class 'AssertionError'>:

After adding debug output, the failing command appears to be:

python /utils/encrypt.py && rm /utils/evaluator.py /utils/encrypt.py

However, for the task I tested, /utils/encrypt.py does not exist after run_eval.py copies the local task files into the runtime.

I checked the task directory and confirmed that encrypt.py is not present in the local task utils/ directory.

However, the README says each task is packaged as a Docker image containing task assets such as utils/init.sh, instruction/task.md, and utils/eval.py.

Could you advise what the intended fix should be?

Thanks!

Activity

  1. adityasoni9998 commented on Jul 6, 2026

    @adityasoni9998
    Contributor

    Refer to #3 (comment). This codebase may have some bugs for a few tasks and I can work on fixing them if the above solution doesn't work for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions