Skip to content

fix(harness): docker-claude arms stop showing dokimi's names to the agent - #351

Open
napetrov wants to merge 1 commit into
intel:mainfrom
napetrov:fidelity-arms
Open

napetrov wants to merge 1 commit into
intel:mainfrom
napetrov:fidelity-arms

Conversation

@napetrov

@napetrov napetrov commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Summary

docker-claude agent sees (before)              after
  cwd      /tmp/dokimi-run-xxxx/work    ->    /tmp/tmpxxxx/work
  plugin   --plugin-dir /opt/abskill    ->    --plugin-dir /opt/claude-plugins
  prompt   "Solve this coding task. Edit files in the current
            directory only. When done, ..."  ->  <task instruction>
           (treatment notice still prepended on the skill arm)

Skills still load through --plugin-dir: a skill in $CLAUDE_CONFIG_DIR/skills
is not offered under the launch's --setting-sources "" (measured, see the
comment in docker_solver.py), so moving skills to ~/.claude/skills is not done here.

Evidence

  • Before: dokimi-run- / abskill / Solve this coding task. appear in the argv and cwd of the docker-claude arms.
    After: tests/test_payload_audit.py::test_docker_claude_arms_send_no_harness_artifacts runs audit_payload over the argv (both notice settings) and the workspace cwd with the default markers, and finds nothing. The prompt assertions in test_container_agent.py and test_agent_sandbox.py now check the raw instruction.
  • Full suite: 12033 passed, 28 skipped, 6 xfailed (pytest -n 6).
  • scripts/check_growth.py --base intel/main: docker_solver.py 899 -> 897, baseline ratcheted, no growth.

Merge Danger

Door: two-way. The prompt and paths revert with a single commit; no data or config is migrated.

Blast Radius: docker-claude and docker-claude-host arms only. Runs recorded before this change used the old prompt text and paths, so their prompt and prompt.txt are not comparable with new runs. The changelog fragment says so.

🤖 Generated with Claude Code

Copilot AI balanced review requested due to automatic review settings November 1, 2026 18:36
@napetrov
napetrov requested a review from a team as a code owner November 1, 2026 18:36
…gent

The agent's cwd was a `dokimi-run-*` temp dir, its skill plugin sat at
`/opt/abskill`, and the prompt opened with a "Solve this coding task…"
wrapper. None of that is in the stock harness. The workspace is now a
plain mkdtemp, the plugin is mounted at `/opt/claude-plugins`, and the
prompt is the task instruction behind the treatment notice only.

Skills still load through `--plugin-dir`: a skill in
`$CLAUDE_CONFIG_DIR/skills` is not offered under `--setting-sources ""`,
so the `~/.claude/skills` location is not used.

A test runs payload_audit over the arms' argv and cwd. The docker_solver
line baseline drops from 899 to 897.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@exai-bot

exai-bot commented Nov 1, 2026

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 10 selected item(s).

Review details
  • Commit reviewed: 2592e75 against merge base 93a5062
  • Coverage: 10 files in 1 groups
  • Engine: OpenCodeReview v1.13.3, us.openai.gpt-6.1-sol
  • Usage: 80k tokens (45k from cache), 12s
  • Tool calls: 10 (8 file_read, 2 code_search)
  • Harness fidelity and ARM leak fixes, documentation, and regression tests: 10 files

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The argv audit test omits the executable path, missing a remaining dokimi marker and overstating its whole-command audit guarantee.

1 open issue
What changed in this PR

1 new finding.

This PR reduces harness-identifying text in dokimi’s Docker Claude runs, helping evaluations better reflect ordinary agent use.

Changes:

  • Uses generic temporary workspace names and renames the container plugin directory.
  • Removes the harness-written prompt wrapper, retaining the task instruction and optional treatment notice.
  • Updates regression tests, documentation, and the growth baseline.
File Description
dokimi/​tests/​test_payload_audit.py Adds argv and workspace marker checks.
dokimi/​tests/​test_container_agent.py Updates plugin-path and prompt assertions.
dokimi/​tests/​test_cleanup_timeouts.py Updates the plugin-removal timeout matcher.
dokimi/​tests/​test_agent_sandbox.py Adjusts treatment-notice assertions.
dokimi/​scripts/​growth_baseline.json Lowers the solver’s file-size baseline.
dokimi/​dokimi/​harnesses/​trigger_probe.py Updates the workspace-name comment.
dokimi/​dokimi/​harnesses/​docker_solver.py Renames the plugin directory and removes prompt framing.
dokimi/​dokimi/​harnesses/​agent_sandbox.py Uses default temporary directory names.
dokimi/​docs/​coding-harnesses.md Documents the updated workspace and plugin paths.
dokimi/​changelog.d/​20261101_120000_napetrov_fidelity-arm-leaks.md Records changes and prompt-comparability implications.

Review effort: Balanced

Comment on lines +143 to +145
argv = h._claude_args(task, treatment, _CONTAINER_PLUGIN, "haiku", out)
assert _CONTAINER_PLUGIN in argv and "abskill" not in " ".join(argv)
assert audit_payload(argv, markers, allowed=allowed) == []

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 2592e75. The prompt, plugin-path and workspace-name changes are right, and the updated assertions pin them. One thing is worth fixing before merge: the new audit test and the changelog claim more than they check.

What I ran, on a Linux review host at 2592e75:

  • make ci-local --with-docker: 35 passed, 0 failed, 7 skipped; 12002 tests passed; changed-line coverage 100%.
  • Mutation run on the new code with the four touched test files: restoring the dokimi-run- prefix, /opt/abskill, the "Solve this coding task" wrapper, an added Task: prefix or suffix, and a dropped or always-on notice are all caught. Two mutants pass: renaming the -host arm's plugin directory to dokimi-abskill, and mkdtemp(prefix="project-") (harmless).

Checked, no issues

  • Prompt: TREATMENT_NOTICE ends in a blank line, so notice plus instruction keeps its separator. Both arms are pinned with prompt == task.instruction.
  • Stale names: no remaining /opt/abskill, no code that sweeps dokimi-run-* temp dirs or strips the old wrapper, and nothing reads prompt.txt in the package.
  • Changelog fragment: ### Changed, licence header, non-comparability stated.

A smaller point on wording: the changelog says the docker-claude arms' cwd changed. The temp tree applies to the -host variants only. The in-container arm runs with docker exec -w /app and never had a dokimi-run-* cwd.

h = DockerSolverHarness(harness_id="docker-claude", mode="claude", skill_notice=notice)
argv = h._claude_args(task, treatment, _CONTAINER_PLUGIN, "haiku", out)
assert _CONTAINER_PLUGIN in argv and "abskill" not in " ".join(argv)
assert audit_payload(argv, markers, allowed=allowed) == []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Copilot here; the repro is below.

The audit covers the arguments after the executable, not the command the agent runs, and the executable still says dokimi.

_claude_args() returns the argv after the executable. _solve_claude_in_container() runs [AGENT_BIN, *args] (docker_solver.py:448), and AGENT_BIN is /opt/dokimi-agent/claude (container_agent.py:80). Audited with the same default markers:

audit_payload(["/opt/dokimi-agent/claude", "-p", "x"])  ->  [Leak(marker="dokimi", path="$[0]")]

The process's argv[0] and the /opt/dokimi-agent/ directory are visible inside the container, so the PR's claim that the arms carry no marker (description and changelog: "asserts the arms' argv and cwd carry no marker") does not hold, and the test can't say so either way.

Suggestion: either audit [AGENT_BIN, *argv] and rename the in-container directory (AGENT_BIN and FILES_DIR) to something neutral such as /opt/claude-bin, or narrow the test comment and the changelog sentence to "the arguments after the executable" and say the /opt/dokimi-agent path is still visible. Also cover the -host arm: its plugin dir is workspace.root / "plugin", and renaming it to dokimi-abskill passes the whole suite.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants