Skip to content

fix(jail): seed writable codex/claude homes and acknowledge receipts on failed turns - #154

Merged
drewstone merged 1 commit into
mainfrom
fix/jail-seed-writable-codex-claude
Aug 12, 2026
Merged

fix(jail): seed writable codex/claude homes and acknowledge receipts on failed turns#154
drewstone merged 1 commit into
mainfrom
fix/jail-seed-writable-codex-claude

Conversation

@drewstone

Copy link
Copy Markdown
Owner

Problem

discovery-lab run strong-lead-smoke-a (profile harness codex, provider openai) died after 3 driver attempts, ~7s, zero tokens, with:

BackendTransportError: bridgeExecutor: run bridge-run-550612be... completed without cli-bridge.profile-materialization.v2 (classified transient)

The bridge traces show the real failure, three times:

codex exited 1: WARNING: proceeding, even though we could not create PATH aliases: Read-only file system (os error 30)
Reading additional input from stdin...
Error: failed to initialize in-process app-server client: Read-only file system (os error 30)

Three defects stacked:

  1. Read-only CLI home under the jail. The fs-jail bind-mounted ~/.codex (and ~/.claude + ~/.claude.json) read-only. Codex must write PATH aliases, app-server state, and session rollouts inside CODEX_HOME before it can run; claude rewrites .claude.json and persists transcripts under .claude/projects. Every jailed codex/claude lead died at spawn. Pi survived because its auth source is copy-writable.
  2. Receipt masking. The thrown backend error dropped the already-computed materialization receipt from the stream, so the runtime misclassified a deterministic failure as transient transport trouble and paid for retries. The SSE writer also renders an error terminal as a bare {error} frame, so a receipt folded into that delta never reaches the caller.
  3. Wire model id passed verbatim. The canonical wire id is harness/provider/model. Codex received -c model="openai/..." and the API rejected it as a nonexistent model; claude received --model anthropic/....

Fix

  • New seed-writable jail auth-source mode: copy a small selected file set (only) into a stable writable path under the jail root, refresh per run, retain across turns so resume finds the CLI's own session state. Codex seeds auth.json + config.toml with CODEX_HOME redirected; claude seeds credentials, settings.json, and .claude.json at the exact HOME-relative paths. Seeds are dereferenced — a dotfiles-managed symlink target is not mounted inside an fs-jail. Side benefit: the host's 130GB ~/.codex/sessions tree is no longer readable inside the jail.
  • Acknowledge the profile receipt on its own buffered event (own seq) before an error/timeout terminal, both in the route wrap (yielded terminals) and in Run.pump (thrown failures). The runtime now sees the receipt and surfaces the real upstream error.
  • Split the wire provider segment: codex gets -c model_provider=... -c model=...; claude strips anthropic/. Unknown providers pass through and fail loud.

Proof

Replayed the runtime's exact request shape against a local fs-jailed bridge built from this branch (WORKER_FS_JAIL=1, same env as the serving bridge):

  • Before: codex exited 1: ... Read-only file system (os error 30), no receipt on the stream — byte-identical to the smoke failure.
  • After (codex, codex/openai/gpt-5.6-sol): boots in the jail, model accepted, receipt event precedes the terminal; the only remaining error is the real account condition ("You've hit your usage limit ... try again at Aug 18th"), now delivered verbatim to the caller instead of the masked transport fault.
  • After (claude, claude-code/anthropic/sonnet): turn 1 "OK", turn 2 "quokka" — a two-turn resume through the jailed CLI with the receipt on both turns and real usage reported.
  • pnpm build clean; pnpm test: 802 passed, 8 skipped, 43 files.

Deploy notes (no restart performed)

  • The serving bridge resolves claude from PATH to ~/bin/claude, a wrapper that execs $HOME/.local/bin/claude — under the jail's HOME redirect that dereference breaks, and the fs-jail derives read binds from the wrapper's location, not the real install. Set CLAUDE_BIN=/home/drew/.local/bin/claude on the serving bridge when deploying this.
  • A runtime-side hardening remains in agent-runtime (src/runtime/supervise/runtime.ts, streamDurableBridgeRun): on done, the missing-receipt check throws before pendingUpstreamError, so a materialization-failure error can still be masked. Throw the pending upstream error first.

🤖 Generated with Claude Code

…on failed turns

A jailed codex or claude lead died at spawn: the jail bind-mounted the CLI
home read-only, and both CLIs must write inside their home to start.
The measured failure is 'codex exited 1: ... Read-only file system (os
error 30)' on discovery-lab run strong-lead-smoke-a. The thrown error also
dropped the computed materialization receipt from the stream. The runtime
then reported 'completed without cli-bridge.profile-materialization.v2
(classified transient)' and retried a deterministic failure three times.

Changes:
- Add the seed-writable auth-source mode. The jail copies a small selected
  file set into a stable writable path under the jail root, refreshes it on
  each run, and retains it so a later turn can resume the CLI session.
- Seed codex with auth.json and config.toml, redirect CODEX_HOME at the
  seed. This also stops exposing the host sessions tree inside the jail.
- Seed claude with credentials, settings, and .claude.json at the exact
  HOME-relative paths the jailed CLI reads.
- Dereference seed sources. A dotfiles-managed symlink target is not
  mounted inside an fs-jail, so the seed must carry real bytes.
- Acknowledge the profile receipt on its own buffered event before an
  error or timeout terminal, in the route wrap and in Run.pump. The SSE
  writer renders an error terminal as a bare error frame, so a receipt
  folded into that delta never reached the caller.
- Split the canonical wire id harness/provider/model for the codex and
  claude CLIs. Codex takes model_provider and model config separately;
  claude takes the bare model id. The qualified form was rejected by the
  API as a nonexistent model.

Verified end to end on a local fs-jailed bridge: a claude lead completes
and resumes across two turns with receipts on both, and a codex lead now
reaches the API and returns its real account error with the receipt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant