[danshapiro]

Problem

The provider parity suite (test/integration/server/provider-parity-contract.test.ts) was failing on main. Managed terminal sessions timed out while starting ("timed out waiting for claude/amplifier replacement incarnation"). The supervisor's deliver_execution_grant gave up after 10 seconds, and Claude recovery got stuck as CAPABILITY_PENDING.

Root causes (with evidence)

1. The session host really was slow to start a provider. New per-stage timing (host.grant_timing) showed that nearly all of a 10.5–12.9 s grant was spent in prepare_provider_state. That stage flushed each of the selected plugin's roughly 4,200 files to disk one at a time, at about 2 ms per file. Run one at a time, the flushes took 13.4 s; run 16 at a time, they took 1.4 s.
2. Terminal grants had the wrong time limit. A terminal grant got the 10 s limit meant for status and stop commands, even though the host runs steps with longer limits of their own while it handles a grant. In the failing runs the host finished at 10.6–12.9 s, but the supervisor had already torn the container down.
3. The fixed 60 s grant limit still could not cover the host's worst case. A terminal Codex start retries its 45 s app-server start up to five times (about 251 s). An exact resume first runs a 25 s provider-owned config refresh, which can add up to 32 s of recovery. So the supervisor could still give up on a dispatch the host was still working on.
4. The Claude recovery callback had time limits that did not fit together. Its callback to the web server waited 35 s, and the scope it hands out expired after 30 s. But that callback runs two Claude source checks back to back, and each is allowed up to 35 s on its own. In an earlier failing run, the second check failed 32.3 s after the recovery was scheduled, inside a callback that was allowed only 35 s.
5. The test Docker proxy blocked Claude recovery (a test bug). It refused the supervisor's check on the recovery scratch volume, so Claude recovery was blocked as CAPABILITY_PENDING and nothing logged why.

Fix

• Provider file flushes now run in parallel (PROVIDER_PROJECTION_SYNC_WORKERS), and every grant records how long each stage took.
• Terminal grants have their own 60 s limit, FRESHELL_RUNTIME_TERMINAL_DISPATCH_TIMEOUT_MS.
• Grants now carry their own time budget (dispatch_budget_ms). The budget is the supervisor's wait minus 5 s, and never less than half the wait (55 s by default). The host runs each awaited step inside that budget and stops it at the deadline. Once the budget is spent, it does not start the provider, so the grant can still be retried. It answers with a typed HostUnreachable error that names the step it was on. host.grant_timing now also records budgetMs and budgetExhaustedBefore. As a result, the supervisor never gives up on a dispatch the host is still working on.
• A Claude recovery callback now waits 105 s (35 s for the callback itself plus two 35 s source checks). Other callbacks keep 35 s. The recovery scope now lasts exactly as long as the callback wait it belongs to.
• The supervisor now logs why a Claude source check was refused (supervisor.claude_source_observation.failed), and the test Docker proxy now allows recovery scratch-volume checks under the same strict rules as the launch fixture.
• The budget tests now give each command type a different limit, so mixing up two limits makes a test fail.
• One session-host test failed the pre-push gate with "Text file busy". Another test thread forked while a just-written fixture was still open for writing, so the fixture could not be run yet. Tests that run freshly written fixtures now retry only that error, for up to 500 ms, the same way the platform crate's fixtures already do. A new test holds the file open for writing to prove the retry works.
• docs/development/managed-runtime.md documents each limit and where it comes from. It also notes that a stop issued while a session is starting waits until the start finishes or fails, up to the grant limit plus the callback limit during a recovery.

Tests run

• cargo test -p freshell-runtime-protocol -p freshell-session-host -p freshell-supervisor: all passed (protocol 66; session host 164 + 4 + 29 + 5; supervisor 195).
• Pre-push gate on the pushed head 6af75b199 (cargo test --workspace --exclude freshell-tauri --locked, clippy, fmt, typecheck): passed.
• New tests, each first seen failing for the right reason:
  • Host: a grant whose budget is already spent is refused before the provider starts, and is not consumed. A slow awaited step is stopped at the deadline. An exact-resume grant with a budget records budgetMs.
  • Supervisor: a grant tells the host its budget (wait minus allowance). Each command type is held to its own limit; swapping the terminal and fresh-agent limits fails two tests. A Claude recovery callback can outlast an ordinary one and its scope is still valid; other callbacks keep the 35 s limit.
• cargo clippy --tests -D warnings and cargo fmt --check on the changed crates: clean. cargo check -p freshell-server --tests: clean.
• pnpm run test:vitest run test/unit/tooling/testing/runtime-test-broker.test.ts --config config/vitest/vitest.config.ts: 61/61 passed.
• pnpm run test:vitest run test/integration/server/provider-parity-contract.test.ts --config config/vitest/vitest.config.ts, run 6 times locally:

Run	Head	Result	Time
1	dd309cb40	7/7 passed	726 s
2	dd309cb40	7/7 passed	337 s
3	dd309cb40	failed (freshclaude create; see below)	98 s
4	4d6cd8f8f (after merging main)	failed (freshclaude create; see below)	469 s
5	4d6cd8f8f	7/7 passed	501 s
6	4d6cd8f8f	7/7 passed	304 s

No Playwright spec was changed; the browser specs that use the test Docker proxy are local-only.

Remaining problem (not fixed here)

Runs 3 and 4 failed for a different reason from the ones above: creating a new freshclaude session failed with FRESH_AGENT_CREATE_FAILED (marked retryable). The supervisor logged Claude candidate reader timed out in one run and Claude candidate scratch cleanup failed in the other.

Before that failure, a single Docker POST /containers/create for a short-lived helper container (freshell-mountpoints-* or freshell-claude-source-*) took 18–31 s. That used up the Claude source reader's own 25 s limit. In a reproduction with timing added to the proxy, one create took 20,010 ms while other Docker calls kept answering in milliseconds and the test process was not stalled. A Docker stack dump taken during a slow create showed dockerd inside createContainerVolumesOS, mounting and unmounting the new container's root filesystem, while containerd was idle. On this shared host, the median time for these helper creates ranged from 52 ms to 501 ms depending on the run, with occasional spikes over 5 s. Session-host container creates never went above 0.4 s.

This is not caused by the grant or callback limits fixed here, and the exact cause inside Docker is still unknown. Fixing it needs a decision: either give the Claude source reader a budget that can absorb slow Docker creates, or retry the retryable create and recovery failures.

🤖 Generated with Claude Code