[danshapiro]

Problem

The managed-provider parity suite (test/integration/server/provider-parity-contract.test.ts) still failed about 2 runs in 6 after #861. Creating a freshclaude session failed because one Docker create for a short-lived helper container (freshell-claude-source-*, freshell-mountpoints-*) took 18–48 s and used up the Claude source reader's single 25 s attempt. A second problem: when such a slow create finished after the reader gave up, the helper container (and, on the candidate path, its scratch volume) was left behind.

Cause

Docker mounts and then unmounts a new container's filesystem while it creates it. On this kernel the unmount flushes everything not yet written to the disk under Docker's data folder. During parity runs up to 8 GB sat unwritten (release builds, other agents), so whichever create came first after a burst of writes waited for the flush: 0.1 s on an idle disk, 4.5 s with 1 GB unwritten, 16.7 s with 3 GB, up to 48 s in the runs. The create right after it is fast again, which is why managed-session creates stayed quick. The same flush can also stall a container's start, the reader's own writes, and Docker's report that the reader exited.

The leak: on a timeout the create request was dropped, and cleanup looked the helper up by name before Docker had made it.

Fix (all in the supervisor's helper lifecycle, crates/freshell-supervisor/src/backend/helper_lifecycle.rs)

• No more leaks. Each helper create runs as its own task, so it keeps going when the caller stops waiting. The supervisor owns a helper until Docker confirms it is gone: cleanup waits for Docker's answer, then removes by exact name. Anything still unanswered, or whose removal ran out of time or failed, goes to a background cleanup task that keeps removing it (ownership is re-checked every try) for up to 120 s. This covers the source reader, the candidate reader and its scratch volume, and the mountpoint helper on both the candidate and launch paths.
• Bounded retry of the reader. A reader attempt that timed out, or whose create or start Docker refused with a server error or never answered, is retried. Client errors, ownership or topology changes, a reader that exits with an error, and unconfirmed cleanups stay final. Each attempt keeps the 25 s limit; the whole read gets up to 90 s (enough to wait out a 48 s stall and then do one full retry). A retry starts only after the previous attempt's helpers are gone, so retries never pile creates onto a stalled Docker. Waits between attempts are 1 s and 2 s.
• Recovery limits unchanged from Managed sessions: fix slow session-host startup and mismatched time limits #861. A Claude recovery's callback still waits 105 s. A read inside a recovery now ends, cleanup included, 5 s before the recovery's own time is up, so a retry only uses what is left of the callback and never outlives it.
• Logs (structured JSON lines in the supervisor lifecycle log): supervisor.helper.retry (warn), supervisor.helper.retry_exhausted (error), supervisor.helper.recovered (info), supervisor.helper.create_slow (warn, any create Docker takes 5 s or more to answer), and supervisor.helper.cleanup.deferred / .removed / .unconfirmed. Retry events name the step the deadline hit (create, start, read, exit, ...), so a Docker stall can be told apart from a reader that hung.
• Test harness. FRESHELL_TEST_BROKER_DELAY_HELPER_CREATE=<prefix>:<ms>[:<count>] holds helper creates in the test Docker proxy to reproduce a stall. Harness cleanup waits for every forwarded request to finish, removes helpers Docker created late, and now also removes any Claude candidate scratch volume the run left behind (through the proxy's ownership check).

docs/development/managed-runtime.md describes the helper lifecycle, the retry and the budgets.

Tests

• Unit tests (each seen failing first): refused create retried; late create removed; late scratch-volume create removed; a removal that runs past its window is finished in the background (candidate scratch volume and launch mountpoint helper); a recovery read ends inside its recovery scope; a read ends by its caller's deadline; retry events name the step; slow creates logged; proxy hold and wait-for-answers; harness removes leftover candidate volumes and refuses a foreign one.
• Regression guards, checked by breaking the code: client errors not retried, exhaustion after three attempts, the launch hand-off, the scratch-volume wait.
• The retry tests use second-long windows; four parallel copies pass at a load of about 440 on 96 cores.
• cargo test -p freshell-supervisor: 205 passed. The push gate's freshell-server run: 1273 passed. Clippy -D warnings, fmt and typecheck clean.

Parity runs (provider-parity-contract.test.ts, local Docker)

On the final head, with the host's 1-minute load at 8–21: 4 of 4 passed (312 s, 309 s, 310 s, 649 s). A run with a 30 s stall injected into the first source-reader create of each rig also passed (464 s): both stalls were retried (step: "create") and the reads recovered at 50 s and 33 s.

Earlier runs while other agents' stress tests held the host's load at 250–490 (on 96 cores) failed 6 of 7 times. None left a helper container or volume behind, and the logs show where each one went:

• Twice the reader's retry recovered, but the whole freshclaude create then ran past the test's own 90 s wait.
• Twice a removal took longer than the flat 5 s cleanup window and made the read fail although time remained. The last fix commit lets a removal use the time the retry already reserved.
• Once the launch-path mountpoint helper (which has no retry) took 29 s to create.
• Once 6.6 GB of unwritten data appeared on the disk within 12 s and took about 70 s to drain, longer than the stall the 90 s budget is sized for.

After merging main (#866), two more runs failed while other work on the host wrote 6–7 GB bursts to the disk: source-reader and mountpoint creates took 65–71 s, and once the OpenCode input publisher's cleanup (not changed here) timed out. Every late helper was still removed. In the four green runs above, the disk never held more than 3 GB unwritten.

Release builds: the parity rig requires the debug session-host build for its fresh-agent fixtures (the server is already built release), so these runs use the harness's normal builds.

Not in this PR

• The launch-path mountpoint helper (30 s, no retry) has the same stall exposure; it can reuse this retry with its own policy.
• The history reader, native-auth reader and input publisher have the same create-then-give-up leak; they can reuse the same create/cleanup pieces.
• A supervisor that crashes mid-read cannot run its own background cleanup (the harness now removes what a test run leaves).
• First-creation (not recovery) Claude preparation has no overall limit at the web server; each read is bounded at 95 s.
• The input publisher's cleanup has the same stall exposure (seen once in a run after the merge).
• The parity test waits 90 s for a freshclaude create, which is shorter than one recovered stall plus a normal create on a loaded host.

🤖 Generated with Claude Code