Repository navigation
fix(platform): end sandbox turns sooner and ride out spawner blips - #4543
Merged
Merged
Conversation
yannickmonney
force-pushed
the
fix/sandbox-turn-overhead
branch
2 times, most recently
from
October 8, 2026 21:12
a1d15f1 to
8896051
Compare
yannickmonney
force-pushed
the
fix/sandbox-turn-overhead
branch
2 times, most recently
from
October 9, 2026 02:40
a888e4b to
a0d642d
Compare
yannickmonney
enabled auto-merge
October 9, 2026 02:40
yannickmonney
force-pushed
the
fix/sandbox-turn-overhead
branch
from
October 9, 2026 02:40
a0d642d to
74075dc
Compare
A Claude Code turn holds its stdin open as the steering channel, so the CLI never exits after its result. The drain window waited out a 1.5 s grace after every turn-ended and then cancelled the exec (a group SIGTERM): every reply paid 1.5 s, the exit code was lost, and the CLI was killed while it could still be writing its transcript. Once the turn has ended and its background-task ledger is empty, the window now closes the held stdin (the runnerd EOF write the harness contract already described but nobody sent). The CLI finishes on its own and the drain reports its real exit; in the stand-in transport the turn ends ~20 ms after the result instead of 1.5 s. The grace and the reap stay as the fallback when the CLI ignores the EOF or the write fails. Stdin stays open while a background task runs, and close-stdin harnesses are never sent an EOF. A steer that lands after the reply now meets STDIN_CLOSED and takes the existing settled-run fallback (a fresh run carrying the comment) instead of being written into a process about to be SIGTERMed.
Every settle re-listed an empty delivery box three times 500 ms apart before accepting it as empty, as a read-after-write guard. A task or automation turn that delivered no file (most review and answer turns) therefore spent 1.5 s in the harvest before it could settle. The listing goes through runnerd in the same container, on the same filesystem the exec wrote to; once the exec has exited nothing is left to land, so its first listing is final. The hosts now tell the harvest when the turn's exec exited on its own (the drain window's `exited`; a script step that ran to completion), and only then skip the re-reads. A reaped linger and a script cut at its timeout, whose processes may still be inside their kill grace, keep the three re-reads.
A Gemini subscription turn stages the member's Google OAuth blob (with its refresh token) as .runtime/home/.gemini/oauth_creds.json, and nothing ever removed it. Every later exec in that session HOME - other tasks of the same standing agent session, member-confined runs, connector calls - could read another member's credential. The file now leaves the session with the turn: the task lane's settle and orphan reap, and the automation lane's settle (whoever won the finalize claim), question park and orphan reap delete it through the idempotent files/delete route. A turn superseded by a steer restart, or an automation run that moved on to a newer exec, leaves it to that exec, which staged its own. A Gemini turn that runs without the subscription removes a leftover before it starts, the backstop for a removal that failed. Removal is best-effort and logged; only harnesses with a staged-file subscription delivery (today Gemini) make the extra call.
A render session was created with the deployment's idle window (30 min by default). When the crawl action died between create and destroy (a killed action, a redeploy), the orphan held one of the organization's 2 render slots for that whole half hour, and the scan parked on capacity it did not have. Render creates now ask for a 3-minute idle window. A batch never sits idle that long: its exec (capped at 4 min by the crawl) keeps the session from being reaped whatever its length, and the create, staging, read-back and destroy around it are seconds apart. The spawner clamps the request to its own ceiling, so a deployment with a shorter idle window keeps it.
The spawner answers 503 session_unavailable with retry-after: 1 while a sibling create of the id is in flight, when its docker ps listing times out, or while a container is still starting; its docs say callers retry. The platform's acquire threw a generic error on it and create retried only the deploy-time 503 draining, so the turn's start failed on an answer that meant "ask again in a second". A spawner restart (ECONNREFUSED, ECONNRESET, EAI_AGAIN) failed starts the same way as SpawnerUnreachableError. Acquire and create now ask again at the spawner's retry-after for session_unavailable, and wait out a refused, reset or unresolved connection on a doubling backoff (250 ms to 5 s), both inside one 20 s budget per call; past it the call fails with the error it fails with today. Any other 503 or connection failure, and a timed-out request, still fail at once, and a 429 still parks the start on capacity.
After the run flips to running, a task agent launch (and a steer restart) read the organization's Custom instructions, the run's confinement, the task's language and the serving model's window one after another - four round trips, the window itself two more queries - before the exec could start. None of them depends on another, so they now run together. The credential env (agent secrets, the GitHub token) is still resolved after all of them answered and only for a run that is not confined, so the mint keeps its place and its gate. The confinement check after the launch flip stays a fresh read rather than reusing the subscription lane's earlier answer: it is the gate on handing out credentials, and now costs no extra latency.
The staged-credential tests built their start exec with `as const`, whose readonly argv the HarnessExec type refuses; the workspace typecheck failed on it while vitest, which does not typecheck, passed.
The automation coverage map now keeps one file per suite, so parallel changes stop appending to the same lines; this change's rows move there, after their own anchor row.
yannickmonney
force-pushed
the
fix/sandbox-turn-overhead
branch
from
October 9, 2026 02:54
74075dc to
eb77bf6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed per-turn costs on the agent turn path: every Claude reply waited 1.5 s and was then killed, every turn without a deliverable re-listed its output three times, a member's Gemini refresh token stayed in the shared session HOME, orphaned render sessions held a render slot for 30 min, and short retryable spawner answers failed turn starts.
Changes
fix(platform): end Claude turns on stdin EOF instead of a reap
A Claude Code turn holds its stdin open as the steering channel, so the
CLI never exits after its result. The drain window waited out a 1.5 s
grace after every turn-ended and then cancelled the exec (a group
SIGTERM): every reply paid 1.5 s, the exit code was lost, and the CLI
was killed while it could still be writing its transcript.
Once the turn has ended and its background-task ledger is empty, the
window now closes the held stdin (the runnerd EOF write the harness
contract already described but nobody sent). The CLI finishes on its
own and the drain reports its real exit; in the stand-in transport the
turn ends ~20 ms after the result instead of 1.5 s. The grace and the
reap stay as the fallback when the CLI ignores the EOF or the write
fails. Stdin stays open while a background task runs, and close-stdin
harnesses are never sent an EOF.
A steer that lands after the reply now meets STDIN_CLOSED and takes the
existing settled-run fallback (a fresh run carrying the comment)
instead of being written into a process about to be SIGTERMed.
perf(platform): skip harvest re-reads when the exec already exited
Every settle re-listed an empty delivery box three times 500 ms apart
before accepting it as empty, as a read-after-write guard. A task or
automation turn that delivered no file (most review and answer turns)
therefore spent 1.5 s in the harvest before it could settle.
The listing goes through runnerd in the same container, on the same
filesystem the exec wrote to; once the exec has exited nothing is left
to land, so its first listing is final. The hosts now tell the harvest
when the turn's exec exited on its own (the drain window's
exited;a script step that ran to completion), and only then skip the re-reads.
A reaped linger and a script cut at its timeout, whose processes may
still be inside their kill grace, keep the three re-reads.
fix(platform): remove a staged Gemini sign-in when the turn ends
A Gemini subscription turn stages the member's Google OAuth blob (with
its refresh token) as .runtime/home/.gemini/oauth_creds.json, and
nothing ever removed it. Every later exec in that session HOME - other
tasks of the same standing agent session, member-confined runs,
connector calls - could read another member's credential.
The file now leaves the session with the turn: the task lane's settle
and orphan reap, and the automation lane's settle (whoever won the
finalize claim), question park and orphan reap delete it through the
idempotent files/delete route. A turn superseded by a steer restart, or
an automation run that moved on to a newer exec, leaves it to that exec,
which staged its own. A Gemini turn that runs without the subscription
removes a leftover before it starts, the backstop for a removal that
failed. Removal is best-effort and logged; only harnesses with a
staged-file subscription delivery (today Gemini) make the extra call.
fix(platform): reap orphaned crawler render sessions within minutes
A render session was created with the deployment's idle window (30 min
by default). When the crawl action died between create and destroy (a
killed action, a redeploy), the orphan held one of the organization's 2
render slots for that whole half hour, and the scan parked on capacity
it did not have.
Render creates now ask for a 3-minute idle window. A batch never sits
idle that long: its exec (capped at 4 min by the crawl) keeps the
session from being reaped whatever its length, and the create, staging,
read-back and destroy around it are seconds apart. The spawner clamps
the request to its own ceiling, so a deployment with a shorter idle
window keeps it.
fix(platform): retry transient spawner answers on acquire and create
The spawner answers 503 session_unavailable with retry-after: 1 while
a sibling create of the id is in flight, when its docker ps listing
times out, or while a container is still starting; its docs say callers
retry. The platform's acquire threw a generic error on it and create
retried only the deploy-time 503 draining, so the turn's start failed
on an answer that meant "ask again in a second". A spawner restart
(ECONNREFUSED, ECONNRESET, EAI_AGAIN) failed starts the same way as
SpawnerUnreachableError.
Acquire and create now ask again at the spawner's retry-after for
session_unavailable, and wait out a refused, reset or unresolved
connection on a doubling backoff (250 ms to 5 s), both inside one 20 s
budget per call; past it the call fails with the error it fails with
today. Any other 503 or connection failure, and a timed-out request,
still fail at once, and a 429 still parks the start on capacity.
perf(platform): read a task turn's launch context concurrently
After the run flips to running, a task agent launch (and a steer
restart) read the organization's Custom instructions, the run's
confinement, the task's language and the serving model's window one
after another - four round trips, the window itself two more queries -
before the exec could start.
None of them depends on another, so they now run together. The
credential env (agent secrets, the GitHub token) is still resolved
after all of them answered and only for a run that is not confined, so
the mint keeps its place and its gate. The confinement check after the
launch flip stays a fresh read rather than reusing the subscription
lane's earlier answer: it is the gate on handing out credentials, and
now costs no extra latency.
test(platform): type the Gemini start fixture as a harness exec
The staged-credential tests built their start exec with
as const, whose readonly argv the HarnessExec type refuses; the workspace typecheck failed on it while vitest, which does not typecheck, passed.Verification
@tale/platformtypecheck and lint; a follow-up commit fixed a typecheck-only fixture issue.Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).