Skip to content

fix(platform): end sandbox turns sooner and ride out spawner blips - #4543

Merged
yannickmonney merged 8 commits into
mainfrom
fix/sandbox-turn-overhead
Oct 9, 2026
Merged

yannickmonney merged 8 commits into
mainfrom
fix/sandbox-turn-overhead

Conversation

@yannickmonney

Copy link
Copy Markdown
Contributor

Fixed per-turn costs on the agent turn path: every Claude reply waited 1.5 s and was then killed, every turn without a deliverable re-listed its output three times, a member's Gemini refresh token stayed in the shared session HOME, orphaned render sessions held a render slot for 30 min, and short retryable spawner answers failed turn starts.

Changes

fix(platform): end Claude turns on stdin EOF instead of a reap

A Claude Code turn holds its stdin open as the steering channel, so the
CLI never exits after its result. The drain window waited out a 1.5 s
grace after every turn-ended and then cancelled the exec (a group
SIGTERM): every reply paid 1.5 s, the exit code was lost, and the CLI
was killed while it could still be writing its transcript.

Once the turn has ended and its background-task ledger is empty, the
window now closes the held stdin (the runnerd EOF write the harness
contract already described but nobody sent). The CLI finishes on its
own and the drain reports its real exit; in the stand-in transport the
turn ends ~20 ms after the result instead of 1.5 s. The grace and the
reap stay as the fallback when the CLI ignores the EOF or the write
fails. Stdin stays open while a background task runs, and close-stdin
harnesses are never sent an EOF.

A steer that lands after the reply now meets STDIN_CLOSED and takes the
existing settled-run fallback (a fresh run carrying the comment)
instead of being written into a process about to be SIGTERMed.

perf(platform): skip harvest re-reads when the exec already exited

Every settle re-listed an empty delivery box three times 500 ms apart
before accepting it as empty, as a read-after-write guard. A task or
automation turn that delivered no file (most review and answer turns)
therefore spent 1.5 s in the harvest before it could settle.

The listing goes through runnerd in the same container, on the same
filesystem the exec wrote to; once the exec has exited nothing is left
to land, so its first listing is final. The hosts now tell the harvest
when the turn's exec exited on its own (the drain window's exited;
a script step that ran to completion), and only then skip the re-reads.
A reaped linger and a script cut at its timeout, whose processes may
still be inside their kill grace, keep the three re-reads.

fix(platform): remove a staged Gemini sign-in when the turn ends

A Gemini subscription turn stages the member's Google OAuth blob (with
its refresh token) as .runtime/home/.gemini/oauth_creds.json, and
nothing ever removed it. Every later exec in that session HOME - other
tasks of the same standing agent session, member-confined runs,
connector calls - could read another member's credential.

The file now leaves the session with the turn: the task lane's settle
and orphan reap, and the automation lane's settle (whoever won the
finalize claim), question park and orphan reap delete it through the
idempotent files/delete route. A turn superseded by a steer restart, or
an automation run that moved on to a newer exec, leaves it to that exec,
which staged its own. A Gemini turn that runs without the subscription
removes a leftover before it starts, the backstop for a removal that
failed. Removal is best-effort and logged; only harnesses with a
staged-file subscription delivery (today Gemini) make the extra call.

fix(platform): reap orphaned crawler render sessions within minutes

A render session was created with the deployment's idle window (30 min
by default). When the crawl action died between create and destroy (a
killed action, a redeploy), the orphan held one of the organization's 2
render slots for that whole half hour, and the scan parked on capacity
it did not have.

Render creates now ask for a 3-minute idle window. A batch never sits
idle that long: its exec (capped at 4 min by the crawl) keeps the
session from being reaped whatever its length, and the create, staging,
read-back and destroy around it are seconds apart. The spawner clamps
the request to its own ceiling, so a deployment with a shorter idle
window keeps it.

fix(platform): retry transient spawner answers on acquire and create

The spawner answers 503 session_unavailable with retry-after: 1 while
a sibling create of the id is in flight, when its docker ps listing
times out, or while a container is still starting; its docs say callers
retry. The platform's acquire threw a generic error on it and create
retried only the deploy-time 503 draining, so the turn's start failed
on an answer that meant "ask again in a second". A spawner restart
(ECONNREFUSED, ECONNRESET, EAI_AGAIN) failed starts the same way as
SpawnerUnreachableError.

Acquire and create now ask again at the spawner's retry-after for
session_unavailable, and wait out a refused, reset or unresolved
connection on a doubling backoff (250 ms to 5 s), both inside one 20 s
budget per call; past it the call fails with the error it fails with
today. Any other 503 or connection failure, and a timed-out request,
still fail at once, and a 429 still parks the start on capacity.

perf(platform): read a task turn's launch context concurrently

After the run flips to running, a task agent launch (and a steer
restart) read the organization's Custom instructions, the run's
confinement, the task's language and the serving model's window one
after another - four round trips, the window itself two more queries -
before the exec could start.

None of them depends on another, so they now run together. The
credential env (agent secrets, the GitHub token) is still resolved
after all of them answered and only for a run that is not confined, so
the mint keeps its place and its gate. The confinement check after the
launch flip stays a fresh read rather than reusing the subscription
lane's earlier answer: it is the gate on handing out credentials, and
now costs no extra latency.

test(platform): type the Gemini start fixture as a harness exec

The staged-credential tests built their start exec with as const, whose readonly argv the HarnessExec type refuses; the workspace typecheck failed on it while vitest, which does not typecheck, passed.

Verification

  • Builder ran the touched vitest suites (external_turn_shared, session_client, session_exec, render_fetch, agent_host, agent_run_host, script_host), @tale/platform typecheck and lint; a follow-up commit fixed a typecheck-only fixture issue.
  • An independent adversarial review is still pending for this branch; I'll add its outcome here.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).

@yannickmonney
yannickmonney force-pushed the fix/sandbox-turn-overhead branch 2 times, most recently from a1d15f1 to 8896051 Compare October 8, 2026 21:12
@yannickmonney
yannickmonney force-pushed the fix/sandbox-turn-overhead branch 2 times, most recently from a888e4b to a0d642d Compare October 9, 2026 02:40
@yannickmonney
yannickmonney force-pushed the fix/sandbox-turn-overhead branch from a0d642d to 74075dc Compare October 9, 2026 02:40
A Claude Code turn holds its stdin open as the steering channel, so the
CLI never exits after its result. The drain window waited out a 1.5 s
grace after every turn-ended and then cancelled the exec (a group
SIGTERM): every reply paid 1.5 s, the exit code was lost, and the CLI
was killed while it could still be writing its transcript.

Once the turn has ended and its background-task ledger is empty, the
window now closes the held stdin (the runnerd EOF write the harness
contract already described but nobody sent). The CLI finishes on its
own and the drain reports its real exit; in the stand-in transport the
turn ends ~20 ms after the result instead of 1.5 s. The grace and the
reap stay as the fallback when the CLI ignores the EOF or the write
fails. Stdin stays open while a background task runs, and close-stdin
harnesses are never sent an EOF.

A steer that lands after the reply now meets STDIN_CLOSED and takes the
existing settled-run fallback (a fresh run carrying the comment)
instead of being written into a process about to be SIGTERMed.
Every settle re-listed an empty delivery box three times 500 ms apart
before accepting it as empty, as a read-after-write guard. A task or
automation turn that delivered no file (most review and answer turns)
therefore spent 1.5 s in the harvest before it could settle.

The listing goes through runnerd in the same container, on the same
filesystem the exec wrote to; once the exec has exited nothing is left
to land, so its first listing is final. The hosts now tell the harvest
when the turn's exec exited on its own (the drain window's `exited`;
a script step that ran to completion), and only then skip the re-reads.
A reaped linger and a script cut at its timeout, whose processes may
still be inside their kill grace, keep the three re-reads.
A Gemini subscription turn stages the member's Google OAuth blob (with
its refresh token) as .runtime/home/.gemini/oauth_creds.json, and
nothing ever removed it. Every later exec in that session HOME - other
tasks of the same standing agent session, member-confined runs,
connector calls - could read another member's credential.

The file now leaves the session with the turn: the task lane's settle
and orphan reap, and the automation lane's settle (whoever won the
finalize claim), question park and orphan reap delete it through the
idempotent files/delete route. A turn superseded by a steer restart, or
an automation run that moved on to a newer exec, leaves it to that exec,
which staged its own. A Gemini turn that runs without the subscription
removes a leftover before it starts, the backstop for a removal that
failed. Removal is best-effort and logged; only harnesses with a
staged-file subscription delivery (today Gemini) make the extra call.
A render session was created with the deployment's idle window (30 min
by default). When the crawl action died between create and destroy (a
killed action, a redeploy), the orphan held one of the organization's 2
render slots for that whole half hour, and the scan parked on capacity
it did not have.

Render creates now ask for a 3-minute idle window. A batch never sits
idle that long: its exec (capped at 4 min by the crawl) keeps the
session from being reaped whatever its length, and the create, staging,
read-back and destroy around it are seconds apart. The spawner clamps
the request to its own ceiling, so a deployment with a shorter idle
window keeps it.
The spawner answers 503 session_unavailable with retry-after: 1 while
a sibling create of the id is in flight, when its docker ps listing
times out, or while a container is still starting; its docs say callers
retry. The platform's acquire threw a generic error on it and create
retried only the deploy-time 503 draining, so the turn's start failed
on an answer that meant "ask again in a second". A spawner restart
(ECONNREFUSED, ECONNRESET, EAI_AGAIN) failed starts the same way as
SpawnerUnreachableError.

Acquire and create now ask again at the spawner's retry-after for
session_unavailable, and wait out a refused, reset or unresolved
connection on a doubling backoff (250 ms to 5 s), both inside one 20 s
budget per call; past it the call fails with the error it fails with
today. Any other 503 or connection failure, and a timed-out request,
still fail at once, and a 429 still parks the start on capacity.
After the run flips to running, a task agent launch (and a steer
restart) read the organization's Custom instructions, the run's
confinement, the task's language and the serving model's window one
after another - four round trips, the window itself two more queries -
before the exec could start.

None of them depends on another, so they now run together. The
credential env (agent secrets, the GitHub token) is still resolved
after all of them answered and only for a run that is not confined, so
the mint keeps its place and its gate. The confinement check after the
launch flip stays a fresh read rather than reusing the subscription
lane's earlier answer: it is the gate on handing out credentials, and
now costs no extra latency.
The staged-credential tests built their start exec with `as const`, whose readonly argv the HarnessExec type refuses; the workspace typecheck failed on it while vitest, which does not typecheck, passed.
The automation coverage map now keeps one file per suite, so parallel changes stop appending to the same lines; this change's rows move there, after their own anchor row.
@yannickmonney
yannickmonney force-pushed the fix/sandbox-turn-overhead branch from 74075dc to eb77bf6 Compare October 9, 2026 02:54
@yannickmonney
yannickmonney merged commit 56e8ff3 into main Oct 9, 2026
@yannickmonney
yannickmonney deleted the fix/sandbox-turn-overhead branch October 9, 2026 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant