Repository navigation
perf(sandbox): drop checkpoint fsyncs and resident harness wrappers - #4546
Merged
Merged
Conversation
yannickmonney
force-pushed
the
perf/sandbox-runtime-turn-io
branch
4 times, most recently
from
October 9, 2026 02:40
db7c508 to
f643365
Compare
yannickmonney
enabled auto-merge
October 9, 2026 02:40
ExecReplay.saveCheckpoint synced the temporary checkpoint file and then the spool directory inside the five-second replay I/O deadline; a slow or failed sync marked the replay REPLAY_UNAVAILABLE and the exec was ended. The platform saves a checkpoint of up to ~300 KB every five seconds per streaming turn, so each active turn cost ~0.4 fsync/s, each one flushing the filesystem journal on a host shared by every session. The syncs bought no durability: the only reader of a checkpoint is the runnerd process that wrote it, each ExecReplay spools into a fresh mkdtemp directory no later process reads, and the entrypoint wipes /agent/.runtime/tmp at every container start. Keep the write to a temporary file and the atomic rename, which is what a reader in the same process needs, and drop both syncs. The tests that injected a failed directory sync and a stalled file sync now inject at the steps that remain (a failed prune after the rename, a stalled write of the temporary file) and keep the same guarantees; a new test holds that a checkpoint syncs no handle and leaves no temporary file.
tale-gemini-run and tale-pi-run started their CLI as a child and waited on it for the whole turn, only to delete a few KB of staged files after it exited. That kept one Python process per Gemini or Pi turn resident (about 19 MB RSS measured for the idle wrapper) beside the CLI. Both wrappers now replace themselves with the CLI (os.execvpe, same pid), so the CLI is the exec's process: its exit status is the exec's, and a process-group SIGTERM ends it by that signal, which runnerd reports as 143, the status the wrapper used to exit with. The prompt reaches the CLI on stdin from an anonymous temporary file (any size; a pipe would block past its buffer with no reader yet), the CLI runs in the exec's workdir, and SIGPIPE and SIGXFSZ get their defaults back as a subprocess child would. A wrapper cancelled before its CLI starts, or whose CLI cannot be started, still removes what it staged. Nothing is left to delete the staged files when the CLI exits, so their names now carry the exec's pid and the next wrapper to start removes every one whose pid no longer runs: Pi's config dir in $TMPDIR (which the entrypoint also empties at every container start), and Gemini's context file under ~/.gemini/, the only place Gemini reads a global context file from, so it cannot move to $TMPDIR. A file whose pid runs stays, so a live exec never loses its files. Gemini 0.62.0 and Pi 1.0.1 read stdin when it is not a TTY, whatever kind of file it is (checked in their published sources). Qwen, Hermes and OpenClaw keep their wrappers as they are.
Every sandbox pull request appended its automation rows at the end of the same table, so any two conflicted there and the merge queue could take them only one at a time. These rows now sit after their own anchor row, apart from their siblings'.
yannickmonney
force-pushed
the
perf/sandbox-runtime-turn-io
branch
from
October 9, 2026 03:00
f643365 to
f037ada
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two per-turn costs inside the session: every replay checkpoint (~300 KB every ~5 s per streaming turn) was fsynced twice although only the same runnerd process ever reads it and the spool is wiped at every container start (and a slow fsync could kill a healthy turn under I/O pressure); and the Gemini and Pi Python wrappers stayed resident (~19 MB) for the whole turn only to delete a few KB afterwards.
Changes
perf(sandbox): commit replay checkpoints without syncing them to disk
ExecReplay.saveCheckpoint synced the temporary checkpoint file and then
the spool directory inside the five-second replay I/O deadline; a slow or
failed sync marked the replay REPLAY_UNAVAILABLE and the exec was ended.
The platform saves a checkpoint of up to ~300 KB every five seconds per
streaming turn, so each active turn cost ~0.4 fsync/s, each one flushing
the filesystem journal on a host shared by every session.
The syncs bought no durability: the only reader of a checkpoint is the
runnerd process that wrote it, each ExecReplay spools into a fresh
mkdtemp directory no later process reads, and the entrypoint wipes
/agent/.runtime/tmp at every container start. Keep the write to a
temporary file and the atomic rename, which is what a reader in the same
process needs, and drop both syncs.
The tests that injected a failed directory sync and a stalled file sync
now inject at the steps that remain (a failed prune after the rename, a
stalled write of the temporary file) and keep the same guarantees; a new
test holds that a checkpoint syncs no handle and leaves no temporary file.
perf(sandbox): exec the Gemini and Pi CLIs from their wrappers
tale-gemini-run and tale-pi-run started their CLI as a child and waited
on it for the whole turn, only to delete a few KB of staged files after
it exited. That kept one Python process per Gemini or Pi turn resident
(about 19 MB RSS measured for the idle wrapper) beside the CLI.
Both wrappers now replace themselves with the CLI (os.execvpe, same pid),
so the CLI is the exec's process: its exit status is the exec's, and a
process-group SIGTERM ends it by that signal, which runnerd reports as
143, the status the wrapper used to exit with. The prompt reaches the CLI
on stdin from an anonymous temporary file (any size; a pipe would block
past its buffer with no reader yet), the CLI runs in the exec's workdir,
and SIGPIPE and SIGXFSZ get their defaults back as a subprocess child
would. A wrapper cancelled before its CLI starts, or whose CLI cannot be
started, still removes what it staged.
Nothing is left to delete the staged files when the CLI exits, so their
names now carry the exec's pid and the next wrapper to start removes
every one whose pid no longer runs: Pi's config dir in $TMPDIR (which the
entrypoint also empties at every container start), and Gemini's context
file under ~/.gemini/, the only place Gemini reads a global context file
from, so it cannot move to $TMPDIR. A file whose pid runs stays, so a
live exec never loses its files. Gemini 0.62.0 and Pi 1.0.1 read stdin
when it is not a TTY, whatever kind of file it is (checked in their
published sources).
Qwen, Hermes and OpenClaw keep their wrappers as they are.
Verification
harness-wrapper-exec.test.ts(new),harness-wrapper-cancel.test.ts,gemini-settings.test.ts17 pass over 3 runs; exec-replay tests rewritten to inject at remaining steps; daemon typecheck clean; opengrep clean on the wrappers.Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).