Skip to content

perf(sandbox): drop checkpoint fsyncs and resident harness wrappers - #4546

Merged
yannickmonney merged 3 commits into
mainfrom
perf/sandbox-runtime-turn-io
Oct 9, 2026
Merged

yannickmonney merged 3 commits into
mainfrom
perf/sandbox-runtime-turn-io

Conversation

@yannickmonney

Copy link
Copy Markdown
Contributor

Two per-turn costs inside the session: every replay checkpoint (~300 KB every ~5 s per streaming turn) was fsynced twice although only the same runnerd process ever reads it and the spool is wiped at every container start (and a slow fsync could kill a healthy turn under I/O pressure); and the Gemini and Pi Python wrappers stayed resident (~19 MB) for the whole turn only to delete a few KB afterwards.

Changes

perf(sandbox): commit replay checkpoints without syncing them to disk

ExecReplay.saveCheckpoint synced the temporary checkpoint file and then
the spool directory inside the five-second replay I/O deadline; a slow or
failed sync marked the replay REPLAY_UNAVAILABLE and the exec was ended.
The platform saves a checkpoint of up to ~300 KB every five seconds per
streaming turn, so each active turn cost ~0.4 fsync/s, each one flushing
the filesystem journal on a host shared by every session.

The syncs bought no durability: the only reader of a checkpoint is the
runnerd process that wrote it, each ExecReplay spools into a fresh
mkdtemp directory no later process reads, and the entrypoint wipes
/agent/.runtime/tmp at every container start. Keep the write to a
temporary file and the atomic rename, which is what a reader in the same
process needs, and drop both syncs.

The tests that injected a failed directory sync and a stalled file sync
now inject at the steps that remain (a failed prune after the rename, a
stalled write of the temporary file) and keep the same guarantees; a new
test holds that a checkpoint syncs no handle and leaves no temporary file.

perf(sandbox): exec the Gemini and Pi CLIs from their wrappers

tale-gemini-run and tale-pi-run started their CLI as a child and waited
on it for the whole turn, only to delete a few KB of staged files after
it exited. That kept one Python process per Gemini or Pi turn resident
(about 19 MB RSS measured for the idle wrapper) beside the CLI.

Both wrappers now replace themselves with the CLI (os.execvpe, same pid),
so the CLI is the exec's process: its exit status is the exec's, and a
process-group SIGTERM ends it by that signal, which runnerd reports as
143, the status the wrapper used to exit with. The prompt reaches the CLI
on stdin from an anonymous temporary file (any size; a pipe would block
past its buffer with no reader yet), the CLI runs in the exec's workdir,
and SIGPIPE and SIGXFSZ get their defaults back as a subprocess child
would. A wrapper cancelled before its CLI starts, or whose CLI cannot be
started, still removes what it staged.

Nothing is left to delete the staged files when the CLI exits, so their
names now carry the exec's pid and the next wrapper to start removes
every one whose pid no longer runs: Pi's config dir in $TMPDIR (which the
entrypoint also empties at every container start), and Gemini's context
file under ~/.gemini/, the only place Gemini reads a global context file
from, so it cannot move to $TMPDIR. A file whose pid runs stays, so a
live exec never loses its files. Gemini 0.62.0 and Pi 1.0.1 read stdin
when it is not a TTY, whatever kind of file it is (checked in their
published sources).

Qwen, Hermes and OpenClaw keep their wrappers as they are.

Verification

  • runtime daemon: harness-wrapper-exec.test.ts (new), harness-wrapper-cancel.test.ts, gemini-settings.test.ts 17 pass over 3 runs; exec-replay tests rewritten to inject at remaining steps; daemon typecheck clean; opengrep clean on the wrappers.
  • Full daemon run on macOS: 13 file-ops/main failures that fail identically on unmodified main (pre-existing, macOS-only).
  • Not run: real Gemini/Pi CLIs with a regular-file stdin (checked in their published sources); the image conformance suite in CI exercises Gemini through the wrapper.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).

@yannickmonney
yannickmonney force-pushed the perf/sandbox-runtime-turn-io branch 4 times, most recently from db7c508 to f643365 Compare October 9, 2026 02:40
ExecReplay.saveCheckpoint synced the temporary checkpoint file and then
the spool directory inside the five-second replay I/O deadline; a slow or
failed sync marked the replay REPLAY_UNAVAILABLE and the exec was ended.
The platform saves a checkpoint of up to ~300 KB every five seconds per
streaming turn, so each active turn cost ~0.4 fsync/s, each one flushing
the filesystem journal on a host shared by every session.

The syncs bought no durability: the only reader of a checkpoint is the
runnerd process that wrote it, each ExecReplay spools into a fresh
mkdtemp directory no later process reads, and the entrypoint wipes
/agent/.runtime/tmp at every container start. Keep the write to a
temporary file and the atomic rename, which is what a reader in the same
process needs, and drop both syncs.

The tests that injected a failed directory sync and a stalled file sync
now inject at the steps that remain (a failed prune after the rename, a
stalled write of the temporary file) and keep the same guarantees; a new
test holds that a checkpoint syncs no handle and leaves no temporary file.
tale-gemini-run and tale-pi-run started their CLI as a child and waited
on it for the whole turn, only to delete a few KB of staged files after
it exited. That kept one Python process per Gemini or Pi turn resident
(about 19 MB RSS measured for the idle wrapper) beside the CLI.

Both wrappers now replace themselves with the CLI (os.execvpe, same pid),
so the CLI is the exec's process: its exit status is the exec's, and a
process-group SIGTERM ends it by that signal, which runnerd reports as
143, the status the wrapper used to exit with. The prompt reaches the CLI
on stdin from an anonymous temporary file (any size; a pipe would block
past its buffer with no reader yet), the CLI runs in the exec's workdir,
and SIGPIPE and SIGXFSZ get their defaults back as a subprocess child
would. A wrapper cancelled before its CLI starts, or whose CLI cannot be
started, still removes what it staged.

Nothing is left to delete the staged files when the CLI exits, so their
names now carry the exec's pid and the next wrapper to start removes
every one whose pid no longer runs: Pi's config dir in $TMPDIR (which the
entrypoint also empties at every container start), and Gemini's context
file under ~/.gemini/, the only place Gemini reads a global context file
from, so it cannot move to $TMPDIR. A file whose pid runs stays, so a
live exec never loses its files. Gemini 0.62.0 and Pi 1.0.1 read stdin
when it is not a TTY, whatever kind of file it is (checked in their
published sources).

Qwen, Hermes and OpenClaw keep their wrappers as they are.
Every sandbox pull request appended its automation rows at the end of the same table, so any two conflicted there and the merge queue could take them only one at a time. These rows now sit after their own anchor row, apart from their siblings'.
@yannickmonney
yannickmonney force-pushed the perf/sandbox-runtime-turn-io branch from f643365 to f037ada Compare October 9, 2026 03:00
@yannickmonney
yannickmonney merged commit be9fc22 into main Oct 9, 2026
@yannickmonney
yannickmonney deleted the perf/sandbox-runtime-turn-io branch October 9, 2026 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant