fix(sandbox): stop busy sessions gracefully and add a critical disk tier

[yannickmonney]

On Docker, every stop removed the session container with docker rm --force. So runnerd's graceful stop never ran: it is what lets harnesses flush transcripts and restore staged files, and what lets the DinD supervisor shut its inner engine down cleanly. The disk floor only gated creates and acquires. Running sessions wrote until ENOSPC, and the replay journal on the same disk turned that into "the sandbox could not replay the complete agent output". The spawner healthcheck also ran a curl exec every 10 s.

Stacked on #4551 (its head 5b658a9), which already reworks stop and exec-temp handling in the same files. After #4551 merges, this is retargeted to main and rebased.

Changes

fix(sandbox): let a busy Docker session end its work before its stop

Every Docker stop removed the container with docker rm --force, a
SIGKILL, so runnerd's SIGTERM path (it gives live execs 2 s to write
their harness transcripts and restore what they staged) and the
Docker-in-sandbox supervisor's clean engine shutdown never ran.

A stop can now carry a grace: the backend first runs docker stop -t
on the same fenced container id, then removes it as before, and the
removal's result is still the one that counts. The max-linger
self-reap, the one stop that meets sessions with live execs, uses it:
5 s for a plain session and 20 s for a Docker-in-sandbox one (runnerd's
2 s plus dockerd's 15 s shutdown timeout for its own containers), four
at a time so graceful stops hold at most a third of the 12 shared
docker CLI slots. Each linger stop now registers in the routes' stop
map, so a sweep or idle reclaim already stopping the session is joined
rather than cut short, and a later sweep cannot start a second one.

Idle stops, pressure reclaims, failed-create cleanup and destroys keep
the instant removal: an idle session has nothing to end, and a destroy
deletes the workspace and inner image store a grace would flush into.

fix(sandbox): add a critical tier to the session disk floor

The free-space floor only holds creates and acquires back. Sessions
already running write until the disk is full, and runnerd's replay
journal sits on the same disk, so a full disk ends every running exec.

Below a critical tier (SANDBOX_CRITICAL_FREE_DISK, unset a quarter of
the floor, at least 1 GiB and never above it: 1 GiB on a 20 GiB disk,
1.25 GiB on 100 GiB, 5 GiB under the 20 GiB floor cap; 0, or a floor of
0, turns it off) the sweep now acts on the probe's five-second reading:

• a released idle Docker-in-sandbox session is stopped at once instead
  of after its full idle window, through runnerd's claim like every idle
  stop, so a turn that acquires it meanwhile keeps it; its stop removes
  its inner image store, the most a stop gives back;
• the three largest workspaces are logged, measured by one du of every
  workspace dir at nice 19 (the lowest best-effort I/O priority), cut
  off after 30 s and run at most every ten minutes;
• the disk going below the tier and coming back is logged once each.

An unknown or unreadable disk is never critical. New streaming execs
are not refused here: every agent turn acquires its session right
before its exec, and acquires are already refused below the floor,
which lies above this tier.

fix(sandbox): name a full sandbox disk instead of a lost transcript

When the disk under a session filled up, runnerd's replay journal failed
its next write and ended the exec with REPLAY_UNAVAILABLE, so the turn
read as a lost transcript and nobody learned the host's disk was full.

The journal now maps a write the disk refuses for want of space (ENOSPC,
or EDQUOT for a spent quota) to DISK_FULL, "The sandbox host ran out of
disk space."; every other storage error stays REPLAY_UNAVAILABLE, and
the failure still stands for every later write. exec-manager reports the
journal's own code, so it needs no change. Both protocol copies accept
the code, the spawner forwards it like the other terminal replay codes,
as the stream's error code with no result, and the platform ends the
drain on it without reattaching, as ExecDiskFullError: "the sandbox host
ran out of disk space", in its own words rather than the payload. That
is the reason the automation lane and a task's start window give; a task
drive window keeps its generic "stopped unexpectedly".

The reason string is not a localized message: run failure reasons are
stored English text, like the replay failure it replaces.

perf(sandbox): probe the spawner every 30 s once it is healthy

The spawner's healthcheck ran curl every 10 s, a runc exec each time:
6 container execs a minute for the life of every spawner.

It now probes every 30 s once healthy (2 a minute, a third of the
cost) and every 2 s during its first 30 s (start_interval), so a
deploy still sees a booting spawner ready within seconds instead of
waiting up to an interval. The start period grows from 15 s to 30 s
because the spawner takes its host lock and runs its boot sweep before
it listens, which can pass 15 s on a busy host, and the fast cadence
only applies inside the start period; failures in it still do not
count, as before.

compose.yml, the CLI generator (whose healthcheck type gains
start_interval) and the image's HEALTHCHECK change together, and a
test holds the three to the same cadence. start_interval needs Docker
Engine 25 or later; the repository states no older minimum.

refactor(sandbox): keep the disk helpers private to their modules

The journal's storage-failure mapping and the workspace measurement's
time limit are read only inside their own modules; exporting them left
two exports nothing imports, which knip reports.

fix(sandbox): name a full disk when a checkpoint save hits it too

A journal output write the disk refused ended the exec with DISK_FULL,
but a checkpoint save it refused still recorded REPLAY_UNAVAILABLE.
That failure stands for every later append and attach, so the turn read
as a lost transcript after all.

A checkpoint is the likelier first write to hit a full disk: the
platform saves one at most every 5 s while output flows, up to 1 MiB,
each into a fresh fsynced file that needs new blocks, where an output
append usually lands in the current segment's last block.

The checkpoint save now maps its failure like an output write: ENOSPC
or EDQUOT is DISK_FULL, anything else stays REPLAY_UNAVAILABLE, and the
failure still stands for every later write.

fix(sandbox): hold the critical disk tier to the floor it sits under

The docs say a floor of 0 turns the critical tier off, but an explicit
SANDBOX_CRITICAL_FREE_DISK still applied: with SANDBOX_MIN_FREE_DISK=0
and a 2g tier, the sweep stopped released Docker-in-sandbox sessions
below 2 GiB while admission, its floor off, kept accepting creates.
An explicit tier above the floor did the same: with the 5 GiB default
floor of a 100 GiB disk and an 8g tier, everything between 5 and 8 GiB
free stopped those sessions while new ones still got in.

The tier is now off whenever the floor is, and an explicit one never
stands above the floor; the env reference (EN, DE, FR), .env.example,
the README and sessions.md say so.

fix(sandbox): drop the healthcheck start interval Engine 24 refuses

The spawner's healthcheck set start_interval: 2s (and the image's
HEALTHCHECK --start-interval=2s) to see a booting spawner ready
within seconds. Docker Compose refuses to create a container with
start_interval on an engine older than 25, and the CLI states Engine
24 as the minimum Tale requires, so docker compose up and tale deploy would fail outright there rather than probe more slowly. The
Dockerfile frontend Engine 24 bundles does not know the flag either,
so a local image build there fails too.

The healthcheck keeps its 30 s interval and 30 s start period (2 runc
execs a minute instead of 6) without a start interval, in compose.yml,
the CLI generator and the image alike. The cost is the first healthy
reading, which comes up to 30 s after the spawner starts instead of
within a few seconds, well inside the deploy's 300 s health wait. The test now also holds
all three free of the start interval.

fix(platform): name a full sandbox disk when a task run's drain ends

A task run's drive window settled every drain failure as "the agent
run stopped unexpectedly", so a run whose sandbox host ran out of disk
read as a crash even though the drain ended on ExecDiskFullError. Task
runs are the long agent runs most likely to fill a disk, and the
automation lane and a task's start window already carry the error's
own words.

The drive window now settles that failure as "the agent run stopped:
the sandbox host ran out of disk space"; every other drain failure
keeps the generic reason, and the failure code stays turn_crashed.

fix(sandbox): ask runnerd whether a lingering session is still busy

The max-linger self-reap gave a session a stop grace only when an exec
ran through this spawner's own stream. The platform follows a long turn
by attach and hangs up at every drain window, so the exec it is still
draining is usually registered in no spawner at the moment the reap
fires, and the busy sessions the grace exists for (harness transcripts
to flush, staged files to restore) were still killed outright.

With no exec of its own, the reap now asks the session's runnerd, within
3 s, whether it has a live exec or an operation under way, and grants
the grace when it does. A daemon that does not answer could not act on
the stop and is removed at once.

A reap that joined a stop already under way and saw it end without
stopping the session (a claim a turn won, a failed removal) used to
skip the session, leaving its compute to the sweep's idle logic. It now
falls through to its own graceful stop of that same incarnation.

fix(sandbox): carry the full-disk exec code under the REPLAY_ prefix

runnerd ended an exec whose journal the disk refused with DISK_FULL, a
code no platform before this one knows. Such a platform reads an unknown
terminal code as a dropped stream: it reattached up to its reconnect
limit, each attach replayed the same sticky failure, and the turn ended
with a bare error, where REPLAY_UNAVAILABLE had ended it at once.

The code is now REPLAY_DISK_FULL. Every platform already ends a turn on
any REPLAY_ code without reattaching, so an older one reads it as a
replay failure, and this one still names the full disk.

fix(sandbox): judge the critical disk tier per watched filesystem

The critical tier read one disk reading, the one the probe picks for
having less headroom against the floor. With Docker's data root watched
apart, that could miss a critical Docker data disk, and it stopped
released Docker-in-sandbox sessions when only the workspace disk was
short, although their inner image stores live on Docker's data root and
stopping them freed nothing there. The log always said "the session
disk".

Each filesystem now has its own verdict: a critical workspace disk logs
the largest workspaces, a critical Docker data disk stops released
Docker-in-sandbox sessions, and each log line names its disk. On a host
where they are one disk, nothing changes. A reading that is missing or a
placeholder for an unverifiable mount no longer flips the verdict or
logs a recovery it did not see.

docs(sandbox): keep the workspace listing's doc with its method

The new largestWorkspaces method was inserted between listWorkspaces' docblock and its declaration, leaving that doc above the wrong method. The workspace-usage header also claimed a niced du gets the lowest I/O priority, which holds only under the CFQ and BFQ schedulers; it now says what the deadline bounds instead.

Verification

• services/sandbox passes (167 tests): docker-session-backend, config, host-disk, docker-data-disk, the runnerd protocol mirror and workspace-usage.
• session-routes.test.ts passes (39 tests) for linger, disk, critical and stop. That includes a busy session found through runnerd's health, falling through a joined stop that ended false, and each filesystem judged apart.
• Runtime exec-replay passes (19), including a checkpoint save hitting ENOSPC, which now ends as REPLAY_DISK_FULL.
• Platform session_client and agent_run_host.reap pass (91), as does the CLI compose generator (healthcheck cadence).
• @tale/sandbox and @tale/platform typecheck pass, @tale/sandbox lint passes, the legacy-flow guard passes and lint:manual is ok.
• runnerd-client.test.ts keeps its 3 macOS-only staging timeouts, which also fail on unmodified main.
• Adversarial review found three major issues, and all are fixed here:
  • "Busy" was read only from this spawner's own exec map. The linger reap now asks runnerd.
  • start_interval broke Docker Engine 24, the stated minimum. It's dropped; the probe runs every 30 s with a 30 s start period.
  • A checkpoint save that hit a full disk still reported REPLAY_UNAVAILABLE.
• Its minors and nits are fixed too:
  • A joined stop that ends false now falls through to the reap's own stop.
  • The critical tier is judged per filesystem.
  • The task lane names a full disk.
  • The tier is clamped to the floor.
  • The wire code carries the REPLAY_ prefix, so an older platform ends the turn instead of reattaching.
  • The unavailable-reading flip, the docblock placement and the du priority comment are corrected.
• Not run against a real full disk or a live Docker daemon.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).