fix(sandbox): keep the spawner healthy through restarts and stuck states

[yannickmonney]

Robustness gaps in the Docker spawner and session boot: sessions competed with the control plane at equal CPU weight (issue #3977 recorded a 4 h 45 min control-plane stall), a missing runtime image failed every create slowly and never healed, containers stuck in created/paused/restarting leaked slots forever, cache-volume permissions depended on Docker Hub (a failed chmod left a volume unwritable for good), a stopped session kept its exec temp and boot deleted it synchronously (or failed outright, bricking the session), the spawner refused its own restart for 40-60 s after a crash, and Docker live-restore went unchecked.

Changes

fix(sandbox): yield session CPU to the control plane under contention

Session containers had a --cpus quota but no --cpu-shares, so on cgroup
v2 every session ran at cpu.weight 100, the same as the database,
backend and spawner. On a 4 vCPU / 16 GB host with six busy sessions
the load sat between 26 and 244 with under 1% idle, and the control
plane stalled for about 4 h 45 min (DB pool timeouts, 503s, docker ps
timeouts).

Agent sessions and their organization's build helpers now run at
--cpu-shares 256 and default-profile sessions (code runs, page renders)
at 128, against the 1024 every control-plane container keeps; runc maps
these to cgroup v2 weights of about 10 and 5 against 100. The quotas
stay, and weights only act under contention, so an idle host still
grants each session its full quota. SANDBOX_AGENT_CPU_SHARES tunes the
agent weight (2-262144, whole numbers). A busy helper takes the new
weight in place via docker update; an idle one is recreated as for any
other bound. Kubernetes needs no change: requests already set weights.

fix(sandbox): heal a missing runtime image instead of failing creates

The session argv had no --pull flag, so on a host without the runtime
image docker run tried an implicit pull of the ~2 GB image inside the
30 s run budget. It could never finish: every create held a Docker CLI
slot for 30 s and failed with a 502. Stopped sessions keep no container,
so an image prune on an idle host deletes the image, and the warmup ran
only once at boot. A failed warmup also read as done: ensureImage
returned false after three pulls, warmImage dropped it, and creates
stopped getting 429 runtime_image.

Session containers now start with --pull=never, so a missing image
fails at once. That create tells the warmup (a new optional backend
hook), which pulls again, and the create answers the existing 429
runtime_image instead of a 502. warmImage throws while the image stays
absent; the warmup then keeps creates waiting and retries after 30 s,
1, 2, 5 and then every 10 minutes, with a Retry-After that follows the
next attempt (5-60 s). /health reports the image as unchecked, pulling,
ready or missing with the last error, without turning unhealthy.

fix(sandbox): reap Docker session containers stuck before they ran

listSessions marks any non-running container degraded and only exited
or dead ones ended, and adoption hands degraded sessions to
reapStaleSession, which only the Kubernetes backend implemented. A
container left in created, paused or restarting (a spawner killed
between the daemon's create and start, or a timed-out run whose cleanup
also timed out) therefore held a capacity slot for ever, answered every
create of the id 429 busy and acquire 503, pinned an old runtime image
and survived spawner restarts.

DockerSessionBackend.reapStaleSession removes such a container once the
create's whole budget (SANDBOX_SESSION_CREATE_TIMEOUT_MS, 180 s by
default) and a minute's slack have passed since both its tale.created
stamp and Docker's own creation time. It is fenced like the other
lifecycle transitions: the stamp must match the listed incarnation, and
the removal targets the immutable container id read together with the
state. Its inner image volume and pin marker go with it; the workspace
stays, as with stop. Running, ended, unknown and fresh containers are
left alone.

fix(sandbox): set cache volume modes with the runtime image

Each new per-organization cache volume was made writable by
docker run busybox:1.36 chmod 1777 /cache. tale deploy does not pre-pull
busybox, so the first create of every organization pulled it from
Docker Hub: an air-gapped host could not, and anonymous pulls are
rate-limited. The volume was labelled before the chmod, so a failed
chmod left a labelled root:root 0755 volume that every later ensure
accepted as ready, and that organization's pip/npm/bun caches failed
EACCES for good.

The chmod now runs the sessions' own runtime image, which is on the
host already: --pull=never, --network none, root, with /bin/chmod
(Debian bookworm coreutils) as its entrypoint, and a 30 s bound instead
of 120 s now that nothing is pulled. When it fails the fresh volume is
removed again, so the next create makes it afresh. A failure for want
of the runtime image itself restarts the image warmup, as a session run
that finds it missing does. The image conformance test runs the exact
invocation.

fix(sandbox): retire exec temp on stop and purge it beside runnerd

stopSession removed the container but left /agent/.runtime/tmp (exec
scratch, pip and npm staging, the runnerd replay spool of up to
256 MiB per session) until the next start, where the entrypoint ran
rm -rf /agent/.runtime/tmp before runnerd started. A resume waited for
however long that tree took to delete, and a session that never
resumed kept the bytes for the workspace's whole life (30 days by
default).

The Docker stop now renames the exec temp into the session root's
trash once the container is confirmed gone, inside the stop the routes
already serialize with a resume; the trash's background pass deletes
it, and an erasure of the workspace waits for it as for the rest. The
workspace is agent-writable, so the workspace, .runtime and tmp are
checked with lstat and anything that is not a plain directory (a
planted symbolic link above all) is left alone; the move never falls
back to a delete in place and never fails the stop.

At every start the entrypoint now renames what is left aside as the
profile uid (never into an older leftover) and deletes it in the
background with nice 19 and the idle I/O class, so readiness no longer
waits on it. The background job starts before PATH names the
workspace's dependency dirs, which a root shell must not run from.

fix(sandbox): let a restarted spawner take its dead predecessor's lock

acquireSpawnerLock refuses while the lock is under 60 s old (it is
refreshed every 20 s) unless the holder's PID is dead on the same host.
The spawner is PID 1 of its container, so after an OOM kill at the 1g
memory limit, a Bun crash or main() exiting 1, the restarted container
(same hostname, PID 1 again) probed its own PID, found it alive and
refused; a recreated container has another hostname and read the old
holder as alive too. Init threw, the server exited 1 and Docker
restart-looped it with backoff for 40-60 s.

The lock now records a random instance id, the kernel's boot id and
the process start time (/proc/self/stat starttime). On the same host a
fresh lock is stale when its PID is ours (an earlier process in this
slot), the boot id differs, no process has its PID, or the process
under it started at another time. A lock from another hostname on the
same kernel that names a container id is stale when Docker, which can
see our own container, finds that container gone or stopped. A live
process or container, another machine's lock, an unreadable payload or
a daemon that cannot say still refuse, and the freshness window stays
the arbiter for them.

fix(sandbox): warn at boot when Docker live restore is off

Live restore is neither enabled by Tale nor checked, so any dockerd
restart (a package upgrade, a daemon.json change) stops every session
container and the spawner: running agent turns, builds and renders end
mid-way. The host's daemon configuration is the operator's, so the
spawner does not change it.

After the lock and the boot sweep, the Docker backend now reads the
daemon's live-restore and Swarm state with one docker info call (10 s
bound, beside startup, never delaying it) and logs one warning when live
restore is off: a Swarm node, which refuses live restore, is told so;
any other host is told to set "live-restore": true in daemon.json and
reload the daemon, which applies it without stopping containers. Both
point to a new section of the self-hosted container architecture page
(EN, DE, FR) that explains the setting, its patch-release-only upgrade
limit and the Swarm incompatibility. A daemon that cannot say is logged
and never fails the boot.

test(sandbox): keep the render pids note beside its own test

The CPU weight test was inserted between the note on the crawler's
headless browser and the default-profile test it explains, so the note
read as the reason for the CPU weights. It now sits above its test
again; no assertion changes.

fix(sandbox): never fail a session's boot on leftover exec temp

With set -e, a leftover exec temp tree that neither the rename nor the
in-place delete could clear (an entry the profile uid may not remove, such
as a directory an inner Docker daemon created as root, or a read-only
module cache), and a stale steer queue that could not be removed, ended
the entrypoint. Every later start of the session then failed the same
way, and the only exit was a destroy, which deletes the workspace.

Both now warn and let the boot go on: the execs use the exec temp as it
is, and steer queues are per exec, so a leftover one is read by no later
exec.

Verification

• services/sandbox: the touched suites (304 pass); runtime daemon: exec-temp-startup.test.ts + entrypoint-dispatch.test.ts pass, the new boot-brick test fails against the previous helper; @tale/sandbox and @tale/sandbox-runtime-daemon typecheck and lint clean.
• The full session-routes.test.ts and runnerd-client.test.ts cancellation tests hang on macOS identically on unmodified main (pre-existing); CI runs them on Linux.
• An independent adversarial review is still pending for this branch.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).