perf(sandbox): cut DinD session start and idle costs

[yannickmonney]

Several costs hit every Docker-in-sandbox (DinD) session start or idle period:

• Each create or resume re-registered the buildx builder by starting Node and the Docker CLI before runnerd, about 150–300 ms.
• Gemini and the per-turn Node helpers compiled tens of MB of JS without a V8 compile cache.
• A session's inner Docker engine pulled every Docker Hub image through the egress proxy, even though the organization's docker.io mirror already ran on a network the session joins.
• A pinned session's inner store was never trimmed.
• A released DinD agent session held its slot for the full 30-minute window even when its engine had never started.

Changes

perf(sandbox): reuse the session's buildx builder without the CLI

Every Docker-in-sandbox create and resume ran setup_shared_buildx_builder
before runnerd started: a root Node process only to SHA-256 the buildkitd
endpoint, then setpriv docker buildx inspect (and buildx create on a
first boot). Each buildx call starts the Docker CLI and its buildx plugin,
two Go binaries; together with Node that is an estimated 150-300 ms and
~40 MB of transient memory on the critical path of every start, although
the builder definition persists in the workspace HOME across resumes.

The name now comes from printf %s | sha256sum | cut -c1-24, the same
"tale-build-" + 24 hex digits the Node hash produced (a test compares
both for several endpoints), and a regular definition file at
~/.docker/buildx/instances/ owned by the agent uid selects the
builder with no buildx call at all. Anything else falls back to inspect,
then create, as before.

The step is now best-effort in fact: a failing node -e used to abort
the whole boot under set -e, despite the comment. The setup runs in an
|| list, a missing or malformed digest selects the local builder, and a
failing create logs and keeps the session starting.

perf(sandbox): keep a V8 compile cache in the session HOME

Every Node program a session runs compiled its JavaScript from scratch on
every start: gemini's bundled CLI (about 94 MiB of JS), the per-turn node
helpers and runnerd itself, on each turn and again after every resume.
Node 24 reuses V8 code caches from NODE_COMPILE_CACHE, so the session
dispatch now exports /agent/.runtime/home/.cache/node-compile-cache next
to HOME and TMPDIR and creates it with the workspace skeleton at the
session uid. It lives in the persistent HOME, never under the exec temp
every container start wipes.

The directory is agent-writable, and V8 trusts a code cache it loads, so
the root Docker supervisor must never read it: the DinD exec now unsets
NODE_COMPILE_CACHE beside NODE_PATH and hands it to runnerd through
TALE_RUNNER_NODE_COMPILE_CACHE, as it already did for NODE_PATH. The
runner environment moved into runnerEnvironment() so a test covers it.
No image rebuild is needed beyond the next runtime release.

perf(sandbox): pull docker.io through the org mirror in DinD sessions

A Docker-in-sandbox session's docker pull / compose pull went to
Docker Hub through the egress proxy every time, because each session's
inner /var/lib/docker starts empty: the same base images were fetched
again per session, costing bandwidth, proxy CPU and Hub rate limit,
while the organization already runs a registry:2 pull-through cache
for docker.io that only its BuildKit daemon used.

When the session has an organization build stack, the spawner now passes
that mirror's host:5000 as TALE_DOCKER_HUB_MIRROR (derived with
buildkitdMirrorRef, validated like every argv value), the root
supervisor hands it to the engine child, and start_inner_dockerd adds
--registry-mirror http://<host>:5000 --insecure-registry <host>:5000
and appends the host to NO_PROXY/no_proxy for dockerd only, so the plain
HTTP pull stays on the private network. dockerd falls back to Docker Hub
when the mirror does not answer. A malformed value is ignored with a
warning; the fresh inner volume per create is unchanged.

perf(sandbox): trim a large inner Docker store before engine sleep

The inner engine of a Docker-in-sandbox session sleeps after five idle
minutes, but its /var/lib/docker only ever grows while the session
container lives: every rebuilt tag leaves a dangling image and every
local build adds cache. A pinned session keeps its container for days,
so nothing bounded its inner store short of the volume filling up.

Before an idle stop the supervisor now reads /system/df through the
private engine socket (images and build cache only) and, above 10 GiB,
prunes dangling images and the build cache down to the 5 GiB used most
recently (reserved-space + max-used-space from API 1.48, keep-storage
before), then logs the bytes reclaimed. It uses the engine API rather
than the Docker CLI, like the idle inventory, so no client config or
plugin runs as root. Every request is bounded (150 s overall), a failed
prune is logged while the other still runs, the stop goes ahead anyway,
and a Docker client arriving during the trim keeps the engine running.
Tagged and in-use images, containers and volumes are never touched; the
fresh inner volume per create is unchanged.

perf(sandbox): stop released DinD sessions whose engine never ran early

Docker-in-sandbox agent sessions kept the full 30-minute released idle
window, so a resume would not cost a re-pull into an empty inner image
store. But the inner engine is lazy: most agent turns never run a Docker
command, and /healthz reported only dockerReady, which is true for a cold
engine. Every such session held its slot and memory 25 minutes longer
than a plain session for an image store that did not exist.

The root supervisor now reports its engine lifecycle on the health
socket (x-tale-docker-engine: cold until the first activation, running
while one starts or runs, stopped after it slept or failed), and runnerd
adds it to /healthz as docker: { engine, used } without activating
anything. The spawner keeps the full window only when used is true;
a never-used engine gets the normal 5-minute released window, and an
older runtime that omits the field keeps today's behaviour. Both copies
of the runnerd protocol carry the field, and the mirror test checks that
each assigns to the other.

docs(manual): record the Docker-in-sandbox start-cost automation

The specs that now own the buildx builder reuse, the session compile
cache, the docker.io mirror for the inner engine, the trim before engine
sleep and the engine state on /healthz get one row, so a manual round
does not re-test them. The session sweep row says that the full released
idle window now depends on the inner engine having run.

docs(sandbox): narrow the full released window to DinD engines that ran

The environment reference said every agent session with Docker inside
keeps the full idle window once released. The spawner now keeps it only
when the session's inner Docker daemon has run (runnerd's /healthz
docker.used), because only then is there an image store a resume would
lose; a session whose daemon never started gets the 5-minute released
window like any other. Updated in English, German and French.

fix(sandbox): never fail a session boot over the Node compile cache

The compile-cache directory sat in the entrypoint's fatal skeleton
mkdir, which runs under set -e. When ~/.cache exists but the session
uid cannot write it, mkdir fails, the entrypoint exits 1 and the
workspace can never boot again. A Docker-in-sandbox session reaches
that state on its own: an inner docker run -v ~/.cache/pip:... before
~/.cache existed makes the root dockerd create the bind source and its
parents root-owned. Before the cache, boot never wrote inside ~/.cache.

The cache is only a speed-up, so it is now created after the skeleton,
at the session uid, and NODE_COMPILE_CACHE is exported only when the
directory exists and is writable; otherwise the boot logs a warning and
Node compiles without it. Reproduced with ~/.cache at mode 0555: the
old entrypoint exits 1 before tini, the new one reaches tini with no
cache set. The dispatch test pins that case.

fix(sandbox): name the docker.io mirror only while it is up

A session's inner Docker engine was told to pull docker.io through the
organization's mirror whenever the session had a builder endpoint. The
mirrors are launched best-effort: one that fails to come up is dropped
from the builder's mapping while the builder itself is still returned.
Every session in that organization then sent each docker.io pull to a
mirror name that did not resolve before falling back to Docker Hub,
paying the session's DNS failure time on each pull.

Provisioning now reports the mirrors that came up with the endpoint, and
a session is told about the docker.io mirror only when it is among them.
The run arguments also refuse a mirror reference that is not the
session's own organization's, so no session can be pointed at another
organization's cache.

fix(sandbox): count an inner store with images as used after a restart

The full released window is for a DinD session whose inner Docker store
is worth keeping, and the health answer named a store used only once its
engine had started in the current supervisor. On Kubernetes a runner
container that restarts inside its Pod keeps the store, and the engine
starts at boot only when containers are left in it. A store holding
pulled or built images but no containers then read as never used, and
the session got the five-minute window, losing the store the full window
exists to keep.

At boot, a store whose image database has entries now marks the engine
as used without starting it.

test(sandbox): fail the mirror test when a health copy drops a field

The protocol mirror test assigned the Docker engine health between the
two RunnerdHealth declarations and back, which passes even when one copy
lacks the optional field: an object with an extra optional property is
assignable either way. The test now also requires the two field types to
be identical, so the typecheck fails when the copies drift.

fix(sandbox): reuse a buildx definition only when it names the endpoint

The boot shortcut selected the session's buildx builder whenever a
regular file the agent owns sat at its definition path, including an
empty or edited one, which buildx inspect would have rejected. Every
docker build in the session then failed instead of falling back to the
local builder. The shortcut now also requires a non-empty definition
that names the organization's endpoint; anything else goes through
buildx as before.

docs(sandbox): tell operators how inner Docker engines pull and trim

The operator environment reference described the organization's registry
mirrors as serving builds only. With the shared build cache on, every
session's inner Docker engine now also pulls Docker Hub images through
the organization's docker.io mirror, falling back to Docker Hub when it
is not running, and an idle inner engine trims a store above 10 GiB
before it sleeps. A new section says both in EN, DE and FR. The service
docs now say that a Kubernetes inner store limited below 10 GiB is
bounded by that limit instead, since the kubelet evicts the Pod there
before the trim could run.

Verification

• Runtime daemon suites pass (101 tests): build-cache startup, entrypoint dispatch, inner Docker health and network, lazy-docker and lazy-docker health. The buildx shortcut's shell is run under /bin/sh, including empty and foreign definitions.
• services/sandbox passes (132 tests): build-cache create, BuildKit provisioning (the docker.io mirror is reported only while it is up), session run arguments (another org's mirror is refused) and the protocol mirror. The protocol test now requires the two health copies to be type-identical.
• session-routes.test.ts released-window cases pass (11).
• @tale/sandbox and daemon typecheck pass, @tale/sandbox lint passes, and the docs suite passes (685 tests, EN/DE/FR).
• Adversarial review found one major issue, now fixed: the compile-cache directory had been in the fatal boot mkdir list, so an unwritable ~/.cache could stop a workspace from booting. Its minors and nits are fixed too: the mirror is named only while up, the store-used flag survives a runner restart, the type-exact mirror test, the buildx definition check, and the operator and K8s trim docs.
• Not measured on a live host. The latency and CPU savings are estimates from the analysis.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).