Repository navigation
perf(sandbox): cut the spawner's idle memory and docker forks - #4542
Merged
Merged
Conversation
Every Docker spawner and every connected device imported the Kubernetes backends (and with them @kubernetes/client-node) through backend/index.ts and capacity.ts, and the Sentry SDK through error-reporting.ts, although the first is only used with SANDBOX_BACKEND=kubernetes and the second only with SENTRY_DSN, which is unset by default. The Kubernetes backends and the capacity reader's API client are now imported when that backend is selected; the /proc parsers the memory guard shares with the capacity reader move to a pure proc-stats module. The SDK is imported only when a DSN is set, and server.ts awaits its init before it builds anything else, so a failure from then on is still reported; until then reporting and flushing are no-ops. Type-only imports stay static. Measured with bun 1.3.10 on macOS arm64, importing server.ts with the Docker backend (no server started), 5 runs each: - resting RSS 138.5-146.0 MiB (median 142.0) -> 53.8-56.2 (median 55.3) - import time 278-454 ms (median 297) -> 76-137 ms (median 117) - JS heap 14.4 -> 2.5 MiB Alone, @kubernetes/client-node holds ~96 MiB and @sentry/bun 61-68 MiB. import-footprint.test.ts runs the real entrypoint in a child process and fails if the module registry lists either package after a Docker boot, with a control that the registry does list them once imported, and a check that the Kubernetes backend still loads its client.
Each capacity snapshot that missed the five-second cache forked three Docker CLI processes: the session inventory (`docker ps --all`), `docker info` for the CPU count, memory total and kernel, and `docker context inspect` for the endpoint, because the spawner image did not set DOCKER_HOST. The Sandboxes settings page polls every 15 s per viewer and every connected device reports its status every 15 s, so each poll was a miss: three forks per 15 s per poller. The daemon totals, kernel and the local-endpoint verdict now hold for ten minutes (an endpoint the CLI could not resolve is asked again after 30 s, and logged instead of silently dropping usage), shared by concurrent snapshots. The inventory stays one `docker ps` per five seconds at most: the session registry cannot replace it, since it knows neither another spawner's sessions on the same daemon nor exited containers. /proc reads, the per-read memory-total check and CPU sampling are unchanged. The image sets DOCKER_HOST=unix:///var/run/docker.sock, the socket every deployment (compose, `tale sandbox connect`, device-apply) mounts and the one the CLI's default context resolves to, so neither the capacity reader nor the host-memory verdict forks `docker context inspect`. An operator's own DOCKER_HOST still wins. Steady state per poller: 3 forks per 15 s -> 1 fork per 15 s plus 1-2 per ten minutes.
The five-minute host sweep listed every tale.sandbox-dind=1 volume and ran `docker volume rm` on each, relying on the daemon refusing the ones a session still mounts. Every running Docker-in-sandbox session therefore cost one failing `docker volume rm` fork per sweep: with 20 such sessions, 21 forks every five minutes to remove nothing. The listing now filters on dangling=true, so only volumes no container (running or stopped) references are candidates, and with none the sweep is the single `docker volume ls`. There was no protection for a volume a create had just made and not yet mounted (the build-cache provisioning sits between `volume create` and `docker run`), and removing it there made `docker run` recreate it without the sweep's label. Candidates are now inspected once together and removed only when their CreatedAt is at least ten minutes old; one without a readable creation time is left alone. A refused removal, no longer expected, is logged.
Every five minutes the host sweep forked `docker ps -a --filter label=tale.sandbox=1` for one-shot containers. No current code creates one, and the boot sweep already removes every one an older release left. That is 288 forks a day per spawner and per connected device to find nothing. DockerBackend now runs that listing (and the one-shot exec-dir sweep it gates) only once an hour after the boot sweep, and again at the next tick when the listing failed. The five-minute tick still retries the workspace trash and reaps orphaned DinD volumes; the DinD pass no longer waits on the container listing, since its own `docker volume ls` fails harmlessly while the daemon is unreachable. 288 -> 24 forks a day.
The upkeep runs beside every 60 s session sweep. Its first step forked a
`docker ps` of the build helpers even with the build cache off, the runc
default, where nothing in the process ever starts one: 1,440 forks a day
per spawner and per connected device to list nothing.
For an organization whose helpers had all been stopped past the idle
window, each pass also inspected the builder and its three mirrors to
learn none was running, then inspected the builder again for its stop
time. That is five inspects a minute, 7,200 a day per such organization,
for up to the 14-day cache retention.
- The helper inventory now carries {{.State}}. A helper it lists as
exited, created or dead is no stop candidate and is not inspected; any
other state is inspected as before, and every stop still follows its
ownership-verifying inspect by immutable id.
- A stopped builder's FinishedAt is remembered by organization and
container id. FinishedAt only moves forward for an id, so the memo is
a lower bound: the builder is inspected again (owner and stop time)
only once the retention could have passed, right before any removal.
Entries leave with the container.
- With the build cache off, an empty helper list is read again at most
hourly, keyed by Docker target; that still finds helpers a deployment
with the cache on left behind.
An all-stopped organization now costs the two `docker ps` of each pass
and one inspect per retention period instead of five inspects a minute.
Docker data-disk admission re-verified its mount with the daemon on a timer. An explicit SANDBOX_DOCKER_DATA_PATH forked `docker info` and `docker inspect <self>` every 60 s (2,880 forks a day), and on every 5 s disk read while that failed (34,560 a day for a misconfigured mount). The discovered /etc/hostname bind forked the same two calls every ten minutes, and every 30 s while unverifiable. A container's binds and its daemon's data-root do not change under a running process, so both now verify once per process and again only when the mount's statfs fails or its /proc/self/mountinfo line changes (compared every minute: a file read, no Docker call). For the explicit path the line is the mount the path lives on: the longest mount point containing it, the last one when stacked. A failed verification is retried after 30 s, the delay doubling to 10 min while the daemon keeps refuting the mount, or while it verifies but its statfs keeps failing. One the Docker CLI could not answer (failed, timed out, no slot) says nothing about the mount and is retried every 30 s instead: an explicit mount that is unverified holds new sessions back, so a daemon outage must not keep admission closed for up to ten minutes after it ends. Steady state: 2,880 -> 0 forks a day for an explicit mount, 288 -> 0 for a discovered bind; a permanently refuted mount settles at 2 forks per 10 min.
Review follow-ups to the idle-footprint changes: - A failed statfs of the configured Docker data path no longer drops the mount's verification or grows a backoff: it says nothing about which filesystem the bind is, so the next five-second read tries statfs again without a Docker call and admission reopens on the first readable one. - A daemon that answers "no such object" while the data root is verified refutes the mount, so its retry delay grows; only an unanswered call is retried at the first delay. - The first periodic sweep after boot lists legacy one-shot containers once more: the boot sweep reads an unanswered daemon as an empty list. - An unresolvable Docker endpoint is reported once until it resolves, not on every 30 s retry while the settings page is open. - A Sentry SDK that fails to load says why.
yannickmonney
force-pushed
the
perf/sandbox-spawner-idle
branch
from
October 8, 2026 12:35
b116e2f to
c714331
Compare
This was referenced Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The spawner (and every connected device running it) paid for things it rarely used: the Kubernetes client and Sentry SDK loaded on every Docker host, and several timers forked
dockerCLIs (~10-20 ms CPU and ~28 MB each) to re-learn facts that cannot change while the process runs.Changes
perf(sandbox): load the Kubernetes client and Sentry only when used
Every Docker spawner and every connected device imported the Kubernetes
backends (and with them @kubernetes/client-node) through backend/index.ts
and capacity.ts, and the Sentry SDK through error-reporting.ts, although
the first is only used with SANDBOX_BACKEND=kubernetes and the second only
with SENTRY_DSN, which is unset by default.
The Kubernetes backends and the capacity reader's API client are now
imported when that backend is selected; the /proc parsers the memory guard
shares with the capacity reader move to a pure proc-stats module. The SDK
is imported only when a DSN is set, and server.ts awaits its init before
it builds anything else, so a failure from then on is still reported;
until then reporting and flushing are no-ops. Type-only imports stay
static.
Measured with bun 1.3.10 on macOS arm64, importing server.ts with the
Docker backend (no server started), 5 runs each:
Alone, @kubernetes/client-node holds ~96 MiB and @sentry/bun 61-68 MiB.
import-footprint.test.ts runs the real entrypoint in a child process and
fails if the module registry lists either package after a Docker boot,
with a control that the registry does list them once imported, and a
check that the Kubernetes backend still loads its client.
perf(sandbox): read the host's fixed facts once per ten minutes
Each capacity snapshot that missed the five-second cache forked three
Docker CLI processes: the session inventory (
docker ps --all),docker infofor the CPU count, memory total and kernel, anddocker context inspectfor the endpoint, because the spawner image didnot set DOCKER_HOST. The Sandboxes settings page polls every 15 s per
viewer and every connected device reports its status every 15 s, so each
poll was a miss: three forks per 15 s per poller.
The daemon totals, kernel and the local-endpoint verdict now hold for ten
minutes (an endpoint the CLI could not resolve is asked again after 30 s,
and logged instead of silently dropping usage), shared by concurrent
snapshots. The inventory stays one
docker psper five seconds at most:the session registry cannot replace it, since it knows neither another
spawner's sessions on the same daemon nor exited containers. /proc reads,
the per-read memory-total check and CPU sampling are unchanged.
The image sets DOCKER_HOST=unix:///var/run/docker.sock, the socket every
deployment (compose,
tale sandbox connect, device-apply) mounts and theone the CLI's default context resolves to, so neither the capacity reader
nor the host-memory verdict forks
docker context inspect. An operator'sown DOCKER_HOST still wins.
Steady state per poller: 3 forks per 15 s -> 1 fork per 15 s plus 1-2
per ten minutes.
perf(sandbox): sweep only unreferenced DinD volumes
The five-minute host sweep listed every tale.sandbox-dind=1 volume and
ran
docker volume rmon each, relying on the daemon refusing the onesa session still mounts. Every running Docker-in-sandbox session therefore
cost one failing
docker volume rmfork per sweep: with 20 such sessions,21 forks every five minutes to remove nothing.
The listing now filters on dangling=true, so only volumes no container
(running or stopped) references are candidates, and with none the sweep
is the single
docker volume ls. There was no protection for a volume acreate had just made and not yet mounted (the build-cache provisioning
sits between
volume createanddocker run), and removing it theremade
docker runrecreate it without the sweep's label. Candidates arenow inspected once together and removed only when their CreatedAt is at
least ten minutes old; one without a readable creation time is left
alone. A refused removal, no longer expected, is logged.
perf(sandbox): list legacy one-shot containers at boot and hourly
Every five minutes the host sweep forked
docker ps -a --filter label=tale.sandbox=1for one-shot containers. No current code createsone, and the boot sweep already removes every one an older release left.
That is 288 forks a day per spawner and per connected device to find
nothing.
DockerBackend now runs that listing (and the one-shot exec-dir sweep it
gates) only once an hour after the boot sweep, and again at the next
tick when the listing failed. The five-minute tick still retries the
workspace trash and reaps orphaned DinD volumes; the DinD pass no longer
waits on the container listing, since its own
docker volume lsfailsharmlessly while the daemon is unreachable. 288 -> 24 forks a day.
perf(sandbox): fork less in the build-cache upkeep
The upkeep runs beside every 60 s session sweep. Its first step forked a
docker psof the build helpers even with the build cache off, the runcdefault, where nothing in the process ever starts one: 1,440 forks a day
per spawner and per connected device to list nothing.
For an organization whose helpers had all been stopped past the idle
window, each pass also inspected the builder and its three mirrors to
learn none was running, then inspected the builder again for its stop
time. That is five inspects a minute, 7,200 a day per such organization,
for up to the 14-day cache retention.
exited, created or dead is no stop candidate and is not inspected; any
other state is inspected as before, and every stop still follows its
ownership-verifying inspect by immutable id.
container id. FinishedAt only moves forward for an id, so the memo is
a lower bound: the builder is inspected again (owner and stop time)
only once the retention could have passed, right before any removal.
Entries leave with the container.
hourly, keyed by Docker target; that still finds helpers a deployment
with the cache on left behind.
An all-stopped organization now costs the two
docker psof each passand one inspect per retention period instead of five inspects a minute.
perf(sandbox): verify the Docker data disk once per process
Docker data-disk admission re-verified its mount with the daemon on a
timer. An explicit SANDBOX_DOCKER_DATA_PATH forked
docker infoanddocker inspect <self>every 60 s (2,880 forks a day), and on every5 s disk read while that failed (34,560 a day for a misconfigured
mount). The discovered /etc/hostname bind forked the same two calls
every ten minutes, and every 30 s while unverifiable.
A container's binds and its daemon's data-root do not change under a
running process, so both now verify once per process and again only
when the mount's statfs fails or its /proc/self/mountinfo line changes
(compared every minute: a file read, no Docker call). For the explicit
path the line is the mount the path lives on: the longest mount point
containing it, the last one when stacked.
A failed verification is retried after 30 s, the delay doubling to
10 min while the daemon keeps refuting the mount, or while it verifies
but its statfs keeps failing. One the Docker CLI could not answer
(failed, timed out, no slot) says nothing about the mount and is retried
every 30 s instead: an explicit mount that is unverified holds new
sessions back, so a daemon outage must not keep admission closed for
up to ten minutes after it ends.
Steady state: 2,880 -> 0 forks a day for an explicit mount, 288 -> 0 for
a discovered bind; a permanently refuted mount settles at 2 forks per
10 min.
Verification
bun test(services/sandbox): all pass exceptsession-routes.test.ts › cancelled status probes…, which hangs identically on unmodified main on macOS (pre-existing).@tale/sandboxtypecheck and lint clean; newimport-footprint.test.tsholds the Docker import graph free of @kubernetes/client-node and @sentry/bun.Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).