Skip to content

perf(sandbox): cut the spawner's idle memory and docker forks - #4542

Merged
yannickmonney merged 8 commits into
mainfrom
perf/sandbox-spawner-idle
Oct 8, 2026
Merged

yannickmonney merged 8 commits into
mainfrom
perf/sandbox-spawner-idle

Conversation

@yannickmonney

Copy link
Copy Markdown
Contributor

The spawner (and every connected device running it) paid for things it rarely used: the Kubernetes client and Sentry SDK loaded on every Docker host, and several timers forked docker CLIs (~10-20 ms CPU and ~28 MB each) to re-learn facts that cannot change while the process runs.

Changes

perf(sandbox): load the Kubernetes client and Sentry only when used

Every Docker spawner and every connected device imported the Kubernetes
backends (and with them @kubernetes/client-node) through backend/index.ts
and capacity.ts, and the Sentry SDK through error-reporting.ts, although
the first is only used with SANDBOX_BACKEND=kubernetes and the second only
with SENTRY_DSN, which is unset by default.

The Kubernetes backends and the capacity reader's API client are now
imported when that backend is selected; the /proc parsers the memory guard
shares with the capacity reader move to a pure proc-stats module. The SDK
is imported only when a DSN is set, and server.ts awaits its init before
it builds anything else, so a failure from then on is still reported;
until then reporting and flushing are no-ops. Type-only imports stay
static.

Measured with bun 1.3.10 on macOS arm64, importing server.ts with the
Docker backend (no server started), 5 runs each:

  • resting RSS 138.5-146.0 MiB (median 142.0) -> 53.8-56.2 (median 55.3)
  • import time 278-454 ms (median 297) -> 76-137 ms (median 117)
  • JS heap 14.4 -> 2.5 MiB
    Alone, @kubernetes/client-node holds ~96 MiB and @sentry/bun 61-68 MiB.

import-footprint.test.ts runs the real entrypoint in a child process and
fails if the module registry lists either package after a Docker boot,
with a control that the registry does list them once imported, and a
check that the Kubernetes backend still loads its client.

perf(sandbox): read the host's fixed facts once per ten minutes

Each capacity snapshot that missed the five-second cache forked three
Docker CLI processes: the session inventory (docker ps --all),
docker info for the CPU count, memory total and kernel, and
docker context inspect for the endpoint, because the spawner image did
not set DOCKER_HOST. The Sandboxes settings page polls every 15 s per
viewer and every connected device reports its status every 15 s, so each
poll was a miss: three forks per 15 s per poller.

The daemon totals, kernel and the local-endpoint verdict now hold for ten
minutes (an endpoint the CLI could not resolve is asked again after 30 s,
and logged instead of silently dropping usage), shared by concurrent
snapshots. The inventory stays one docker ps per five seconds at most:
the session registry cannot replace it, since it knows neither another
spawner's sessions on the same daemon nor exited containers. /proc reads,
the per-read memory-total check and CPU sampling are unchanged.

The image sets DOCKER_HOST=unix:///var/run/docker.sock, the socket every
deployment (compose, tale sandbox connect, device-apply) mounts and the
one the CLI's default context resolves to, so neither the capacity reader
nor the host-memory verdict forks docker context inspect. An operator's
own DOCKER_HOST still wins.

Steady state per poller: 3 forks per 15 s -> 1 fork per 15 s plus 1-2
per ten minutes.

perf(sandbox): sweep only unreferenced DinD volumes

The five-minute host sweep listed every tale.sandbox-dind=1 volume and
ran docker volume rm on each, relying on the daemon refusing the ones
a session still mounts. Every running Docker-in-sandbox session therefore
cost one failing docker volume rm fork per sweep: with 20 such sessions,
21 forks every five minutes to remove nothing.

The listing now filters on dangling=true, so only volumes no container
(running or stopped) references are candidates, and with none the sweep
is the single docker volume ls. There was no protection for a volume a
create had just made and not yet mounted (the build-cache provisioning
sits between volume create and docker run), and removing it there
made docker run recreate it without the sweep's label. Candidates are
now inspected once together and removed only when their CreatedAt is at
least ten minutes old; one without a readable creation time is left
alone. A refused removal, no longer expected, is logged.

perf(sandbox): list legacy one-shot containers at boot and hourly

Every five minutes the host sweep forked docker ps -a --filter label=tale.sandbox=1 for one-shot containers. No current code creates
one, and the boot sweep already removes every one an older release left.
That is 288 forks a day per spawner and per connected device to find
nothing.

DockerBackend now runs that listing (and the one-shot exec-dir sweep it
gates) only once an hour after the boot sweep, and again at the next
tick when the listing failed. The five-minute tick still retries the
workspace trash and reaps orphaned DinD volumes; the DinD pass no longer
waits on the container listing, since its own docker volume ls fails
harmlessly while the daemon is unreachable. 288 -> 24 forks a day.

perf(sandbox): fork less in the build-cache upkeep

The upkeep runs beside every 60 s session sweep. Its first step forked a
docker ps of the build helpers even with the build cache off, the runc
default, where nothing in the process ever starts one: 1,440 forks a day
per spawner and per connected device to list nothing.

For an organization whose helpers had all been stopped past the idle
window, each pass also inspected the builder and its three mirrors to
learn none was running, then inspected the builder again for its stop
time. That is five inspects a minute, 7,200 a day per such organization,
for up to the 14-day cache retention.

  • The helper inventory now carries {{.State}}. A helper it lists as
    exited, created or dead is no stop candidate and is not inspected; any
    other state is inspected as before, and every stop still follows its
    ownership-verifying inspect by immutable id.
  • A stopped builder's FinishedAt is remembered by organization and
    container id. FinishedAt only moves forward for an id, so the memo is
    a lower bound: the builder is inspected again (owner and stop time)
    only once the retention could have passed, right before any removal.
    Entries leave with the container.
  • With the build cache off, an empty helper list is read again at most
    hourly, keyed by Docker target; that still finds helpers a deployment
    with the cache on left behind.

An all-stopped organization now costs the two docker ps of each pass
and one inspect per retention period instead of five inspects a minute.

perf(sandbox): verify the Docker data disk once per process

Docker data-disk admission re-verified its mount with the daemon on a
timer. An explicit SANDBOX_DOCKER_DATA_PATH forked docker info and
docker inspect <self> every 60 s (2,880 forks a day), and on every
5 s disk read while that failed (34,560 a day for a misconfigured
mount). The discovered /etc/hostname bind forked the same two calls
every ten minutes, and every 30 s while unverifiable.

A container's binds and its daemon's data-root do not change under a
running process, so both now verify once per process and again only
when the mount's statfs fails or its /proc/self/mountinfo line changes
(compared every minute: a file read, no Docker call). For the explicit
path the line is the mount the path lives on: the longest mount point
containing it, the last one when stacked.

A failed verification is retried after 30 s, the delay doubling to
10 min while the daemon keeps refuting the mount, or while it verifies
but its statfs keeps failing. One the Docker CLI could not answer
(failed, timed out, no slot) says nothing about the mount and is retried
every 30 s instead: an explicit mount that is unverified holds new
sessions back, so a daemon outage must not keep admission closed for
up to ten minutes after it ends.

Steady state: 2,880 -> 0 forks a day for an explicit mount, 288 -> 0 for
a discovered bind; a permanently refuted mount settles at 2 forks per
10 min.

Verification

  • bun test (services/sandbox): all pass except session-routes.test.ts › cancelled status probes…, which hangs identically on unmodified main on macOS (pre-existing).
  • @tale/sandbox typecheck and lint clean; new import-footprint.test.ts holds the Docker import graph free of @kubernetes/client-node and @sentry/bun.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).

Every Docker spawner and every connected device imported the Kubernetes
backends (and with them @kubernetes/client-node) through backend/index.ts
and capacity.ts, and the Sentry SDK through error-reporting.ts, although
the first is only used with SANDBOX_BACKEND=kubernetes and the second only
with SENTRY_DSN, which is unset by default.

The Kubernetes backends and the capacity reader's API client are now
imported when that backend is selected; the /proc parsers the memory guard
shares with the capacity reader move to a pure proc-stats module. The SDK
is imported only when a DSN is set, and server.ts awaits its init before
it builds anything else, so a failure from then on is still reported;
until then reporting and flushing are no-ops. Type-only imports stay
static.

Measured with bun 1.3.10 on macOS arm64, importing server.ts with the
Docker backend (no server started), 5 runs each:
- resting RSS 138.5-146.0 MiB (median 142.0) -> 53.8-56.2 (median 55.3)
- import time 278-454 ms (median 297) -> 76-137 ms (median 117)
- JS heap 14.4 -> 2.5 MiB
Alone, @kubernetes/client-node holds ~96 MiB and @sentry/bun 61-68 MiB.

import-footprint.test.ts runs the real entrypoint in a child process and
fails if the module registry lists either package after a Docker boot,
with a control that the registry does list them once imported, and a
check that the Kubernetes backend still loads its client.
Each capacity snapshot that missed the five-second cache forked three
Docker CLI processes: the session inventory (`docker ps --all`),
`docker info` for the CPU count, memory total and kernel, and
`docker context inspect` for the endpoint, because the spawner image did
not set DOCKER_HOST. The Sandboxes settings page polls every 15 s per
viewer and every connected device reports its status every 15 s, so each
poll was a miss: three forks per 15 s per poller.

The daemon totals, kernel and the local-endpoint verdict now hold for ten
minutes (an endpoint the CLI could not resolve is asked again after 30 s,
and logged instead of silently dropping usage), shared by concurrent
snapshots. The inventory stays one `docker ps` per five seconds at most:
the session registry cannot replace it, since it knows neither another
spawner's sessions on the same daemon nor exited containers. /proc reads,
the per-read memory-total check and CPU sampling are unchanged.

The image sets DOCKER_HOST=unix:///var/run/docker.sock, the socket every
deployment (compose, `tale sandbox connect`, device-apply) mounts and the
one the CLI's default context resolves to, so neither the capacity reader
nor the host-memory verdict forks `docker context inspect`. An operator's
own DOCKER_HOST still wins.

Steady state per poller: 3 forks per 15 s -> 1 fork per 15 s plus 1-2
per ten minutes.
The five-minute host sweep listed every tale.sandbox-dind=1 volume and
ran `docker volume rm` on each, relying on the daemon refusing the ones
a session still mounts. Every running Docker-in-sandbox session therefore
cost one failing `docker volume rm` fork per sweep: with 20 such sessions,
21 forks every five minutes to remove nothing.

The listing now filters on dangling=true, so only volumes no container
(running or stopped) references are candidates, and with none the sweep
is the single `docker volume ls`. There was no protection for a volume a
create had just made and not yet mounted (the build-cache provisioning
sits between `volume create` and `docker run`), and removing it there
made `docker run` recreate it without the sweep's label. Candidates are
now inspected once together and removed only when their CreatedAt is at
least ten minutes old; one without a readable creation time is left
alone. A refused removal, no longer expected, is logged.
Every five minutes the host sweep forked `docker ps -a --filter
label=tale.sandbox=1` for one-shot containers. No current code creates
one, and the boot sweep already removes every one an older release left.
That is 288 forks a day per spawner and per connected device to find
nothing.

DockerBackend now runs that listing (and the one-shot exec-dir sweep it
gates) only once an hour after the boot sweep, and again at the next
tick when the listing failed. The five-minute tick still retries the
workspace trash and reaps orphaned DinD volumes; the DinD pass no longer
waits on the container listing, since its own `docker volume ls` fails
harmlessly while the daemon is unreachable. 288 -> 24 forks a day.
The upkeep runs beside every 60 s session sweep. Its first step forked a
`docker ps` of the build helpers even with the build cache off, the runc
default, where nothing in the process ever starts one: 1,440 forks a day
per spawner and per connected device to list nothing.

For an organization whose helpers had all been stopped past the idle
window, each pass also inspected the builder and its three mirrors to
learn none was running, then inspected the builder again for its stop
time. That is five inspects a minute, 7,200 a day per such organization,
for up to the 14-day cache retention.

- The helper inventory now carries {{.State}}. A helper it lists as
  exited, created or dead is no stop candidate and is not inspected; any
  other state is inspected as before, and every stop still follows its
  ownership-verifying inspect by immutable id.
- A stopped builder's FinishedAt is remembered by organization and
  container id. FinishedAt only moves forward for an id, so the memo is
  a lower bound: the builder is inspected again (owner and stop time)
  only once the retention could have passed, right before any removal.
  Entries leave with the container.
- With the build cache off, an empty helper list is read again at most
  hourly, keyed by Docker target; that still finds helpers a deployment
  with the cache on left behind.

An all-stopped organization now costs the two `docker ps` of each pass
and one inspect per retention period instead of five inspects a minute.
Docker data-disk admission re-verified its mount with the daemon on a
timer. An explicit SANDBOX_DOCKER_DATA_PATH forked `docker info` and
`docker inspect <self>` every 60 s (2,880 forks a day), and on every
5 s disk read while that failed (34,560 a day for a misconfigured
mount). The discovered /etc/hostname bind forked the same two calls
every ten minutes, and every 30 s while unverifiable.

A container's binds and its daemon's data-root do not change under a
running process, so both now verify once per process and again only
when the mount's statfs fails or its /proc/self/mountinfo line changes
(compared every minute: a file read, no Docker call). For the explicit
path the line is the mount the path lives on: the longest mount point
containing it, the last one when stacked.

A failed verification is retried after 30 s, the delay doubling to
10 min while the daemon keeps refuting the mount, or while it verifies
but its statfs keeps failing. One the Docker CLI could not answer
(failed, timed out, no slot) says nothing about the mount and is retried
every 30 s instead: an explicit mount that is unverified holds new
sessions back, so a daemon outage must not keep admission closed for
up to ten minutes after it ends.

Steady state: 2,880 -> 0 forks a day for an explicit mount, 288 -> 0 for
a discovered bind; a permanently refuted mount settles at 2 forks per
10 min.
Review follow-ups to the idle-footprint changes:

- A failed statfs of the configured Docker data path no longer drops the
  mount's verification or grows a backoff: it says nothing about which
  filesystem the bind is, so the next five-second read tries statfs again
  without a Docker call and admission reopens on the first readable one.
- A daemon that answers "no such object" while the data root is verified
  refutes the mount, so its retry delay grows; only an unanswered call is
  retried at the first delay.
- The first periodic sweep after boot lists legacy one-shot containers
  once more: the boot sweep reads an unanswered daemon as an empty list.
- An unresolvable Docker endpoint is reported once until it resolves, not
  on every 30 s retry while the settings page is open.
- A Sentry SDK that fails to load says why.
@yannickmonney
yannickmonney force-pushed the perf/sandbox-spawner-idle branch from b116e2f to c714331 Compare October 8, 2026 12:35
@yannickmonney
yannickmonney merged commit dee066e into main Oct 8, 2026
79 checks passed
@yannickmonney
yannickmonney deleted the perf/sandbox-spawner-idle branch October 8, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant