perf(sandbox): cut BuildKit stack upkeep, idle time and disk

[yannickmonney]

Each organization's build stack (builder plus three registry mirrors) cost a lot on the session critical path and while idle:

• A warm DinD create ran about 13–15 docker CLI calls (3–4 of them docker exec) to re-verify the stack: roughly 0.6–0.9 s.
• The idle sweep inspected dormant organizations' helpers every minute for up to 14 days.
• Helpers lingered 30 minutes after the last session.
• Idle organizations kept their mirror caches.
• The cache cap was a fixed 20 GB per organization, whatever the disk.
• Legacy global BuildKit volumes were kept forever on upgraded hosts.

Stacked on #4551 (its head 5b658a9), which already changes buildkitd.ts, cleanup.ts and config.ts. After #4551 merges, this is retargeted to main and rebased.

Changes

perf(sandbox): read both session firewall families in one exec

Attaching a DinD session to its organization's build network read the
FORWARD chain once per address family before the connect and again
after it: four docker execs on a guarded runtime, six where ip6tables
is missing and the IPv6 switches had to be read too. Each exec is a CLI
process and a round trip on the session create's path.

One root sh -c now prints both families' listings, each behind a line
with its exit status, plus the IPv6-disabled verdict: one exec before
the connect and one after it (three when an old runtime needs the guard
installed). A listing missing a section is refused, never read as an
empty chain. The session and build-network inspects before the connect,
and the firewall and membership checks after it, only read, so each
pair runs together; the firewall's verdict is still reported first.

perf(sandbox): confirm verified build helpers with one docker inspect

Every DinD agent create ran ensureBuildkitd in full before the session
container: the organization network inspect, an inspect of the egress
network and of every container on it to find the proxy, an iptables
listing in the proxy, a network connect that normally failed with
"already exists", one inspect per helper (4), three execs for the
builder's fence, cached image-id lookups, and the backend then inspected
the organization network again for the attach plan: 13-15 docker CLI
processes, about 0.6-0.9 s on the create's critical path, repeated for
every create of an organization.

The full check now costs 5-7 calls: the organization network is
inspected once and answers both whether the proxy is attached (no
connect when it is) and the session's plan, which provisionBuildkitd
returns to the backend; the proxy is re-read by the container id it was
last found under, the egress network searched again only when that id
no longer answers to the proxy's name; the four helpers are read in one
inspect (a refused mirror still fails alone); the fence marker, pinned
[dns] and current proxy address are read in one sh -c exec with the
host passed as a positional parameter.

When the full check finds every helper running as it would be launched
now and the fence conclusively current, creates in the next 60 s confirm
it with ONE inspect of the proxy and the four helpers by id: any restart
(StartedAt), replacement, stopped helper, moved proxy address or network,
or changed stamp falls back to the full check. A fence can only go stale
through one of those, so the memo never masks a drifted fence; an
unreadable fence is reused but never remembered; adoption's reconcile
passes fresh and always runs the full check; idle stops and removals
forget the organization.

perf(sandbox): keep build-cache upkeep inspects off the reserved lane

The minute sweep already lists helpers with their state and skips
organizations whose helpers all stopped, and remembers a stopped
builder's stop time by container id. Two costs remained.

Every inspect the sweep made took spawn-util's priority lane, whose
four reserved slots exist so a create's identity inspect or a health
probe never waits behind long calls. Upkeep that can wait a minute now
waits in the shared lane: the stop candidates' inspects, the retention
read and the disk-pressure reads all pass 'shared'; the create path
keeps the priority lane.

While the session disk stays short, every sweep inspected each stopped
organization's three mirrors and its builder before choosing which
caches to remove: 4 inspects per stopped organization per minute for
as long as the disk stays short. The choice is now made from the
inventory's states and the remembered stop time (a builder listed as
anything but stopped forgets it), so a remembered organization costs
none; its builder is read again under the organization's lock right
before a removal, which keeps the ownership check.

The helper stamp label is not added to the inventory: no sweep decision
reads it, and the create path inspects the helpers it may start anyway.

perf(sandbox): stop idle build helpers sooner and start them by id

An organization's builder and three mirrors kept running for the whole
session idle window (SANDBOX_SESSION_MAX_IDLE_MS, 30 min) after its last
agent session that builds had ended, holding up to twice an agent's
memory (8 GiB by default) plus 4 x 512 MB of mirror limits for nothing.
Restarting them is cheap: the builder's cache survives in its volume.

SANDBOX_BUILDKITD_IDLE_MS (default 600000, 10 min; at least 60000 since
the sweep runs once a minute) now sets the helpers' own window, apart
from the session one, cutting their idle tail by two thirds.

The next build also stops paying for a full recreate: a stopped helper
whose stamp and image are current is docker started by the immutable
id its inspect showed (mirrors before the builder, which pulls through
them), keeping the container and its writable layer instead of rm -f
plus docker run. A drifted helper, or one whose start fails, is still
recreated, and the entrypoint reinstalls the egress fence on every start
as it does for a restart.

The variable is read from the .env like the other SANDBOX_BUILDKITD_*
tuning knobs and documented in .env.example, the environment reference
(EN/DE/FR) and both service READMEs.

perf(sandbox): drop idle organizations' registry mirror caches

Each organization keeps three registry:2 pull-through mirrors with a
volume each. When its helpers stopped for want of agent sessions, the
mirror volumes stayed for the whole cache retention (14 days by
default) although nothing pulled through them, while the builder's own
cache, pruned to the idle budget (5 GB) right before the stop, already
holds the base layers its builds used: every dormant organization paid
for the same layers twice, and the mirror copies only shrink through
registry expiry after seven days.

An idle stop now removes the three mirrors and then their volumes, so
mirror bytes for an organization that is not building drop to zero; the
next build recreates them empty before starting the builder. Each mirror
is inspected by its immutable id right before its removal and must be
owned, private and stopped (one running again keeps its container and
cache); its volume goes only afterwards, by its exact name and with the
organization's ownership label checked, and Docker refuses a volume any
container still uses. The first sweep past the idle window also removes
mirrors an earlier stop left, once; an organization with none costs no
call. A mirror that cannot go is logged and kept.

perf(sandbox): size each builder's cache cap from the session disk

The GC policy shipped a static maxUsedSpace of 20GB per organization
(services/sandbox-buildkitd/buildkitd.toml), so N organizations that
build could hold 20N GB whatever the disk: ten of them claim 200 GB of
a 250 GB host. minFreeSpace (5%) only prunes while a builder runs.

The spawner now sizes the cap and passes it to the builder in bytes:
SANDBOX_BUILDKITD_MAX_CACHE (at least 1g, since zero is no cap to the
GC policy), else a tenth of the session disk's size, at least 1 GiB and
at most the shipped 20 GiB (100 GiB disk: 10 GiB cap). The size is read
the way admission reads the disk, statfs on the session root (blocks x
block size), again at most every ten minutes; a failed read keeps the
last size, or the shipped cap before any read. The reserved floor that
disk pressure never prunes below follows as a tenth of the cap, at most
the shipped 2 GiB; minFreeSpace stays the shipped 5%, already relative
to the disk. Both are rounded to whole MiB.

The entrypoint substitutes TALE_BUILDKITD_MAX_USED and
TALE_BUILDKITD_RESERVED into the live config like max-parallelism
(digits only, at least 1; unset keeps the shipped values). BuildKit's
size parser reads a plain digit string as bytes. Both values join the
builder's stamp, so an idle builder adopts a changed cap when it is
recreated; the first idle recreate after the upgrade applies it.

fix(sandbox): remove retired global build helpers after the retention

Hosts upgraded from the global build cache kept its five helper
containers and four cache volumes forever: retireLegacyBuildkitd only
stopped the helpers once no session used the global endpoint, and the
global cache alone could hold up to 20 GB plus three registry mirrors
nobody pulls through any more.

Once every legacy helper has been stopped longer than
SANDBOX_BUILDKITD_CACHE_RETENTION (14 days by default, read from each
container's FinishedAt; off keeps them) and the existing check still
finds no running, paused, restarting or starting session using the
global endpoint, the spawner removes exactly those containers by the
ids it just inspected them under, then the four exact legacy volumes
(tale-buildkitd-cache, tale-buildkitd-mirror-cache-docker-io, -ghcr-io,
-quay-io). Each must carry tale.buildkitd=1 and no tale.org; anything
else is refused and logged, Docker refuses a running container or an
in-use volume, and every removal is logged. A removal Docker refuses is
tried again in an hour.

The retirer no longer stops looking once the helpers are stopped: it
remembers when the retention could next have passed (the latest stop
time plus the retention) and inspects nothing until then, so the sweep
every minute costs no call in between. Both callers pass the configured
retention; its 14-day default now lives with the legacy retirement and
the organization expiry reads it from there.

fix(sandbox): keep legacy cache removal off creates and retry volumes

A session create ran the legacy retirement inside its 5 s provisioning
budget, and the removal pass past the retention is about 20 Docker calls,
four of them volume rm with 30 s bounds over the old global cache. Once
the budget ran out mid-loop, an aborted rm or inspect left a volume behind
after its helpers were already gone; the next pass saw no helpers and
marked the retirement done, so the volumes leaked for good and the create
lost its build cache to the rejected ensure.

A create now only stops drained helpers; removal past the retention is
left to the minute maintenance sweep, which runs without a create budget,
and a create never waits on a removal pass under way. Each legacy volume
is removed on its own: one whose inspect or rm fails is logged and tried
again an hour later, the process remembering that retirement removed the
helpers, while a refused or absent volume is not retried.

fix(sandbox): start a stopped builder only with the mirrors it needs

A stopped builder whose stamp and image still match is now started by id
instead of recreated, but the stamp does not cover what the builder was
launched with beyond its bounds: its registry-mirror mapping and its
egress proxy. A builder created while one mirror failed to launch left
that registry out of its mapping, and before this change the idle-stop
and recreate cycle healed it; a start keeps the launch's mapping, so it
would have stayed without that registry's cache until an image or bound
change, a broken fence or the retention.

A builder now carries a label recording the registries its mapping
covers and a hash of its proxy (never the proxy itself, which may carry
credentials). A stopped builder is started again only when that record
covers every registry whose mirror serves now and names the current
proxy; otherwise it is recreated. A running builder missing a registry
is drifted and is recreated once idle. A registry it maps whose mirror
is down now is no drift: BuildKit cannot pull it without the mirror
either way. A builder from before the label is recreated once when idle.

fix(sandbox): size the builder's disk cap in whole GiB

The disk-sized build-cache cap went into the builder's stamp at whole-MiB
precision. Some filesystems report a total that moves with use: on ZFS
f_blocks counts what is referenced plus what is available, shared with
every dataset in the pool. Below 200 GiB, the cap then changed at nearly
every ten-minute disk re-read, the stamp with it, and an idle builder was
recreated on a create path each time, while its verified memo never
formed. The disk-sized cap is now whole GiB, so a total moving by a few
MiB keeps it; the operator's cap stays whole MiB.

fix(sandbox): let a create cut an idle mirror retirement short

An idle organization's three registry mirrors and their volumes were
removed one after another under the organization's lock, after a single
check that nothing wanted them. A session create that took the
organization's lease meanwhile waited behind that lock with only its
5 s provisioning budget, and could fall back to its local builder with a
cold cache, while the idle stop of the helpers already stops for a
create before each step. The retirement now stops before the next mirror
once a create holds the lease; the next sweep removes what is left.

fix(sandbox): pull a missing mirror image beside creates, not in them

Idle organizations' mirror containers are now removed, so on a host
where every organization is idle nothing references the mirror image,
and an image prune can delete it. The next build's mirror launch then
had to pull it inside a create's 5 s provisioning budget; a slow or
rate-limited pull was cut short at every attempt, and the organization
got no shared mirror cache until one finished in time.

Mirrors now launch with --pull never. A launch that finds the image
missing reports that mirror unavailable for this create, as any mirror
that fails to come up already is, and starts one background pull of the
image, outside every create's budget and shared by all the launches that
asked; the next launch after it lands starts the mirror.

fix(sandbox): count a never-started legacy helper from its creation

A legacy build helper that never ran reports Docker's zero time as its stop time, which read as stopped now on every pass. The retirement then put a full retention ahead each time, and none of the five legacy helpers or their four volumes were ever removed. Such a helper now counts from its creation time.

Verification

• services/sandbox passes (318 tests): BuildKit lifecycle, buildkitd, BuildKit resources, the network guard, config, build-cache create, docker-session-backend and workspaces. That includes the verified memo, the folded firewall exec, docker start by id, mirror retirement (including a create cutting it short), the disk-sized cap (in whole GiB), legacy removal with volume retries and never-started helpers, the launch-label drift check, and a missing mirror image pulled once in the background.
• The new F4 test fails without its fix.
• @tale/sandbox typecheck and lint pass, the docs suite passes (EN/DE/FR) and lint:manual is ok.
• Adversarial review found two major issues, both fixed here:
  • Legacy volume removal could leak permanently. It now never runs under a create's budget, and its volume pass is retried.
  • A stopped builder could be started again with an incomplete mirror mapping. It now carries a launch label for its mirrors and proxy, and drift forces a recreate.
• Its minors and nit are fixed too: the whole-GiB cap, a create cutting mirror retirement short, the mirror image pulled outside create budgets, and the never-started legacy helper.
• buildkit-lifecycle.test.ts can hit the default 5 s timeout under heavy machine load. It passes with --timeout 30000.
• Not run against a real Docker daemon. The latency savings are worked out from call counts, not measured.

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).