Skip to content

fix(sandbox): supervise egress DNS and clear deleted orgs' gateway keys - #4545

Merged
yannickmonney merged 6 commits into
mainfrom
fix/sandbox-egress-supervision
Oct 9, 2026
Merged

yannickmonney merged 6 commits into
mainfrom
fix/sandbox-egress-supervision

Conversation

@yannickmonney

Copy link
Copy Markdown
Contributor

Robustness gaps in the shared sandbox services: the egress container's dnsmasq could die silently while the healthcheck stayed green (a fleet-wide DNS outage for nested containers and BuildKit), tinyproxy fsynced every log line into an unrotated file, the BuildKit and registry-mirror helper images floated, and deleting an organization left its upstream provider keys in the model gateway.

Changes

fix(sandbox): log the egress proxy to stdout instead of a file

tinyproxy wrote its log to /var/log/tinyproxy/tinyproxy.log in the
container's writable layer, where nothing rotated it, and a busybox
tail -F polled the file once a second to mirror it into the container
log. tinyproxy 1.11 also fsyncs the log after every line under its global
log mutex, so on a busy fleet proxy each Notice line cost a journal flush
while every connection thread queued behind it.

With neither LogFile nor Syslog set, tinyproxy 1.11 logs to stdout
(open_log_file(NULL) takes fileno(stdout)), which is the container log the
logging driver already rotates (10 MB x 3). Drop the LogFile directive,
the log file's touch and chown, the tail poller and the one-second wait
the supervisor kept for it; the shutdown line still reaches docker logs
because tinyproxy writes it straight to the inherited stdout. That is one
process and one file per egress container fewer.

DAC_OVERRIDE and CHOWN stay in the capability set: no entrypoint step is
known to need them any more, but the set was verified live and dropping
them waits on a live boot without them.

fix(sandbox): supervise the egress DNS forwarder and probe it

The egress entrypoint started dnsmasq in the background but PID 1 waited
only on tinyproxy, and the health check curled only :3128. Nested Docker
containers and BuildKit RUN steps resolve names only through this
dnsmasq, so its death (or a failure to bind :53 at start) left a proxy
reporting healthy while every session's nested DNS was down, with no exit
for the restart policy to act on.

The supervisor now watches both daemons: it checks every two seconds
(one sleep fork, well under 0.1 % of a core) whether either is still
running, using /proc so no signal permission is needed for the
nobody-owned tinyproxy. When either exits on its own it logs which one
and with what status, stops the other and exits non-zero (the daemon's
status, 1 for a clean exit), so the container restarts whole. A stop
request still stops both cleanly and exits with tinyproxy's status, and
the trap is now set before the daemons start.

The readiness probe also asks dnsmasq: busybox nslookup (an applet of the
alpine 3.20 image, built with the full nslookup that exits 1 on a timeout
or an error reply) queries 127.0.0.1 for sandbox-egress-health.invalid,
which dnsmasq answers from its own --host-record, so the probe needs no
upstream resolver. With nslookup's one-second timeout the probe stays
under the 5 s health timeout after curl's 3 s. The probe stays
byte-identical in the Dockerfile, compose.yml and the CLI generator; the
Kubernetes readiness probe in the docs follows it and gets the same 5 s
timeout instead of Kubernetes' default of 1 s.

dnsmasq's cache grows from its default of 150 names to 4096: the whole
fleet resolves through this one forwarder, and 150 names evicted entries
long before their TTLs, sending repeat lookups upstream. 4096 entries
take well under 1 MB.

fix(sandbox): pin the buildkit and registry mirror images by digest

The shared buildkitd image copied its binaries from
moby/buildkit:buildx-stable-1, a tag that moves with every buildkit
release (it last moved on 2026-09-30, to v0.33.1), so two builds of the
same commit could ship different buildkit versions. The pull-through
registry mirror the spawner runs for every organization defaulted to
registry:2, so each host ran whatever that tag pointed to when it
first pulled it.

Pin both by version and digest, at what the tags resolve to today:
moby/buildkit:v0.33.1@sha256:cec9f139… and registry:2.8.3@sha256:a3d8aaa6….
Renovate's Dockerfile manager tracks the buildkit FROM line; the mirror
default lives in environment defaults no built-in manager reads, so a
regex manager (validated against Renovate's schema) tracks it in
compose.yml, the spawner config and the CLI. The CLI holds the default in
one constant, and a parity test keeps compose.yml, the generator and the
spawner config on it.

tale deploy's runtime preparation now accepts a source that names the
mirror by tag (sources from before this pin) or by version and digest,
pulls it as named, and pins the resolved digest in the bundle as before.

Spawners that ran the old default recreate each organization's mirror
once, while no build runs, keeping its cache volume.

fix(platform): remove a deleted organization's keys from the gateway

Retiring an organization revoked the virtual keys minted for its
sessions, but the provider credentials those keys spent through stayed in
the model gateway's store: the organization's upstream key on every shared
standard provider record (named tale--), and its own
per-(connector, model) custom records, which carry its key and its
endpoint. Nothing ever removed them, so a deleted organization's raw API
keys outlived it in the gateway indefinitely.

The last retirement slice now removes them right after the virtual keys:
it lists every provider record, deletes the records named __… with
their keys, and on every other record deletes only the key with the
organization's exact key name. A record or key already gone counts as
removed, so a second run finds nothing to do. Every record is tried, and
the step throws when the listing fails or any removal is refused, which
fails the job for the queue's backoff like the rest of retirement. The
process also forgets the key ids and fingerprints it remembered for the
organization. The step is not held back for offline devices or the
other slices, since a virtual key whose upstream key is gone reaches
nothing.

The sandbox spec's SBX-R13 card and the admin docs (EN/DE/FR) now say
the deletion removes the provider keys too.

Verification

  • services/sandbox: egress-proxy-config.test.ts, config.test.ts 65 pass; tools/cli compose-parity.test.ts, runtime-prepare.test.ts 166 pass; platform llm_gateway_admin.test.ts, workspace-cleanup.test.ts + domain-spec guard 133 pass; @tale/platform typecheck clean; lint:manual ok.
  • Not observed live: the alpine supervisor and nslookup probe, and the gateway teardown against a real Bifrost (tested against the existing stand-in).

Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).

tinyproxy wrote its log to /var/log/tinyproxy/tinyproxy.log in the
container's writable layer, where nothing rotated it, and a busybox
`tail -F` polled the file once a second to mirror it into the container
log. tinyproxy 1.11 also fsyncs the log after every line under its global
log mutex, so on a busy fleet proxy each Notice line cost a journal flush
while every connection thread queued behind it.

With neither LogFile nor Syslog set, tinyproxy 1.11 logs to stdout
(open_log_file(NULL) takes fileno(stdout)), which is the container log the
logging driver already rotates (10 MB x 3). Drop the LogFile directive,
the log file's touch and chown, the tail poller and the one-second wait
the supervisor kept for it; the shutdown line still reaches docker logs
because tinyproxy writes it straight to the inherited stdout. That is one
process and one file per egress container fewer.

DAC_OVERRIDE and CHOWN stay in the capability set: no entrypoint step is
known to need them any more, but the set was verified live and dropping
them waits on a live boot without them.
The egress entrypoint started dnsmasq in the background but PID 1 waited
only on tinyproxy, and the health check curled only :3128. Nested Docker
containers and BuildKit RUN steps resolve names only through this
dnsmasq, so its death (or a failure to bind :53 at start) left a proxy
reporting healthy while every session's nested DNS was down, with no exit
for the restart policy to act on.

The supervisor now watches both daemons: it checks every two seconds
(one `sleep` fork, well under 0.1 % of a core) whether either is still
running, using /proc so no signal permission is needed for the
nobody-owned tinyproxy. When either exits on its own it logs which one
and with what status, stops the other and exits non-zero (the daemon's
status, 1 for a clean exit), so the container restarts whole. A stop
request still stops both cleanly and exits with tinyproxy's status, and
the trap is now set before the daemons start.

The readiness probe also asks dnsmasq: busybox nslookup (an applet of the
alpine 3.20 image, built with the full nslookup that exits 1 on a timeout
or an error reply) queries 127.0.0.1 for sandbox-egress-health.invalid,
which dnsmasq answers from its own --host-record, so the probe needs no
upstream resolver. With nslookup's one-second timeout the probe stays
under the 5 s health timeout after curl's 3 s. The probe stays
byte-identical in the Dockerfile, compose.yml and the CLI generator; the
Kubernetes readiness probe in the docs follows it and gets the same 5 s
timeout instead of Kubernetes' default of 1 s.

dnsmasq's cache grows from its default of 150 names to 4096: the whole
fleet resolves through this one forwarder, and 150 names evicted entries
long before their TTLs, sending repeat lookups upstream. 4096 entries
take well under 1 MB.
The shared buildkitd image copied its binaries from
moby/buildkit:buildx-stable-1, a tag that moves with every buildkit
release (it last moved on 2026-09-30, to v0.33.1), so two builds of the
same commit could ship different buildkit versions. The pull-through
registry mirror the spawner runs for every organization defaulted to
`registry:2`, so each host ran whatever that tag pointed to when it
first pulled it.

Pin both by version and digest, at what the tags resolve to today:
moby/buildkit:v0.33.1@sha256:cec9f139… and registry:2.8.3@sha256:a3d8aaa6….
Renovate's Dockerfile manager tracks the buildkit FROM line; the mirror
default lives in environment defaults no built-in manager reads, so a
regex manager (validated against Renovate's schema) tracks it in
compose.yml, the spawner config and the CLI. The CLI holds the default in
one constant, and a parity test keeps compose.yml, the generator and the
spawner config on it.

`tale deploy`'s runtime preparation now accepts a source that names the
mirror by tag (sources from before this pin) or by version and digest,
pulls it as named, and pins the resolved digest in the bundle as before.

Spawners that ran the old default recreate each organization's mirror
once, while no build runs, keeping its cache volume.
Retiring an organization revoked the virtual keys minted for its
sessions, but the provider credentials those keys spent through stayed in
the model gateway's store: the organization's upstream key on every shared
standard provider record (named tale-<org>-<provider>), and its own
per-(connector, model) custom records, which carry its key and its
endpoint. Nothing ever removed them, so a deleted organization's raw API
keys outlived it in the gateway indefinitely.

The last retirement slice now removes them right after the virtual keys:
it lists every provider record, deletes the records named <org>__… with
their keys, and on every other record deletes only the key with the
organization's exact key name. A record or key already gone counts as
removed, so a second run finds nothing to do. Every record is tried, and
the step throws when the listing fails or any removal is refused, which
fails the job for the queue's backoff like the rest of retirement. The
process also forgets the key ids and fingerprints it remembered for the
organization. The step is not held back for offline devices or the
other slices, since a virtual key whose upstream key is gone reaches
nothing.

The sandbox spec's SBX-R13 card and the admin docs (EN/DE/FR) now say
the deletion removes the provider keys too.
…odule

The released legacy agent flow reuses one function of llm_gateway_admin.ts,
hashVirtualKey, and its provenance pinned the whole file's hash, so any
change elsewhere in the module (here the organization teardown) failed the
guard although the helper the flow runs is unchanged. It is now pinned by
that declaration's hash, the way the guard already pins the sections it
reuses from external_turn_shared.ts and others.
@yannickmonney
yannickmonney force-pushed the fix/sandbox-egress-supervision branch from beb030b to bcec988 Compare October 8, 2026 12:29
…down

An organization's own gateway records are <org>__<slug>__<model>. A record
that goes on with another '_' after <org>__ belongs to an organization
whose id continues with '_', and the teardown now leaves it alone instead
of deleting it with the retiring organization's records.
@yannickmonney
yannickmonney added this pull request to the merge queue Oct 9, 2026
@yannickmonney
yannickmonney removed this pull request from the merge queue due to a manual request Oct 9, 2026
@yannickmonney
yannickmonney merged commit a0a2854 into main Oct 9, 2026
85 checks passed
@yannickmonney
yannickmonney deleted the fix/sandbox-egress-supervision branch October 9, 2026 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant