Repository navigation
fix(sandbox): supervise egress DNS and clear deleted orgs' gateway keys - #4545
Merged
Merged
Conversation
tinyproxy wrote its log to /var/log/tinyproxy/tinyproxy.log in the container's writable layer, where nothing rotated it, and a busybox `tail -F` polled the file once a second to mirror it into the container log. tinyproxy 1.11 also fsyncs the log after every line under its global log mutex, so on a busy fleet proxy each Notice line cost a journal flush while every connection thread queued behind it. With neither LogFile nor Syslog set, tinyproxy 1.11 logs to stdout (open_log_file(NULL) takes fileno(stdout)), which is the container log the logging driver already rotates (10 MB x 3). Drop the LogFile directive, the log file's touch and chown, the tail poller and the one-second wait the supervisor kept for it; the shutdown line still reaches docker logs because tinyproxy writes it straight to the inherited stdout. That is one process and one file per egress container fewer. DAC_OVERRIDE and CHOWN stay in the capability set: no entrypoint step is known to need them any more, but the set was verified live and dropping them waits on a live boot without them.
The egress entrypoint started dnsmasq in the background but PID 1 waited only on tinyproxy, and the health check curled only :3128. Nested Docker containers and BuildKit RUN steps resolve names only through this dnsmasq, so its death (or a failure to bind :53 at start) left a proxy reporting healthy while every session's nested DNS was down, with no exit for the restart policy to act on. The supervisor now watches both daemons: it checks every two seconds (one `sleep` fork, well under 0.1 % of a core) whether either is still running, using /proc so no signal permission is needed for the nobody-owned tinyproxy. When either exits on its own it logs which one and with what status, stops the other and exits non-zero (the daemon's status, 1 for a clean exit), so the container restarts whole. A stop request still stops both cleanly and exits with tinyproxy's status, and the trap is now set before the daemons start. The readiness probe also asks dnsmasq: busybox nslookup (an applet of the alpine 3.20 image, built with the full nslookup that exits 1 on a timeout or an error reply) queries 127.0.0.1 for sandbox-egress-health.invalid, which dnsmasq answers from its own --host-record, so the probe needs no upstream resolver. With nslookup's one-second timeout the probe stays under the 5 s health timeout after curl's 3 s. The probe stays byte-identical in the Dockerfile, compose.yml and the CLI generator; the Kubernetes readiness probe in the docs follows it and gets the same 5 s timeout instead of Kubernetes' default of 1 s. dnsmasq's cache grows from its default of 150 names to 4096: the whole fleet resolves through this one forwarder, and 150 names evicted entries long before their TTLs, sending repeat lookups upstream. 4096 entries take well under 1 MB.
The shared buildkitd image copied its binaries from moby/buildkit:buildx-stable-1, a tag that moves with every buildkit release (it last moved on 2026-09-30, to v0.33.1), so two builds of the same commit could ship different buildkit versions. The pull-through registry mirror the spawner runs for every organization defaulted to `registry:2`, so each host ran whatever that tag pointed to when it first pulled it. Pin both by version and digest, at what the tags resolve to today: moby/buildkit:v0.33.1@sha256:cec9f139… and registry:2.8.3@sha256:a3d8aaa6…. Renovate's Dockerfile manager tracks the buildkit FROM line; the mirror default lives in environment defaults no built-in manager reads, so a regex manager (validated against Renovate's schema) tracks it in compose.yml, the spawner config and the CLI. The CLI holds the default in one constant, and a parity test keeps compose.yml, the generator and the spawner config on it. `tale deploy`'s runtime preparation now accepts a source that names the mirror by tag (sources from before this pin) or by version and digest, pulls it as named, and pins the resolved digest in the bundle as before. Spawners that ran the old default recreate each organization's mirror once, while no build runs, keeping its cache volume.
Retiring an organization revoked the virtual keys minted for its sessions, but the provider credentials those keys spent through stayed in the model gateway's store: the organization's upstream key on every shared standard provider record (named tale-<org>-<provider>), and its own per-(connector, model) custom records, which carry its key and its endpoint. Nothing ever removed them, so a deleted organization's raw API keys outlived it in the gateway indefinitely. The last retirement slice now removes them right after the virtual keys: it lists every provider record, deletes the records named <org>__… with their keys, and on every other record deletes only the key with the organization's exact key name. A record or key already gone counts as removed, so a second run finds nothing to do. Every record is tried, and the step throws when the listing fails or any removal is refused, which fails the job for the queue's backoff like the rest of retirement. The process also forgets the key ids and fingerprints it remembered for the organization. The step is not held back for offline devices or the other slices, since a virtual key whose upstream key is gone reaches nothing. The sandbox spec's SBX-R13 card and the admin docs (EN/DE/FR) now say the deletion removes the provider keys too.
…odule The released legacy agent flow reuses one function of llm_gateway_admin.ts, hashVirtualKey, and its provenance pinned the whole file's hash, so any change elsewhere in the module (here the organization teardown) failed the guard although the helper the flow runs is unchanged. It is now pinned by that declaration's hash, the way the guard already pins the sections it reuses from external_turn_shared.ts and others.
yannickmonney
force-pushed
the
fix/sandbox-egress-supervision
branch
from
October 8, 2026 12:29
beb030b to
bcec988
Compare
…down An organization's own gateway records are <org>__<slug>__<model>. A record that goes on with another '_' after <org>__ belongs to an organization whose id continues with '_', and the teardown now leaves it alone instead of deleting it with the retiring organization's records.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Robustness gaps in the shared sandbox services: the egress container's dnsmasq could die silently while the healthcheck stayed green (a fleet-wide DNS outage for nested containers and BuildKit), tinyproxy fsynced every log line into an unrotated file, the BuildKit and registry-mirror helper images floated, and deleting an organization left its upstream provider keys in the model gateway.
Changes
fix(sandbox): log the egress proxy to stdout instead of a file
tinyproxy wrote its log to /var/log/tinyproxy/tinyproxy.log in the
container's writable layer, where nothing rotated it, and a busybox
tail -Fpolled the file once a second to mirror it into the containerlog. tinyproxy 1.11 also fsyncs the log after every line under its global
log mutex, so on a busy fleet proxy each Notice line cost a journal flush
while every connection thread queued behind it.
With neither LogFile nor Syslog set, tinyproxy 1.11 logs to stdout
(open_log_file(NULL) takes fileno(stdout)), which is the container log the
logging driver already rotates (10 MB x 3). Drop the LogFile directive,
the log file's touch and chown, the tail poller and the one-second wait
the supervisor kept for it; the shutdown line still reaches docker logs
because tinyproxy writes it straight to the inherited stdout. That is one
process and one file per egress container fewer.
DAC_OVERRIDE and CHOWN stay in the capability set: no entrypoint step is
known to need them any more, but the set was verified live and dropping
them waits on a live boot without them.
fix(sandbox): supervise the egress DNS forwarder and probe it
The egress entrypoint started dnsmasq in the background but PID 1 waited
only on tinyproxy, and the health check curled only :3128. Nested Docker
containers and BuildKit RUN steps resolve names only through this
dnsmasq, so its death (or a failure to bind :53 at start) left a proxy
reporting healthy while every session's nested DNS was down, with no exit
for the restart policy to act on.
The supervisor now watches both daemons: it checks every two seconds
(one
sleepfork, well under 0.1 % of a core) whether either is stillrunning, using /proc so no signal permission is needed for the
nobody-owned tinyproxy. When either exits on its own it logs which one
and with what status, stops the other and exits non-zero (the daemon's
status, 1 for a clean exit), so the container restarts whole. A stop
request still stops both cleanly and exits with tinyproxy's status, and
the trap is now set before the daemons start.
The readiness probe also asks dnsmasq: busybox nslookup (an applet of the
alpine 3.20 image, built with the full nslookup that exits 1 on a timeout
or an error reply) queries 127.0.0.1 for sandbox-egress-health.invalid,
which dnsmasq answers from its own --host-record, so the probe needs no
upstream resolver. With nslookup's one-second timeout the probe stays
under the 5 s health timeout after curl's 3 s. The probe stays
byte-identical in the Dockerfile, compose.yml and the CLI generator; the
Kubernetes readiness probe in the docs follows it and gets the same 5 s
timeout instead of Kubernetes' default of 1 s.
dnsmasq's cache grows from its default of 150 names to 4096: the whole
fleet resolves through this one forwarder, and 150 names evicted entries
long before their TTLs, sending repeat lookups upstream. 4096 entries
take well under 1 MB.
fix(sandbox): pin the buildkit and registry mirror images by digest
The shared buildkitd image copied its binaries from
moby/buildkit:buildx-stable-1, a tag that moves with every buildkit
release (it last moved on 2026-09-30, to v0.33.1), so two builds of the
same commit could ship different buildkit versions. The pull-through
registry mirror the spawner runs for every organization defaulted to
registry:2, so each host ran whatever that tag pointed to when itfirst pulled it.
Pin both by version and digest, at what the tags resolve to today:
moby/buildkit:v0.33.1@sha256:cec9f139… and registry:2.8.3@sha256:a3d8aaa6….
Renovate's Dockerfile manager tracks the buildkit FROM line; the mirror
default lives in environment defaults no built-in manager reads, so a
regex manager (validated against Renovate's schema) tracks it in
compose.yml, the spawner config and the CLI. The CLI holds the default in
one constant, and a parity test keeps compose.yml, the generator and the
spawner config on it.
tale deploy's runtime preparation now accepts a source that names themirror by tag (sources from before this pin) or by version and digest,
pulls it as named, and pins the resolved digest in the bundle as before.
Spawners that ran the old default recreate each organization's mirror
once, while no build runs, keeping its cache volume.
fix(platform): remove a deleted organization's keys from the gateway
Retiring an organization revoked the virtual keys minted for its
sessions, but the provider credentials those keys spent through stayed in
the model gateway's store: the organization's upstream key on every shared
standard provider record (named tale--), and its own
per-(connector, model) custom records, which carry its key and its
endpoint. Nothing ever removed them, so a deleted organization's raw API
keys outlived it in the gateway indefinitely.
The last retirement slice now removes them right after the virtual keys:
it lists every provider record, deletes the records named __… with
their keys, and on every other record deletes only the key with the
organization's exact key name. A record or key already gone counts as
removed, so a second run finds nothing to do. Every record is tried, and
the step throws when the listing fails or any removal is refused, which
fails the job for the queue's backoff like the rest of retirement. The
process also forgets the key ids and fingerprints it remembered for the
organization. The step is not held back for offline devices or the
other slices, since a virtual key whose upstream key is gone reaches
nothing.
The sandbox spec's SBX-R13 card and the admin docs (EN/DE/FR) now say
the deletion removes the provider keys too.
Verification
egress-proxy-config.test.ts,config.test.ts65 pass; tools/clicompose-parity.test.ts,runtime-prepare.test.ts166 pass; platformllm_gateway_admin.test.ts,workspace-cleanup.test.ts+ domain-spec guard 133 pass;@tale/platformtypecheck clean; lint:manual ok.Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).