Repository navigation
fix(platform): size the gateway's provider worker pools by record kind - #4574
Conversation
|
REQUEST CHANGES — independent API-only review of head
Per-topic dispositions:
Both platform unit shards actually executed (Turbo MISS, exit 0) on merge Limits: no checkout, local tests, install, live gateway/provider, runtime benchmark, browser render, Ops read, push or merge. PR remains behind main. Merge readiness needs settled source-bound CI and independent verification of the requested proof/docs. Manager/implementation owner should route these findings; check queued CI on the next bounded pass rather than rerunning unchanged work. |
ad3e572 to
2fc235c
Compare
Every provider record the platform puts into the sandbox model gateway asked for 1,000 request workers and a 5,000-deep queue. The gateway (Bifrost core v1.11.0 in the pinned v2.2.4 image) starts a record's workers as goroutines eagerly: for every stored record at boot (Init -> prepareProvider) and again on every config or key write (UpdateProvider), and they stay parked whether or not anything calls the record. A custom connector gets one record per (organization, connector, model) plus its Anthropic-lane sibling, so parked workers grow with organizations times custom models. At 2-8 KiB of stack per goroutine a record costs about 2-8 MiB, so 64 to 256 records fill the gateway's 512 MiB limit on stacks alone, and a gateway over its limit is killed again at every boot. A shared standard record now gets 512 workers and an org-scoped custom record 64, each with a queue 16 deep per worker (the gateway refuses a queue shorter than the pool): a hundred custom records park 6,400 workers instead of 100,000. A stream holds its worker only until the upstream's response headers arrive, and a sandbox host sizes itself to at most 256 sessions, so 64 covers one organization's burst on one model; a request past the pool waits in the queue. SANDBOX_LLM_GATEWAY_PROVIDER_CONCURRENCY and SANDBOX_LLM_GATEWAY_CUSTOM_PROVIDER_CONCURRENCY override the two sizes. A value that is not a positive whole number keeps the default with one warning, since the gateway would refuse every provision with it; one above 5,000, the gateway's default connections per upstream host, is held there. The backend reads both from its .env like the other gateway settings, so no compose or CLI change is needed. The pool is part of the provision fingerprint, so a resized record is rewritten at its next provision.
The gateway's provider records now carry smaller worker pools and queues (64 workers and 1,024 queue slots on a custom record). A request that finds a record's queue full is held until there is room, as long as the gateway's client setting drop_excess_requests is off, which is its default (Bifrost core v1.11.0, tryRequest and tryStreamRequest). With the setting on, the same request is answered 503 "request dropped: queue is full" at once. The platform merged whatever the gateway's store held and never looked at the flag, so a store restored with it on, or an operator's edit, would turn a burst past 1,088 requests on one record into failed agent calls. The posture check run on every sandbox create now treats the flag as drift too, and the config write keeps it off, the same way it keeps inference behind virtual keys and content out of the request log. A store that already has it off (every gateway the platform set up) is left alone, with no extra write.
Only a provision that names a record writes its worker pool, and a session provisions just the records of its own allowed models. A record nothing provisions again (a model an organization removed or renamed, an __anthropic sibling of a lane no session rides, an inactive organization's records, a rarely used model) kept the 1,000 workers and 5,000-deep queue it was stored with, and the gateway restarts those workers at every boot. A hundred such records still park 100,000 goroutines, so the smaller pools alone did not bound the gateway's memory. Each backend process now runs one shrink pass in the background once its first provision has applied the gateway's auth posture: it lists every record (GET /api/providers) and writes each one with more workers than its kind's pool back with that pool. PUT /api/providers/:name replaces network_config whole and refuses a custom record without its custom_provider_config (Bifrost transports v2.2.4, updateProvider), so the pass sends the record's own network block, custom upstream and raw-payload switches back with it, and leaves out the blocks the PUT keeps when absent, whose secrets the listing masks. The pass must never write an old config over a newer one. A record this process has provisioned is skipped, a provision of a record waits for the pass's write to it, and each record is read again right before its write, which narrows a race with another backend process to the moment between that read and the write. A failed listing is tried again at the next provision; a record the gateway refuses keeps its pool, with a warning, until the next process.
The gateway (Bifrost core v1.11.0) starts every stored provider record's workers inside Init, before it serves HTTP. A store whose records still carry the gateway's default 1,000 workers needs about 2-8 MiB of goroutine stacks per record, so 64 to 256 such records exceed the 512 MiB limit that tale deploy (mem_limit) and the Kubernetes manifest (limits.memory) set. The gateway is then killed at every boot, and the backend, which resizes records only through the management API, never gets to run its shrink pass. Nothing told an operator how to get out of that state, and the compose comment warns against wiping the gateway volume, which holds the admin account and the minted keys with their spend. The gateway README and the environment reference (EN, DE, FR) now give the path: raise the container's memory limit until the gateway starts (docker update on a tale deploy host, which keeps the restart policy and needs no compose override, or the Deployment's limit on Kubernetes), start one sandbox session or model-endpoint request so the backend resizes the records and logs it, then restore the limit the deployment sets (512m memory, 1g memory plus swap) and restart.
The environment reference and .env.example said SANDBOX_LLM_GATEWAY_PROVIDER_CONCURRENCY sizes the built-in providers (OpenAI, Anthropic, OpenRouter). That holds for their shared records, but Claude Code through OpenRouter's Anthropic endpoint rides an org-scoped record of its own (resolveGatewayRouting names it <org>__openrouter__<model>__anthropic), and the gateway sizes that record with SANDBOX_LLM_GATEWAY_CUSTOM_PROVIDER_CONCURRENCY: 64 workers, not 512, as the pool test for that lane already pins. An operator raising the built-in value for Claude Code load on OpenRouter would see no effect. The CUSTOM row (EN, DE, FR) and the .env.example block now say that lane gets an organization-and-model provider of its own and follows the custom value.
The shrink pass gave each record's PUT the 15 s every management call gets and counted a record only on a success answer. The gateway (Bifrost transports v2.2.4, updateProvider) resolves a custom record's base URL (up to 5 s), stores the record and restarts its workers, and only then asks the upstream for its models (up to 15 s) before it answers. A record whose upstream is unreachable was therefore resized in the gateway yet logged as failed and left out of the count, and the pass printed its "resized" line only when that count was above zero: the line the recovery steps tell an operator to wait for might never come. A resize write now gets 30 s (5 s + 15 s, plus margin). One the gateway fails with a 5xx or does not answer is read back, and counts as resized when the record holds the new pool, else as unconfirmed. The pass always ends with one line, "[llm-gateway] provider worker resize finished: <n> resized, <n> refused, <n> unconfirmed", zeros included; the gateway README, the environment reference (EN, DE, FR) and the automation register name it. The pass also checks again after its read, right before the write, whether a provision of this process took the record, so a provision waits only for a write already sent, never for the read before it.
The gateway's PUT /api/providers/:name is an upsert: for a record it does not hold it calls AddProvider with the payload (Bifrost transports v2.2.4, updateProvider). The shrink pass sent its write through the management client, which sends a write the gateway's store turned away with a 5xx again after 250, 750 and 2,000 ms, blind. A record deleted in those seconds (a base-type recreate by another process, an operator's cleanup, an organization teardown) came back without keys, with 64 or 512 workers and its old base URL, and nothing deleted it again. The resize write now takes the first answer and decides itself. After a 5xx it waits the same 250, 750 and 2,000 ms, reads the record, and sends the write again only while the record is still there and still over its pool, with the config it reads then; a record gone or resized meanwhile settles the outcome. A write the gateway did not answer may still be under way there, so it is read back and never sent again.
The gateway's provider write copies the record's keys when the request arrives and stores that copy (Bifrost transports v2.2.4, updateProvider), and it resolves a custom record's base URL (up to 5 s) before it stores. A key or config another backend process writes to the same record in that window is undone by the shrink pass's write, and that process's fingerprint memo then keeps it from writing the record again until it restarts. Every fresh process rewrites each record and key it provisions once, which is also when keys rotated through the environment land, and backend-api runs two replicas on Kubernetes, so a pass started at the first provision ran inside exactly that burst, in both replicas at once. The first provision now schedules the pass two minutes plus up to three more at random later, once per process, with an unref'd timer; a pass whose listing failed is scheduled again by the next provision. The window remains for a write that lands while the gateway handles the pass's write to that record, now outside the deploy burst; the shrinkProviderPools comment and the gateway README describe it. The environment reference (EN, DE, FR), README, .env.example and the automation register say the resize comes two to five minutes after the first session or model-endpoint request.
The gateway resolves the base URL every provider write carries (Bifrost transports v2.2.4, updateProvider: ValidateExternalURL, a DNS lookup bounded at 5 s) and answers 400 "Invalid base URL: failed to resolve hostname" when the host is gone. The records whose upstream disappeared (a removed connector's, a decommissioned LAN host's) are exactly the orphans no provision rewrites, so the shrink pass of every backend process was refused on them and they kept the pool they were stored with, up to the gateway's 1,000 workers (2-8 MiB of parked stacks each). The environment reference said a lower count reaches every other provider, and the recovery steps had no remedy when the gateway still did not start within 512 MiB. A refused resize of a record that carries a base URL now logs that the record keeps its workers and names DELETE /api/providers/<name> as the remedy, since a provision sets up a record still in use again. A built-in provider's record carries no base URL and holds every organization's key, so its refusal says only that it keeps its workers. The environment reference (EN, DE, FR), the gateway README's recovery steps, .env.example and the automation register document the exception and the delete, sent from a backend container with curl and the admin password it already holds.
The automation coverage map now keeps one file per suite; this change's row moves there.
The organization teardown deletes a deleted organization's own provider records, and the shrink pass resizes oversized records by writing them back with a PUT, which the gateway answers by creating a record it does not hold. A resize whose read came before the teardown's delete and whose write came after it brought the deleted organization's record back, with no keys, after the teardown had reported success. Within a process the two are now fenced: a teardown marks each record it deletes and waits for a resize of it already in flight before deleting, and the pass never writes a marked record; a provision takes the mark off again. A teardown in another backend process is not seen; the comment on the mark says what that race leaves behind and how it goes.
2fc235c to
9a765b4
Compare
1 participant
Every provider record the platform writes into the sandbox model gateway (Bifrost) asked for 1,000 request workers and a 5,000-request queue. Bifrost starts a record's workers as goroutines as soon as it loads the record, at boot and on every write, and keeps them parked whether or not anything calls it. Custom providers get one record per organization, connector and model, plus an Anthropic-lane sibling. So the gateway's idle memory grew with organizations times custom models, against its 512 MiB limit. A gateway over that limit crash-loops at boot.
Changes
test(platform): pin the legacy flow's gateway helper, not its whole module
The released legacy agent flow reuses one function of llm_gateway_admin.ts,
hashVirtualKey, and its provenance pinned the whole file's hash, so any
change elsewhere in the module (here the provider records' worker pools)
failed the guard although the helper the flow runs is unchanged. It is now
pinned by that declaration's hash, the way the guard already pins the
sections it reuses from external_turn_shared.ts and others. The fixture is
byte-identical to the same switch on the egress-supervision branch, so the
two merge cleanly.
fix(platform): size the gateway's provider worker pools by record kind
Every provider record the platform puts into the sandbox model gateway
asked for 1,000 request workers and a 5,000-deep queue. The gateway
(Bifrost core v1.11.0 in the pinned v2.2.4 image) starts a record's
workers as goroutines eagerly: for every stored record at boot (Init ->
prepareProvider) and again on every config or key write (UpdateProvider),
and they stay parked whether or not anything calls the record. A custom
connector gets one record per (organization, connector, model) plus its
Anthropic-lane sibling, so parked workers grow with organizations times
custom models. At 2-8 KiB of stack per goroutine a record costs about
2-8 MiB, so 64 to 256 records fill the gateway's 512 MiB limit on stacks
alone, and a gateway over its limit is killed again at every boot.
A shared standard record now gets 512 workers and an org-scoped custom
record 64, each with a queue 16 deep per worker (the gateway refuses a
queue shorter than the pool): a hundred custom records park 6,400
workers instead of 100,000. A stream holds its worker only until the
upstream's response headers arrive, and a sandbox host sizes itself to
at most 256 sessions, so 64 covers one organization's burst on one
model; a request past the pool waits in the queue.
SANDBOX_LLM_GATEWAY_PROVIDER_CONCURRENCY and
SANDBOX_LLM_GATEWAY_CUSTOM_PROVIDER_CONCURRENCY override the two sizes.
A value that is not a positive whole number keeps the default with one
warning, since the gateway would refuse every provision with it; one
above 5,000, the gateway's default connections per upstream host, is
held there. The backend reads both from its .env like the other
gateway settings, so no compose or CLI change is needed. The pool is
part of the provision fingerprint, so a resized record is rewritten at
its next provision.
fix(platform): keep the gateway queueing a burst instead of dropping it
The gateway's provider records now carry smaller worker pools and
queues (64 workers and 1,024 queue slots on a custom record). A request
that finds a record's queue full is held until there is room, as long
as the gateway's client setting drop_excess_requests is off, which is
its default (Bifrost core v1.11.0, tryRequest and tryStreamRequest). With the setting on, the
same request is answered 503 "request dropped: queue is full" at once.
The platform merged whatever the gateway's store held and never looked
at the flag, so a store restored with it on, or an operator's edit,
would turn a burst past 1,088 requests on one record into failed
agent calls.
The posture check run on every sandbox create now treats the flag as
drift too, and the config write keeps it off, the same way it keeps
inference behind virtual keys and content out of the request log. A
store that already has it off (every gateway the platform set up) is
left alone, with no extra write.
Review follow-ups
An adversarial review found that only a record a later provision names gets resized. Records nothing provisions again would keep 1,000 workers forever, and a gateway already crash-looping at boot had no recovery path. A second review pass then hardened the resize pass. The fixes, one commit each:
fix(platform): resize gateway provider records no provision rewrites
Only a provision that names a record writes its worker pool, and a
session provisions just the records of its own allowed models. A record
nothing provisions again (a model an organization removed or renamed, an
__anthropic sibling of a lane no session rides, an inactive
organization's records, a rarely used model) kept the 1,000 workers and
5,000-deep queue it was stored with, and the gateway restarts those
workers at every boot. A hundred such records still park 100,000
goroutines, so the smaller pools alone did not bound the gateway's
memory.
Each backend process now runs one shrink pass in the background once its
first provision has applied the gateway's auth posture: it lists every
record (GET /api/providers) and writes each one with more workers than
its kind's pool back with that pool. PUT /api/providers/:name replaces
network_config whole and refuses a custom record without its
custom_provider_config (Bifrost transports v2.2.4, updateProvider), so
the pass sends the record's own network block, custom upstream and
raw-payload switches back with it, and leaves out the blocks the PUT
keeps when absent, whose secrets the listing masks.
The pass must never write an old config over a newer one. A record this
process has provisioned is skipped, a provision of a record waits for
the pass's write to it, and each record is read again right before its
write, which narrows a race with another backend process to the moment
between that read and the write. A failed listing is tried again at the
next provision; a record the gateway refuses keeps its pool, with a
warning, until the next process.
docs(sandbox): recover a model gateway killed while it boots
The gateway (Bifrost core v1.11.0) starts every stored provider
record's workers inside Init, before it serves HTTP. A store whose
records still carry the gateway's default 1,000 workers needs about
2-8 MiB of goroutine stacks per record, so 64 to 256 such records
exceed the 512 MiB limit that tale deploy (mem_limit) and the
Kubernetes manifest (limits.memory) set. The gateway is then killed at
every boot, and the backend, which resizes records only through the
management API, never gets to run its shrink pass. Nothing told an
operator how to get out of that state, and the compose comment warns
against wiping the gateway volume, which holds the admin account and
the minted keys with their spend.
The gateway README and the environment reference (EN, DE, FR) now give
the path: raise the container's memory limit until the gateway starts
(docker update on a tale deploy host, which keeps the restart policy
and needs no compose override, or the Deployment's limit on
Kubernetes), start one sandbox session or model-endpoint request so the
backend resizes the records and logs it, then restore the limit the
deployment sets (512m memory, 1g memory plus swap) and restart.
docs(sandbox): size the Anthropic lane of a built-in provider as custom
The environment reference and .env.example said
SANDBOX_LLM_GATEWAY_PROVIDER_CONCURRENCY sizes the built-in providers
(OpenAI, Anthropic, OpenRouter). That holds for their shared records,
but Claude Code through OpenRouter's Anthropic endpoint rides an
org-scoped record of its own (resolveGatewayRouting names it
openrouter__anthropic), and the gateway sizes that
record with SANDBOX_LLM_GATEWAY_CUSTOM_PROVIDER_CONCURRENCY: 64
workers, not 512, as the pool test for that lane already pins. An
operator raising the built-in value for Claude Code load on OpenRouter
would see no effect.
The CUSTOM row (EN, DE, FR) and the .env.example block now say that
lane gets an organization-and-model provider of its own and follows
the custom value.
fix(platform): confirm a gateway resize whose answer came too late
The shrink pass gave each record's PUT the 15 s every management call
gets and counted a record only on a success answer. The gateway (Bifrost
transports v2.2.4, updateProvider) resolves a custom record's base URL
(up to 5 s), stores the record and restarts its workers, and only then
asks the upstream for its models (up to 15 s) before it answers. A
record whose upstream is unreachable was therefore resized in the
gateway yet logged as failed and left out of the count, and the pass
printed its "resized" line only when that count was above zero: the
line the recovery steps tell an operator to wait for might never come.
A resize write now gets 30 s (5 s + 15 s, plus margin). One the gateway
fails with a 5xx or does not answer is read back, and counts as resized
when the record holds the new pool, else as unconfirmed. The pass
always ends with one line, "[llm-gateway] provider worker resize
finished: resized, refused, unconfirmed", zeros included;
the gateway README, the environment reference (EN, DE, FR) and the
automation register name it. The pass also checks again after its read,
right before the write, whether a provision of this process took the
record, so a provision waits only for a write already sent, never for
the read before it.
fix(platform): never let a gateway resize recreate a deleted record
The gateway's PUT /api/providers/:name is an upsert: for a record it
does not hold it calls AddProvider with the payload (Bifrost transports
v2.2.4, updateProvider). The shrink pass sent its write through the
management client, which sends a write the gateway's store turned away
with a 5xx again after 250, 750 and 2,000 ms, blind. A record deleted in
those seconds (a base-type recreate by another process, an operator's
cleanup, an organization teardown) came back without keys, with 64 or
512 workers and its old base URL, and nothing deleted it again.
The resize write now takes the first answer and decides itself. After a
5xx it waits the same 250, 750 and 2,000 ms, reads the record, and
sends the write again only while the record is still there and still
over its pool, with the config it reads then; a record gone or resized
meanwhile settles the outcome. A write the gateway did not answer may
still be under way there, so it is read back and never sent again.
fix(platform): start the gateway shrink pass after the startup burst
The gateway's provider write copies the record's keys when the request
arrives and stores that copy (Bifrost transports v2.2.4,
updateProvider), and it resolves a custom record's base URL (up to 5 s)
before it stores. A key or config another backend process writes to the
same record in that window is undone by the shrink pass's write, and
that process's fingerprint memo then keeps it from writing the record
again until it restarts. Every fresh process rewrites each record and
key it provisions once, which is also when keys rotated through the
environment land, and backend-api runs two replicas on Kubernetes, so a
pass started at the first provision ran inside exactly that burst, in
both replicas at once.
The first provision now schedules the pass two minutes plus up to three
more at random later, once per process, with an unref'd timer; a pass
whose listing failed is scheduled again by the next provision. The
window remains for a write that lands while the gateway handles the
pass's write to that record, now outside the deploy burst; the
shrinkProviderPools comment and the gateway README describe it. The
environment reference (EN, DE, FR), README, .env.example and the
automation register say the resize comes two to five minutes after the
first session or model-endpoint request.
fix(platform): say how to clear a gateway record no resize can reach
The gateway resolves the base URL every provider write carries (Bifrost
transports v2.2.4, updateProvider: ValidateExternalURL, a DNS lookup
bounded at 5 s) and answers 400 "Invalid base URL: failed to resolve
hostname" when the host is gone. The records whose upstream disappeared
(a removed connector's, a decommissioned LAN host's) are exactly the
orphans no provision rewrites, so the shrink pass of every backend
process was refused on them and they kept the pool they were stored
with, up to the gateway's 1,000 workers (2-8 MiB of parked stacks each).
The environment reference said a lower count reaches every other
provider, and the recovery steps had no remedy when the gateway still
did not start within 512 MiB.
A refused resize of a record that carries a base URL now logs that the
record keeps its workers and names DELETE /api/providers/ as the
remedy, since a provision sets up a record still in use again. A
built-in provider's record carries no base URL and holds every
organization's key, so its refusal says only that it keeps its workers.
The environment reference (EN, DE, FR), the gateway README's recovery
steps, .env.example and the automation register document the exception
and the delete, sent from a backend container with curl and the admin
password it already holds.
Verification
maximhq/bifrost:v2.2.4(core v1.11.0): workers start up front at Init and on UpdateProvider; the PUT refusesconcurrencyof 0 or abovebuffer_size; when the queue is full anddrop_excess_requestsis off, a request waits instead of getting a 503.llm_gateway_admin.test.ts102 pass. That includes the defaults, both overrides, invalid values warning once, the 5,000 cap, a resized record being rewritten through the fingerprint, and a store with dropping turned on being rewritten. The posture tests fail with that fix removed.@tale/platformtypecheck and lint are clean, and commitlint passes.Risks
The once-per-process resize pass runs 2–5 minutes after the first provision. It can race with a key or config write from another backend replica to the same record. That replica's memo then keeps the record as the resize wrote it until it next restarts. A lost key comes back at the next provision.
When fix(sandbox): supervise egress DNS and clear deleted orgs' gateway keys #4545 lands, its organization teardown and this resize pass need a fence, so a resize can't recreate a record the teardown just deleted. That will be wired in at rebase time.
A record is only rewritten when an organization that uses it next provisions (a sandbox session or a model-endpoint call). A gateway that is already crash-looping from memory needs its memory limit raised once so those rewrites can land.
Shrinking a live record goes through Bifrost's UpdateProvider, which cancels any queued requests beyond the new queue length (1,024 on a custom record). This happens once, and only under very heavy load.
Part of the sandbox footprint and robustness pass (#4533, #4538 and siblings).
After the follow-ups:
llm_gateway_admin.test.ts120 pass,gateway_provisioning.test.ts36 pass, the legacy-flow guard passes,@tale/platformtypecheck passes, oxlint on the touched files is clean, the docs locale/structure checks pass and lint:manual is ok.