sandbox: release the token only when it was acquired - #48
Merged
Conversation
The deferred Release was registered above the ErrBusy check, so a rejected request put back a token it never took and the next caller started a worker beyond pool_size. With pool_size 1, a burst of 8 requests ran 4 containers side by side. Tokens do not accumulate, so this is not a leak: Release is non-blocking and the channel holds at most pool_size of them. The excess lives inside a burst, where every rejection hands out one more slot, so the ceiling follows the load instead of the setting. The doc comment on Exec already promises no more than pool.Size() concurrent workers, so this brings the code to what is documented rather than changing the design.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Execregistersdefer semaphore.Release()before checking whetherAcquire()returnedErrBusy, so a request rejected as busy puts back atoken it never took. The next caller acquires it and starts a worker beyond
pool_size.I measured it with
pool_size: 1and a burst of 8 requests sleeping 3 seconds,sampling
docker psalongside:The sandbox and the container runtime do not matter — the defect is in
internal/sandbox, above the engine.Sequential requests never show it:
Releaseis non-blocking, so at rest thesemaphore is back to
pool_size. It takes a burst, where every rejection handsout one more slot: for K requests hitting a pool with s free tokens, up to
(K + s) / 2get in instead of s. The ceiling follows the load, not thesetting.
To see it fail without Docker, put the
deferback above theErrBusycheckand run
go test ./internal/sandbox/:The
busysubtest now checks that a rejection leaves no free tokens and thatthe request right after it is rejected too — the contract the doc comment on
Execalready promises. Theexecsubtest checks that a finished workerreturns its token.
The
Execbody is unchanged since the first public commit (8447197), soevery release is affected. I did not find this in the tracker.
Unrelated:
pool_sizehas no default and is not validated, so a configwithout the key gives
NewSemaphore(0)and a busy response to everything.