Repository navigation
Conversation
…started on it, so the requests waiting on them end, and a connection handed over as it stops is closed (#610) Release hands a connection straight to the first waiting request, which yields to the reactor's post queue before it takes it. A stop in that same pass ended the loop before the yield ran, and the teardown's last drain resumed it on the thread pool, where the yield threw ObjectDisposedException. The connection, in neither the idle list nor the waiter queue and never returned to SendAsync, was never closed, and the requests queued behind it waited forever: only a Release, a Discard, a failed open, Dispose or the sweep wakes them, and the sweep is a ticker a stopped reactor no longer runs. The stop also left every pool's idle connections open and RingHttpClient's TLS context allocated, since only their Dispose frees them, and a Dispose called after the stop is a post the stopped reactor drops. - reactor: AddStopper registers a callback that the teardown runs first, on the reactor thread, before the connections close and before posts close. Each runs guarded, so one that throws is logged and the rest of the teardown still runs. - httpclient: HttpClientPool.Start and RingHttpClient.Start register their Dispose, so the stop closes the idle connections, frees the TLS context and wakes the waiting requests, which end with ObjectDisposedException on the thread pool. Run first, a pool also refuses a request from a handler the stop's closes resume, which the last submit would otherwise send to the origin while its caller was told it failed. - httpclient: a request whose yield throws disposes the connection it was handed. Tests: Http's HttpClientStopTests (4) and E2E's StopParkedTests stopper test, each failing when its part of the fix is reverted.
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #610.
Touches core: src/ioxide/Reactor/Reactor.RingHost.cs and src/ioxide/Reactor/Reactor.Runner.cs, because a satellite client has no way to run on the reactor thread when it stops, before posts close: tickers end with the loop, the teardown frees only the TLS services core owns, and a
Disposecalled after the stop is a post the stopped reactor drops.What was wrong
HttpClientPool.Releasehands a connection straight to the first waiting request withTrySetResult, and that request yields to the reactor's post queue (ReactorSwitch.Yield) before it takes it. Posted from a completion, the yield runs in the next pass'sDrainPostQ, which comes after the loop's stop check, so a stop in that same pass ended the loop with the yield still queued. The teardown's last drain then ran it on the thread pool, where it threwObjectDisposedException. That lost two things:_idlenor_waiters, andSendAsyncnever received it, so neitherReleasenorDiscardclosed its socket or freed its TLS session and native buffers.Release, aDiscard, a failed open,Disposeor the sweep wakes them, and the sweep is a ticker a stopped reactor no longer runs. With nothing else in flight on the pool, they waited forever.RingHttpClient's per-origin pools take the same path. The stop also left what the pools hold: a pool's idle connections are closed, andRingHttpClient'sTlsClientContextfreed, only by theirDispose, and aDisposecalled after the stop is posted to a reactor that drops every post but a switch continuation.Reproduced at main 19aec5c with #584's harness (
PoolSize1 to a live origin, 16 requests looping on the reactor, a stop at a random 100 to 300 ms): 4 of 10 trials left 15 of the 16 requests still waiting 2 s after the stop. On this branch, 0 of 20 trials left any request waiting. The tests below reproduce each part deterministically.What changes
Reactor.AddStopper(Action)registers a callback thatTeardownruns first, on the reactor thread, right after it sets_stopRequested: before the accepted connections are closed and the queue is submitted for the last time, and so before posts close. Each runs in a try/catch that logs[rN] stopper faulted: ..., so one that throws cannot leave the ring, the sockets and the memory unfreed. With none registered it costs a null check per stop.HttpClientPool.StartandRingHttpClient.Startregister theirDispose. The stop then closes the idle connections, freesRingHttpClient's TLS context, and wakes the waiting requests. Their yields are posted before the last drain, which hands them to the thread pool, where they end withObjectDisposedExceptionas any switch after a stop does.SSL_CTX_freedrops one reference, so a connection still holding an SSL keeps the context alive until it goes.AcquireAsyncthrows, the request disposes the connection it was handed. Only that request holds it, andHttpClientConnection.Disposeis frees and aclose, safe on the thread pool. Nothing of the pool's own state is touched from there: the waiters are woken by the stopper on the reactor, so no thread-pool continuation reaches the pool'sQueue, which another stopped continuation (a second hand-over in the same pass, withPoolSizeabove 1) could be reaching at the same time.Why first in the teardown rather than just before posts close: the stop's own closes resume handlers inline (
MarkAcceptedTcpConnectionsClosed). A handler that then asks a pool not yet disposed for a request gets an idle connection, its send is queued, and the teardown's lastSubmitQueuedsends it to the origin.CompleteParkedOpsthen reports that send as-ECANCELED, whether or not the kernel ran it, so the origin serves a request whose caller was told it failed. Run first, the pool is already disposed and refuses the request before anything is sent. A test pins this.Behaviour a caller would notice
HttpClientPool.Start, and aRingHttpClientwith every pool it opened, is disposed by its reactor's stop. A request waiting for a connection at the stop now ends withObjectDisposedExceptioninstead of waiting forever, and a request a handler makes while the stop runs fails at once withHttpClientException("pool for host:port is disposed", "client is disposed") instead of going out in the ring's last submit.Disposeafter the stop sees no difference: that call was already a dropped post, and the client is now disposed by then anyway.Reactor.AddStopper(Action). Nothing removed.Tests
Each failure below is from reverting only that part of the fix, rebuilding, and running the test.
Http, new
HttpClientStopTests:PoolSize1 and three requests; the first answer hands the connection to the second, and the first's continuation callsStopin that same pass, so the hand-over at the stop is deterministic. Expectedsecond: ObjectDisposedException, third: ObjectDisposedException, origin connections closed: 1 of 1.AcquireAsync:... origin connections closed: 0 of 1.RunStoppers()call:second: ObjectDisposedException, third: never ended, ....idle 2, origin connections closed: 0 of 2.RunStoppers()moved to just before_postsClosed = true,send failed: errno 125; origin requests 2, connections closed: 1 of 1wherepool for 127.0.0.1:<port> is disposed; origin requests 1, ...is expected.RingHttpClient's stopper,200 200, origins 2; origin connections closed: 0 of 1, TLS context freed: False. The context is read throughUnsafeAccessoron its private_disposed.E2E,
StopParkedTests: "stop: the stoppers run on the reactor thread before its connections close, and one that throws leaves the teardown to finish", expectedon the reactor True, its connection closed False; the next one.RunStoppers()call:got [].got [on the reactor True, its connection closed False], and the runner failsa test reactor died and no test observed it: ... a stopper that throws.RunStoppers()moved to just before_postsClosed = true:got [on the reactor True, its connection closed True; the next one].Without the core call all four Http tests fail too. Moved late, tests 1, 2 and 4 still pass: the stoppers would still fix what the issue reports, and only the ordering tests see the difference.
Suites
Built with
dotnet build -c Release ioxide.slnx, everything pinned to CPUs 0-3, 6-7, 14-31:Overlaps
git merge-tree --write-tree origin/fix-<n> HEADexits 0 against every open PR branch from #594 to #604 and against #619 to #623.HttpClientPool.cs(its backoff retry inAcquireAsyncandOpenOneAsync), and merges cleanly: both register their tests in Http'sProgram.cs, and this one registers its own a few lines away from httpclient: a request waiting out a pool's reconnect backoff ends when its reactor stops, rather than waiting forever (#584) #596's so the two don't touch. They fit together: at the stop the stopper sets_disposedbeforeCompleteParkedOpscancels httpclient: a request waiting out a pool's reconnect backoff ends when its reactor stops, rather than waiting forever (#584) #596's backoff wait, which then takes its_disposedbranch, and the requests it would wake have already been woken. Built and run together (a scratch merge of the two branches), Http passes 111 of 111, httpclient: a request waiting out a pool's reconnect backoff ends when its reactor stops, rather than waiting forever (#584) #596's three backoff tests among them.Reactor.Currentat the end ofTeardown. The stoppers run at its start, whereOnReactorThreadstill holds, so a stopper'sDisposestill runs inline. Merges cleanly.Teardownand merge cleanly. reactor: a ring op submitted from a completion the stop gives fails as one handed over after the stop does, rather than never completing (#609) #621 makes an op submitted during the stop's cancellations fail rather than never complete, which also ends an open whose connect is retried on the origin's next address during the stop.RingHttpClient.Disposefrees the context under a connect in flight). This PR frees that context at the stop too, but after the stoppers no open can still reachSSL_new: it runs only once a connect succeeds, and a stopping reactor delivers no successful connect (CompleteParkedOpsreports every parked one as-ECANCELED, and a resolve's switch back throws). So freeing it at the stop opens no way to httpclient: RingHttpClient.Dispose frees its TLS context while a pool may still be connecting, and that connect then calls SSL_new on the freed context #611's use after free, and the two PRs change different files.tests/Ioxide.Tests.E2E/Program.cs; this PR adds nothing to that.For the reviewer
AcquireAsyncsurrounds only the yield a waiting request makes after it is woken: the path that finds an idle connection never reaches it, and it costs nothing unless the yield throws. The alternative without one is to take a handed connection at once, in the completer's frame, and yield only on a wake withnull, which is what keepsWakeWaitersfrom spinning. That removes the window instead of cleaning up after it, but changes where a handed request runs, which http2, http3, reactor: a call from another thread runs on the reactor, and the reactor installs no SynchronizationContext, so library code it resumes stays there (#441) #442 chose on purpose (waiters "resume inline and yield back to the reactor, out of the code that completed them"), so this PR keeps the yield.AddStopperis new public core API, named afterAddTicker, and reactor-thread only like it andAddService(call it fromOnStart). It takes anActionrather than anIDisposable, so a client can register something other than itsDispose. The stoppers also run on a failed start, for whatOnStartregistered before it threw.docs/learn/clients.html's rules for a custom client would be the place, besideAddService.