Skip to content

httpclient, reactor: a reactor's stop disposes the pools and clients started on it, so the requests waiting on them end, and a connection handed over as it stops is closed (#610) - #625

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-610

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Fixes #610.
Touches core: src/ioxide/Reactor/Reactor.RingHost.cs and src/ioxide/Reactor/Reactor.Runner.cs, because a satellite client has no way to run on the reactor thread when it stops, before posts close: tickers end with the loop, the teardown frees only the TLS services core owns, and a Dispose called after the stop is a post the stopped reactor drops.

What was wrong

HttpClientPool.Release hands a connection straight to the first waiting request with TrySetResult, and that request yields to the reactor's post queue (ReactorSwitch.Yield) before it takes it. Posted from a completion, the yield runs in the next pass's DrainPostQ, which comes after the loop's stop check, so a stop in that same pass ended the loop with the yield still queued. The teardown's last drain then ran it on the thread pool, where it threw ObjectDisposedException. That lost two things:

  • The handed connection. It was in neither _idle nor _waiters, and SendAsync never received it, so neither Release nor Discard closed its socket or freed its TLS session and native buffers.
  • The requests queued behind it. Only a Release, a Discard, a failed open, Dispose or the sweep wakes them, and the sweep is a ticker a stopped reactor no longer runs. With nothing else in flight on the pool, they waited forever.

RingHttpClient's per-origin pools take the same path. The stop also left what the pools hold: a pool's idle connections are closed, and RingHttpClient's TlsClientContext freed, only by their Dispose, and a Dispose called after the stop is posted to a reactor that drops every post but a switch continuation.

Reproduced at main 19aec5c with #584's harness (PoolSize 1 to a live origin, 16 requests looping on the reactor, a stop at a random 100 to 300 ms): 4 of 10 trials left 15 of the 16 requests still waiting 2 s after the stop. On this branch, 0 of 20 trials left any request waiting. The tests below reproduce each part deterministically.

What changes

  • reactor: Reactor.AddStopper(Action) registers a callback that Teardown runs first, on the reactor thread, right after it sets _stopRequested: before the accepted connections are closed and the queue is submitted for the last time, and so before posts close. Each runs in a try/catch that logs [rN] stopper faulted: ..., so one that throws cannot leave the ring, the sockets and the memory unfreed. With none registered it costs a null check per stop.
  • httpclient: HttpClientPool.Start and RingHttpClient.Start register their Dispose. The stop then closes the idle connections, frees RingHttpClient's TLS context, and wakes the waiting requests. Their yields are posted before the last drain, which hands them to the thread pool, where they end with ObjectDisposedException as any switch after a stop does. SSL_CTX_free drops one reference, so a connection still holding an SSL keeps the context alive until it goes.
  • httpclient: when the yield in AcquireAsync throws, the request disposes the connection it was handed. Only that request holds it, and HttpClientConnection.Dispose is frees and a close, safe on the thread pool. Nothing of the pool's own state is touched from there: the waiters are woken by the stopper on the reactor, so no thread-pool continuation reaches the pool's Queue, which another stopped continuation (a second hand-over in the same pass, with PoolSize above 1) could be reaching at the same time.

Why first in the teardown rather than just before posts close: the stop's own closes resume handlers inline (MarkAcceptedTcpConnectionsClosed). A handler that then asks a pool not yet disposed for a request gets an idle connection, its send is queued, and the teardown's last SubmitQueued sends it to the origin. CompleteParkedOps then reports that send as -ECANCELED, whether or not the kernel ran it, so the origin serves a request whose caller was told it failed. Run first, the pool is already disposed and refuses the request before anything is sent. A test pins this.

Behaviour a caller would notice

  • A pool from HttpClientPool.Start, and a RingHttpClient with every pool it opened, is disposed by its reactor's stop. A request waiting for a connection at the stop now ends with ObjectDisposedException instead of waiting forever, and a request a handler makes while the stop runs fails at once with HttpClientException ("pool for host:port is disposed", "client is disposed") instead of going out in the ring's last submit.
  • A host that called Dispose after the stop sees no difference: that call was already a dropped post, and the client is now disposed by then anyway.
  • New public API: Reactor.AddStopper(Action). Nothing removed.

Tests

Each failure below is from reverting only that part of the fix, rebuilding, and running the test.

Http, new HttpClientStopTests:

  1. "httpclient h1: a connection handed to a waiting request as its reactor stops is closed, and the requests queued behind it end": PoolSize 1 and three requests; the first answer hands the connection to the second, and the first's continuation calls Stop in that same pass, so the hand-over at the stop is deterministic. Expected second: ObjectDisposedException, third: ObjectDisposedException, origin connections closed: 1 of 1.
    • Without the dispose in AcquireAsync: ... origin connections closed: 0 of 1.
    • Without the pool's stopper, or without the core's RunStoppers() call: second: ObjectDisposedException, third: never ended, ....
  2. "httpclient h1: a pool's idle connections are closed by its reactor's stop": without the pool's stopper, idle 2, origin connections closed: 0 of 2.
  3. "httpclient h1: a request a handler makes as its reactor stops is refused, and never reaches the origin": with RunStoppers() moved to just before _postsClosed = true, send failed: errno 125; origin requests 2, connections closed: 1 of 1 where pool for 127.0.0.1:<port> is disposed; origin requests 1, ... is expected.
  4. "ring client: its reactor's stop closes its origins' idle connections and frees its TLS context": without RingHttpClient's stopper, 200 200, origins 2; origin connections closed: 0 of 1, TLS context freed: False. The context is read through UnsafeAccessor on its private _disposed.

E2E, StopParkedTests: "stop: the stoppers run on the reactor thread before its connections close, and one that throws leaves the teardown to finish", expected on the reactor True, its connection closed False; the next one.

  • Without the RunStoppers() call: got [].
  • Without the try/catch: got [on the reactor True, its connection closed False], and the runner fails a test reactor died and no test observed it: ... a stopper that throws.
  • With RunStoppers() moved to just before _postsClosed = true: got [on the reactor True, its connection closed True; the next one].

Without the core call all four Http tests fail too. Moved late, tests 1, 2 and 4 still pass: the stoppers would still fix what the issue reports, and only the ordering tests see the difference.

Suites

Built with dotnet build -c Release ioxide.slnx, everything pinned to CPUs 0-3, 6-7, 14-31:

Suite Passed Failed Skipped
Unit 263 0 0
E2E 731 (1 new) 0 0
Http 108 (4 new) 0 0
Tls 269 0 0
Chaos 105 0 0
File 13 0 0
Pg 7 0 19 (no server)
Redis 0 0 6 (no server), 1 pending

Overlaps

git merge-tree --write-tree origin/fix-<n> HEAD exits 0 against every open PR branch from #594 to #604 and against #619 to #623.

For the reviewer

  • The try/catch in AcquireAsync surrounds only the yield a waiting request makes after it is woken: the path that finds an idle connection never reaches it, and it costs nothing unless the yield throws. The alternative without one is to take a handed connection at once, in the completer's frame, and yield only on a wake with null, which is what keeps WakeWaiters from spinning. That removes the window instead of cleaning up after it, but changes where a handed request runs, which http2, http3, reactor: a call from another thread runs on the reactor, and the reactor installs no SynchronizationContext, so library code it resumes stays there (#441) #442 chose on purpose (waiters "resume inline and yield back to the reactor, out of the code that completed them"), so this PR keeps the yield.
  • AddStopper is new public core API, named after AddTicker, and reactor-thread only like it and AddService (call it from OnStart). It takes an Action rather than an IDisposable, so a client can register something other than its Dispose. The stoppers also run on a failed start, for what OnStart registered before it threw.
  • The docs do not mention it yet; docs/learn/clients.html's rules for a custom client would be the place, beside AddService.

…started on it, so the requests waiting on them end, and a connection handed over as it stops is closed (#610)

Release hands a connection straight to the first waiting request, which yields to the reactor's post
queue before it takes it. A stop in that same pass ended the loop before the yield ran, and the
teardown's last drain resumed it on the thread pool, where the yield threw ObjectDisposedException.
The connection, in neither the idle list nor the waiter queue and never returned to SendAsync, was
never closed, and the requests queued behind it waited forever: only a Release, a Discard, a failed
open, Dispose or the sweep wakes them, and the sweep is a ticker a stopped reactor no longer runs.
The stop also left every pool's idle connections open and RingHttpClient's TLS context allocated,
since only their Dispose frees them, and a Dispose called after the stop is a post the stopped
reactor drops.

- reactor: AddStopper registers a callback that the teardown runs first, on the reactor thread,
  before the connections close and before posts close. Each runs guarded, so one that throws is
  logged and the rest of the teardown still runs.
- httpclient: HttpClientPool.Start and RingHttpClient.Start register their Dispose, so the stop
  closes the idle connections, frees the TLS context and wakes the waiting requests, which end with
  ObjectDisposedException on the thread pool. Run first, a pool also refuses a request from a
  handler the stop's closes resume, which the last submit would otherwise send to the origin while
  its caller was told it failed.
- httpclient: a request whose yield throws disposes the connection it was handed.

Tests: Http's HttpClientStopTests (4) and E2E's StopParkedTests stopper test, each failing when its
part of the fix is reverted.
@MDA2AV

MDA2AV commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Folded into #636: its Reactor.AddStopper and its tests are there, with the stoppers run at the start of the stop's first pass rather than in Teardown. The try/catch around the yield in AcquireAsync is not taken: the drain runs that yield on the reactor. #610 is closed by #636.

@MDA2AV MDA2AV closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

httpclient: a connection handed to a waiting request in the pass where its reactor stops is never closed, and the requests behind it wait forever

1 participant