Repository navigation
Closed
Conversation
…n its reactor stops, rather than waiting forever (#584) After a failed connect the pool holds new connects back for 200 to 400 ms, and a request that found no connection inside that window queued with nothing in flight. Only the pool's sweep, a reactor ticker, could then open the next connection or fail the request at its acquire deadline, and a stopped reactor runs no ticker, so the request waited forever. RingHttpClient's per-origin pools share the acquire path, and had the same gap. Such a request now starts the retry itself: an open that waits out the rest of the backoff on the ring, one IRingHost.SubmitTimeout on a RingOpSource the pool holds, and then connects. It counts in _opening while it waits, so one runs at a time and the sweep's refill stands aside. The stop cancels the wait, and the retry then connects nothing and reports no failure, but wakes the waiters, which end with ObjectDisposedException as they already did behind a cancelled connect. A retry behind the backoff now fires when the backoff ends rather than on the next 250 ms tick, and ConnectionCount counts it while it waits. Tests, in Http's HttpClientBackoffTests: a request queued behind the backoff ends with ObjectDisposedException when its reactor stops, through HttpClientPool and through RingHttpClient; on main both are still waiting 2 s after the stop. A request waiting 2 s on an origin whose every open fails sees 3 to 12 opens; with the retry connecting at once it saw 22495.
This was referenced Nov 14, 2026
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #584.
What was wrong
After a failed connect,
HttpClientPoolholds new connects back for 200 to 400 ms. A request that found no connection inside that window queued with nothing in flight: only the pool's sweep, a reactor ticker, would open the next connection or fail the request at its acquire deadline. A stopped reactor runs no ticker. Its teardown completes only what is on the ring (it cancels the parked client ops) and the switches still in its post queue. So once the reactor stopped, that request waited forever, its own acquire deadline included. A request queued behind a connect or a response in flight already ended at the stop, withObjectDisposedException("The reactor has stopped."), because the stop cancels that op and its failure wakes the waiters.RingHttpClient's per-origin pools share the acquire path and had the same gap.What changes
src/clients/ioxide.httpclient/Http1/HttpClientPool.csonly.IRingHost.SubmitTimeouton oneRingOpSourcethe pool holds), then connects. The request now waits behind something the stop completes._openingwhile it waits, so only one runs at a time, and the sweep's refill stands aside as it does for any open in flight. No new counter or flag. One SQE per failed connect; no syscall, noTask.Delay, no thread-pool hop.-ETIME), the retry connects nothing, logs nothing and leaves the backoff alone. It gives back its_livecount and wakes the waiters, which then end as they do behind a cancelled connect: each one's yield runs from the stopped reactor's queue on the thread pool and throwsObjectDisposedException. A pool disposed during the wait connects nothing either.What a caller would notice
ObjectDisposedException("The reactor has stopped."), as a request queued behind a connect already did.ConnectionCountcounts the retry while it waits, so a pool backing off with a request queued reports 1 where main reported 0.Tests
New
tests/Ioxide.Tests.Http/HttpClientBackoffTests.cs:httpclient h1: a request waiting out the reconnect backoff ends when its reactor stops: a warm pool of one to a dead port. The request is made on the reactor; 50 ms on, inside the backoff, the test thread stops the reactor and joins it. The request must end withObjectDisposedException.ring client: a request waiting out the reconnect backoff ends when its reactor stops: the same throughRingHttpClient's on-demand pool.httpclient h1: a request waiting on an origin whose every open fails retries at the backoff's pace: an origin that accepts each connection and closes it before the TLS handshake, so every open fails after a connect the origin can count. The request must fail on its 2 s acquire timeout after 3 to 12 opens.Each part reverted on its own:
expected [ObjectDisposedException: The reactor has stopped. Object name: 'Reactor'.], got [still waiting 2 s after its reactor stopped].22495 opens in the request's 2 s, where its backoff allows 6 to 11(25849 on a second run).got [still waiting 2 s after its reactor stopped]. That connect is submitted from inside the teardown's cancel loop and takes a slot the loop has already passed, so nothing completes it.The existing
dead origin fails inside the acquire budget, no connect stormstill passes.Suites
Built and run pinned to CPUs 0-3, 6-7 and 14-31:
For the reviewer
Reactorhook: teardown walks no registered services (it frees only its TLS services, through an internal list), and httpclient sees no core internals. Keeping a request that waits out the backoff behind something on the ring needs none.RingHttpClient's TLS context, since onlyDisposedoes that and a stopped reactor drops it. That needs the teardown hook too.