Repository navigation
Conversation
…ns on until they have been let go or a second has passed, and then frees what nothing holds (#634) The stop had one path per thing a handler could still hold once the loop had exited: posts, parked ops, TLS services, flushes from other threads, and then connections, buffer returns and the ops a cancel's completion submits, each a bug of its own. Now the stop has one path. On its first pass the reactor shuts every listener and connection down, as the sweep ends a connection it condemns, evicts its QUIC connections, closes its UDP sockets, and cancels every op in its table. The loop then runs on as usual, posts, switches, returns and completions included, so each connection lets go through its own completions and its handler's release, as it would while running. It ends once no connection is left unrecycled and the op table is empty, or a second after the stop began. Teardown then frees everything in one place. A connection a handler still holds past the second stays allocated, its descriptor shut down but not closed, and so does the shared receive slab its queued items and slices point into. From the stop on, an accept is closed rather than served or armed again, a paused accept is not re-armed on a number the stop closed, an op submitted while the stop drains is cancelled within a tick, and a datagram on a socket the stop closed is dropped. A buffer return that finds its queue full after the stop is dropped instead of spinning on a queue nothing drains. Removed, dead with the drain: the teardown's MarkAcceptedTcpConnectionsClosed, SubmitQueued, CloseAcceptedTcpSockets, ReleaseAcceptedTcpConnections, ReturnReleasedTcpConnections, CompleteParkedOps and CancelAndWait, the IORING_REGISTER_SYNC_CANCEL constant and io_uring_sync_cancel_reg, Ring.SqPending, ENOENT, and ReturnToPool, now inlined into Recycle.
…rted on the reactor, so their idle connections close and the requests waiting on them fail with ObjectDisposedException (#634) Reactor.AddStopper(Action) registers a callback the stop runs once, on the reactor thread, at the start of its first pass, before the listeners and connections close: one list, since the service registry keeps one entry per type and a reactor can run several pools of one type. A stopper that throws is reported and the next one runs. A loop that faulted runs them in Teardown, through the same CloseAll. The first pass then drains the posts that its stoppers and closes made, since they come after that pass's own drain and the loop would otherwise park, or end, before running them. HttpClientPool.Start, RingHttpClient.Start, PgPool.Start and AssetReader.CreatePool register their Dispose. The pools close their idle connections, RingHttpClient frees its TLS context, and the requests waiting on a connection, and any made after, fail with ObjectDisposedException, where httpclient threw HttpClientException. With the loop still running, a connection handed to a waiting request as the reactor stops is taken by that request's yield and let go when it ends. PgPool gets a Dispose: the connections close, the slow path's waiters fail with ObjectDisposedException, a connection whose open completes after is closed, and the sweep opens no more. RingPool gets one: idle resources are disposed, and so is each one given back after, and a waiting or later rent fails with ObjectDisposedException. A TLS client handshake on a freed TlsClientContext is refused with ObjectDisposedException and closes its socket, rather than calling SSL_new on freed memory: a RingHttpClient disposed by the stop frees its context while a connect of its can still complete in the same pass.
…keeps its descriptor, and four tests written around the stop closing descriptors go (#634) The stop never closes a descriptor a handler still holds, so what those tests guarded against, a release or a send reaching a socket that took a closed number, can no longer happen. Removed: StopParkedTests "a send the last pass left queued ends before its connection's number is freed, and is not sent" and "a send a handler woken by the stop queues on another connection ends before that one's number is freed", and TcpTimeoutTests "a handler that lets go after its reactor stopped" and "a ShutdownWrite after its reactor stopped", with their ReleaseAfterStop helper. In their place, the deadline test ("a handler that never lets go") checks that after the stop the held connection's number still names its own socket: the stop neither closed it nor let it be handed out again.
…a reset, nothing queued is sent, and every send in flight ends at once, zero-copy ones included (#634) On the stop's first pass each connection in the table is shut down as before, and then disconnected with connect(fd, AF_UNSPEC), the kernel's tcp_disconnect: the peer gets a reset, the send queue is dropped, and every send on the socket completes at once. A zero-copy send's notification comes only once the kernel lets go of its pages, which after a shutdown alone waits for a client that may never read, and held its handler's flush, the connection and the stop until the deadline. A connection the sweep had already condemned is disconnected too. The descriptor stays the connection's own until it is let go, as before.
…throws ObjectDisposedException, so the stop cancels each op once and the ticker that cancelled them again each tick goes (#634) A client op or timeout submitted on the reactor thread once Stop() has been called throws ObjectDisposedException, as one handed over from another thread already did after the teardown; a handover now throws from the stop on too. One handed over before the stop and taken off the queue after it completes with -ECANCELED rather than reaching the kernel past the stop's one cancel. So the first pass's cancel is the last, and nothing is left for a per-tick cancel to find: a wait that is armed again after the stop cancelled it, as a timer loop does, now ends at once instead of holding the stop to its deadline. ioxide.redis.stackexchange: a connect the stopped reactor refuses closes the socket it opened. StackExchange.Redis reconnects through the tunnel for as long as its multiplexer lives, and each refused attempt left a socket open. The docs of Stop, IRingHost and ReadFileAsync say so. The #609 test now expects the wait submitted again from the stop's cancel to throw, and the hand-over test the second request's send to be refused.
…d that a connection still open then stays allocated, its descriptor included, until the process exits (#634) The teardown after a fault runs the same close as a stop's first pass, its stoppers, listeners and connection aborts, and frees what nothing holds. It does not run the loop on, so no completion lets a connection still open go: it stays allocated with its descriptor and the receive memory it can reach.
…multiplexer has on its ring (#634) The transport registers no stopper: each one always has a receive on the ring, so the stop's cancel reaches every one, and the transport closes through its own receive-failure path, on the reactor thread. Needs a redis server, so it skips without one. Without the stop's cancel the server still sees both connections after the stop.
This was referenced Nov 15, 2026
…s any socket, so no entry reaches whatever takes a number it closed, and the stop's docs say a kTLS connection is shut down rather than aborted (#634) The pass's own drains can queue entries on a UDP socket or a listener, a QUIC send or an accept armed again, and the pass then closes both. Left queued past the closes, those entries were submitted after them, against a closed number (EBADF, "udp send error: -9") or against whatever socket another thread opened on it. They are now submitted first, until the kernel takes no more, since a failed prep ends a submit early. A connection's descriptor is not closed by the stop, so its entries needed none of this. connect(AF_UNSPEC) fails with EOPNOTSUPP on a kTLS socket, which then ends by the shutdown alone: its client sees a FIN, and what it already queued may still go out. It never sends zero-copy, so the shutdown is enough to end its sends. Stop()'s docs and the comment on CloseTcpConnections say so.
…rees its buffers when its deadline is refused (#634) ConnectAsync submitted its open deadline before the try, and from the stop on that submit throws ObjectDisposedException on the reactor thread too, so a pool's sweep or a handler the stop wakes that opened a connection left its socket and its buffers behind. The submit is now inside the try, whose catch disposes the connection.
…nnection, so their stops end at once rather than at the deadline, and a stop test that repeated another goes (#634) Their TCP handler returned without DecRef, so the probe's connection was held for good and each of those reactors' stops, about 300 in E2E, ran the full second and exercised the deadline rather than the teardown. The RingTimer loop test failed on the same two reverts as the test for a ring op parked at the stop, which covers its wait-again too.
This was referenced Nov 15, 2026
Merged
MDA2AV
added a commit
that referenced
this pull request
Nov 15, 2026
All twelve csproj Version elements, 0.26.563 to 0.27.637. Carries the 24 PRs from #564 to #636. Package validation against 0.26.563 finds breaks in ioxide alone: Ring.Dispose(bool), Ring.SqPending, Native.IORING_REGISTER_SYNC_CANCEL and Native.io_uring_sync_cancel_reg are removed. The other ten published packages are clean. Suites on this tree: Unit 306, E2E 797, Http 113, Chaos 105, File 14, Tls 269 (kTLS included), Pg 10 (19 skipped, no server here), Redis 1 (7 skipped and 1 pending, no server here).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #634.
Fixes #610.
Fixes #584.
Fixes #611.
Touches core:
src/ioxide/Reactor/Reactor.Runner.cs,Reactor.cs,Reactor.RingHost.cs,Loop/Reactor.Loop.cs,Loop/Reactor.Drainers.cs,Loop/Reactor.Loop.Incremental.cs,Transport/Tcp/Reactor.Tcp.cs,Transport/Udp/Reactor.Udp.cs,src/ioxide/Client/RingPool.cs,src/ioxide/Native/Native.IoUring.csandsrc/ioxide/io_uring/Ring.cs, because the stop is the reactor's, and the docs ofsrc/ioxide/IRingHost.csandConnection/Tcp/TcpConnection.FileRead.cs. Clients:ioxide.httpclient,ioxide.pg,ioxide.file,ioxide.redis.stackexchange.The stop sequence
Stop()sets the flag and wakes the loop, as before. From that call on, a ring op submitted on the reactor throwsObjectDisposedException, on the reactor thread as from any other. Then, on the reactor's first pass after it:Reactor.AddStopper), and the stop runs them once, on the reactor thread, before anything closes: everyHttpClientPoolandRingHttpClient, everyPgPool, and everyAssetReaderpool. Each does what itsDisposedoes: it closes its idle connections, fails the requests waiting on it withObjectDisposedException, and frees what it holds (RingHttpClient's TLS context, the readers' buffers). The same pass runs what that wakes.connect(fd, AF_UNSPEC), the kernel'stcp_disconnect. Its client gets a reset (an idle one sees the shutdown's FIN just before it), anything still queued to send is dropped, and every send in flight completes at once, zero-copy notifications included. A kTLS socket refuses the disconnect (EOPNOTSUPP) and ends by the shutdown alone: its client sees a FIN, and what it had already queued may still go out. It never sends zero-copy, so the shutdown is enough to end its sends.-ECANCELEDrather than reaching the kernel; anything submitted sinceStop()has already thrown. The ticker that cancelled ops again each tick is gone.The loop then runs on as normal (posts, switches, buffer returns, recycles, flushes, tickers, completions) until no accepted connection is left unrecycled and no op is pending, or the deadline. Teardown then frees everything in one place, in the order it did: the wake fd, the timespecs, the buffer rings, the ring, the pooled connections, the shared receive slab when no connection holds it, the UDP memory, the posts and ops still queued (their switches fail on the thread pool), the TLS services.
What a handler still holds by then stays allocated: its connection, with its descriptor, which the stop never closes, and in shared mode the receive slab its queued items and its reader's slices point into. What it calls on the reactor after that fails as it already did: a ring op or a switch throws
ObjectDisposedException, a read or a flush reports the connection closed, a buffer return or a release is dropped.Only what a
StartorCreatePoolstarts on a reactor registers a stopper:PgConnection, aRingFile, aRingSocket) stays its own to dispose; step 5 ends whatever it has in flight.New state: the stopper list, and a disposed flag on
PgPoolandRingPool, which had noDispose;_tcpLiveand the loop's stop check are as in the first version. On a request's path the PR adds one read of the stop flag (a volatile bool) per ring op a client submits on the reactor thread, and nothing per TCP read, write or flush.On a fault
A fault or a failed start reaches Teardown without the drain. Teardown runs the same close first, steps 2 to 5 (after a stop, nothing is left for it but disconnecting again what is still held), and frees what nothing holds, but drains nothing: a connection still open stays allocated until the process exits, its descriptor and the receive memory it can reach included, since no completion will let it go.
OnFault's docs say so.What was removed, and why it is dead
MarkAcceptedTcpConnectionsClosedTcpSweepClose.SubmitQueuedandRing.SqPendingSqPending. A connection's descriptor is not closed by the stop at all:Recyclecloses it, after the kernel has completed everything on it.CloseAcceptedTcpSocketsRecyclecloses each connection's descriptor once it is let go, and one still held keeps its number, so nothing its holder does later reaches a socket that took the number.ReleaseAcceptedTcpConnectionsReturnReleasedTcpConnectionsRecycleduring the drain, its buffer ring unregistered there, into the pool the teardown frees.CompleteParkedOps,CancelAndWait,IORING_REGISTER_SYNC_CANCEL,io_uring_sync_cancel_reg,ENOENTCancelClientOps). Nothing waits on the kernel synchronously.CancelClientOpstickerStop()throws, and one handed over before it is completed as cancelled when taken off the queue.ReturnToPoolRecycle): inlined.Kept, because they still guard what comes after the drain:
DrainStoppedPosts,DrainStoppedOps,HandOff'sObjectDisposedException(now fromStop()on), the switches'PostsClosedcheck,ScheduleContinuationandTlsEncryptingPipeWriter's use of it (#580), andUdpSendTo's direct send, since the first pass sends QUIC's farewells and closes the UDP sockets right after.Not taken from #625: its try/catch around the yield in
HttpClientPool.AcquireAsync. A connection handed to a waiting request as the reactor stops reaches it: every post made on the reactor is drained before the loop's stop check, the first pass's included, so the yield runs on the reactor, the request takes the connection, and its refused send lets it go and closes it.Across
src/: 294 lines added, 197 removed (core 203 and 170, clients 91 and 27). The first version was 105 added, 164 removed; most of what was added since is the clients' stop hooks (PgPoolandRingPoolhad noDispose) and docs.Behaviour a host would notice
HttpClientPool,RingHttpClient,PgPooland theAssetReaderpool. A request waiting on one fails withObjectDisposedException, a pool's idle connections close, andRingHttpClient's TLS context is freed. Before, a request waiting behind a pool's reconnect backoff (httpclient: a request waiting behind HttpClientPool's reconnect backoff is never ended once its reactor stops #584), or queued behind a connection handed over in the stop's pass (httpclient: a connection handed to a waiting request in the pass where its reactor stops is never closed, and the requests behind it wait forever #610), waited forever, and the idle connections and the TLS context were never freed.HttpClientPoolorRingHttpClientthrowsObjectDisposedException, where it threwHttpClientException(pool for host:port is disposed,client is disposed). A handler that catchesHttpClientExceptionto answer a 502 no longer catches a request made on a disposed pool.Stop()on throwsObjectDisposedException, on the reactor thread as from another, so a loop that waits again after a cancelled wait ends at once. On main one handed over from another thread threw only once the teardown had ended, and one submitted on the reactor from a completion the stop gave was never completed.TcpConnection.ReadFileAsynccan throw it too, betweenStop()and the first pass, and its docs say so. APgConnection.ConnectAsyncrefused that way closes its socket and frees its buffers.TlsClientContextthrowsObjectDisposedExceptionand closes its socket, where it calledSSL_newon freed memory: aRingHttpClientthe stop disposes frees its context while a connect of its can still complete in the same pass.Reactor.AddStopper(Action);PgPoolisIDisposable;RingPool<T>isIDisposable, and a resource given back to a disposed pool is disposed.Runreturns once every connection has been let go and every ring op has completed, a pass or two for handlers the stop wakes. A handler busy elsewhere holds it up to the deadline. Before,Runreturned at once.DecRefpost, a TLS flush handed to the reactor, an httpclient request's yield. Before, a switch's continuation went to the thread pool and threw there, and any other post was dropped.RecvSlotsxRecvBufferSize, 128 MiB by default) is then kept as well; before, it was freed under that connection's reads.OnDatagram.Native.IORING_REGISTER_SYNC_CANCEL,Native.io_uring_sync_cancel_reg,Ring.SqPending. Nothing in the repo used them but the code removed here.The deadline
A fixed second (
StopDrainMs), with no new option and noStopparameter. A handler the stop wakes lets go within a pass or two, so the deadline only bounds one busy elsewhere, and a second stays inside the joins hosts already make: Kestrel's listener joins each reactor for 5 s, the test harness for 2 s. The loop's wait is bounded by the ticker, so the deadline is seen within 250 ms of passing.Tests
Each was run with its part reverted and fails with the message shown; restored, it passes.
New this round, E2E
Core/StopDrainTests.cs:connect(fd, AF_UNSPEC)the handler's flush never returnedexpected [on the reactor True, its connection closed False; the next one], got []got [on the reactor True, its connection closed True; the next one]got [on the reactor True, its connection closed False], and the reactor died ofa stopper that throwsDrainRemoteOpsa wait handed over before the stop, and taken after its cancel, never endedthe reply queued in the pass that stopped the reactor was never sentChanged this round, E2E:
the handler's wait parked at the stop, or the one after it, never endedclose(fd)after the first pass's shutdown)after the stop fd 55 names nothing, not the socket:[20827467] its handler still holdsHandOffrefusing only once the teardown has ended, as on mainexpected ObjectDisposedException (Reactor), but nothing was thrownThe stop now waits about a millisecond for the zero-copy response: 0 to 1 ms over 10 rounds, where the test allows 125 ms.
In the harness, the datagram and QUIC hosts' TCP handler now lets go of the readiness probe's connection. Held, it made about 300 of the UDP, QUIC and h3 tests' stops run the full second, so that half of E2E took about 13 min against 7 on main, and those tests exercised the deadline rather than the teardown.
New, Http
HttpClientStopTests.cs, from #625 and #596:expected [second: ObjectDisposedException, third: ObjectDisposedException, origin connections closed: 1 of 1], got [second: ObjectDisposedException, third: never ended, origin connections closed: 1 of 1]got [second: answered 200, third: ObjectDisposedException, origin connections closed: 1 of 1]got [idle 2, origin connections closed: 0 of 2]got [ObjectDisposedException: Cannot access a disposed object. Object name: 'ioxide.Reactor'.; ...]: refused by the ring rather than by the pool. On the stoppers' own commit, before the refusal:got [answered 200; origin requests 2, connections closed: 0 of 1]got [200 200, origins 2; origin connections closed: 0 of 1, TLS context freed: False]HttpClientExceptioninPoolForexpected [ObjectDisposedException], got [HttpClientException]still waiting 2 s after its reactor stoppedObjectDisposedException: The reactor has stopped. Object name: 'Reactor'.: the request ended on the thread pool, after the teardownHttpClientExceptionfor a disposed poolHttpClientException: pool for 127.0.0.1:38601 is disposedChanged, Http
HttpClientConnectTests, "a connect its reactor's stop cancels, and the one to the next address it goes on to, both end with the stop": unchanged, but what ends the second connect is now the refusal, not the per-tick cancel. Without the refusal:the connect is still waiting 3 s after its reactor was stopped.New, Pg
PgStopTests.cs, against fake servers, so they run without postgres:expected [waiting: ObjectDisposedException, connections closed: 1], got [waiting: still waiting, connections closed: 0]got [accepted 1, closed 0, connections 1]got [accepted 4, closed 4, connections 0]tryinConnectAsync, as on mainexpected [ObjectDisposedException, sockets left open: 0], got [ObjectDisposedException, sockets left open: 8]New, File
FileTests.cs:got [idle freed: False, waiting: still waiting, given back after the stop freed: False, rent after: still waiting]RingPool.RentAsync's disposed checkgot [..., rent after: still waiting]RingPool.Return's disposed branchgot [..., given back after the stop freed: False, ...]New, Redis
StackExchangeTransportTests.cs:expected [ObjectDisposedException, sockets left open: 0], got [ObjectDisposedException, sockets left open: 8]expected [connections before the stop 2, after it 0], got [connections before the stop 2, after it 2]Removed, because what they guarded can no longer happen once the stop never closes a held descriptor:
getpeernameno longer tells which socket it is; the deadline test's new assertion checks the number by its inode instead.ReleaseAfterStophelper.From the first version, unchanged and still failing as shown with their part reverted, E2E
Core/StopDrainTests.cs:the stop ended while a handler still held its connectionthe stop never ended, with a handler that never lets gothe stop freed the receive memory a connection its handler still holds points intothe receive memory the read points into was freed at the stop, under a connection its handler still holdsthe stop ended with a handler still holding its connectionthe receive memory the read points into was freed at the stop, under a connection its handler still holds0 of the 32 buffers given back during the stop were takena connect during the stop was takenthe stopped reactor armed an accept on its closed listener's number and took the connection of the socket that took it_udpFds[i] = -1inCloseUdpFds8 datagrams reached OnDatagram after the stop closed their socketAnd Unit
StoppedReturnTests, from #619: a buffer return that finds its queue full waits while the reactor runs, and leaves when it stops (shared ring, incremental). Without the drop in the spin:a return waiting on the full queue still spins after the stop, where nothing will drain it.Looped: E2E StopDrainTests and StopParkedTests, 3 rounds on the final head, 78 of 78; Http HttpClientStopTests and HttpClientConnectTests, 5 rounds at 5fc919b, 75 of 75.
Suites
Built and run pinned to CPUs 0-3, 6-7 and 14-31, one suite at a time, on 00ff024:
tlsmodule loaded, so the kTLS tests ran.CI on 00ff024, x64 and arm64 alike, with its postgres and redis sidecars: Unit 265, E2E 736 (7 skipped: the QUIC and h3 tests that need a validating client), Http 113, Pg 29, Redis 8 (1 pending), Tls 269, File 14, Chaos 105, none failed.
Bench
Playground/Tcp/Raw, 2 reactors pinned to CPUs 14-15,wrk -t8 -c64on CPUs 16-27, keep-alive, 2 s warm-up then 10 s, origin/main and this branch alternating, 5 pairs:-0.6% req/s and +0.6% CPU per request, inside the box's 5% noise (two pairs swap order). Nothing on a TCP read, write or flush changed; a client's ring op reads the stop flag once.
Overlaps
git merge-tree --write-tree origin/<branch> HEADfor each.Reactor.AddStopperand its tests are folded in here, with the stoppers run at the start of the stop's first pass rather than in Teardown, and the expectation for the request handed the connection changed: it now takes the connection during the drain and its send is refused, so the connection is let go and closed. Its try/catch around the yield is not taken (see above). It conflicts with this branch inHttpClientPool.cs,RingHttpClient.csandReactor.RingHost.cs.SSL_newinTlsClientStream, is the one here, line for line, so this closes httpclient: RingHttpClient.Dispose frees its TLS context while a pool may still be connecting, and that connect then calls SSL_new on the freed context #611 too.