Skip to content

reactor: a stop closes every connection and cancels every ring op, runs on until they have been let go or a second has passed, and then frees what nothing holds (#634) - #636

Merged
MDA2AV merged 10 commits into
mainfrom
fix-634
Nov 15, 2026

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Fixes #634.
Fixes #610.
Fixes #584.
Fixes #611.

Touches core: src/ioxide/Reactor/Reactor.Runner.cs, Reactor.cs, Reactor.RingHost.cs, Loop/Reactor.Loop.cs, Loop/Reactor.Drainers.cs, Loop/Reactor.Loop.Incremental.cs, Transport/Tcp/Reactor.Tcp.cs, Transport/Udp/Reactor.Udp.cs, src/ioxide/Client/RingPool.cs, src/ioxide/Native/Native.IoUring.cs and src/ioxide/io_uring/Ring.cs, because the stop is the reactor's, and the docs of src/ioxide/IRingHost.cs and Connection/Tcp/TcpConnection.FileRead.cs. Clients: ioxide.httpclient, ioxide.pg, ioxide.file, ioxide.redis.stackexchange.

The stop sequence

Stop() sets the flag and wakes the loop, as before. From that call on, a ring op submitted on the reactor throws ObjectDisposedException, on the reactor thread as from any other. Then, on the reactor's first pass after it:

  1. What the pass has queued is submitted, while every socket its entries name is still its own. The pass's drains can queue a QUIC send or an accept armed again, and left queued past the closes below, such an entry would be submitted against a closed number, or against whatever socket another thread opened on it. It submits until the kernel takes no more, since a failed prep ends a submit early.
  2. The clients started on the reactor stop. Each registered a stopper when it started (Reactor.AddStopper), and the stop runs them once, on the reactor thread, before anything closes: every HttpClientPool and RingHttpClient, every PgPool, and every AssetReader pool. Each does what its Dispose does: it closes its idle connections, fails the requests waiting on it with ObjectDisposedException, and frees what it holds (RingHttpClient's TLS context, the readers' buffers). The same pass runs what that wakes.
  3. The listeners close, so new connections are refused.
  4. Every accepted TCP connection is aborted: shut down as the sweep ends one it condemns, then disconnected with connect(fd, AF_UNSPEC), the kernel's tcp_disconnect. Its client gets a reset (an idle one sees the shutdown's FIN just before it), anything still queued to send is dropped, and every send in flight completes at once, zero-copy notifications included. A kTLS socket refuses the disconnect (EOPNOTSUPP) and ends by the shutdown alone: its client sees a FIN, and what it had already queued may still go out. It never sends zero-copy, so the shutdown is enough to end its sends.
  5. QUIC connections get their CONNECTION_CLOSE, then the UDP sockets close, as before.
  6. Every pending ring op is cancelled, once. An op handed over from another thread before the stop and taken off the queue after it completes with -ECANCELED rather than reaching the kernel; anything submitted since Stop() has already thrown. The ticker that cancelled ops again each tick is gone.

The loop then runs on as normal (posts, switches, buffer returns, recycles, flushes, tickers, completions) until no accepted connection is left unrecycled and no op is pending, or the deadline. Teardown then frees everything in one place, in the order it did: the wake fd, the timespecs, the buffer rings, the ring, the pooled connections, the shared receive slab when no connection holds it, the UDP memory, the posts and ops still queued (their switches fail on the thread pool), the TLS services.

What a handler still holds by then stays allocated: its connection, with its descriptor, which the stop never closes, and in shared mode the receive slab its queued items and its reader's slices point into. What it calls on the reactor after that fails as it already did: a ring op or a switch throws ObjectDisposedException, a read or a flush reports the connection closed, a buffer return or a release is dropped.

Only what a Start or CreatePool starts on a reactor registers a stopper:

  • ioxide.redis.stackexchange registers none. Each transport always has its receive in flight on the ring, so step 5 reaches every one: the cancelled receive closes the transport through its own failure path, on the reactor thread, which frees its socket and buffer and tells StackExchange.Redis. Its reconnects through the tunnel are refused at once, and a refused one no longer leaves its socket open. A stopper per transport would grow the list with every reconnect, and one per tunnel would need a set of live transports.
  • What a handler opens itself (a PgConnection, a RingFile, a RingSocket) stays its own to dispose; step 5 ends whatever it has in flight.

New state: the stopper list, and a disposed flag on PgPool and RingPool, which had no Dispose; _tcpLive and the loop's stop check are as in the first version. On a request's path the PR adds one read of the stop flag (a volatile bool) per ring op a client submits on the reactor thread, and nothing per TCP read, write or flush.

On a fault

A fault or a failed start reaches Teardown without the drain. Teardown runs the same close first, steps 2 to 5 (after a stop, nothing is left for it but disconnecting again what is still held), and frees what nothing holds, but drains nothing: a connection still open stays allocated until the process exits, its descriptor and the receive memory it can reach included, since no completion will let it go. OnFault's docs say so.

What was removed, and why it is dead

Removed Why it is dead
MarkAcceptedTcpConnectionsClosed The first pass marks every connection closed through TcpSweepClose.
SubmitQueued and Ring.SqPending Step 1 keeps their rule (#531) for the listeners and UDP sockets the pass closes, looping on the count the kernel took instead of on SqPending. A connection's descriptor is not closed by the stop at all: Recycle closes it, after the kernel has completed everything on it.
CloseAcceptedTcpSockets Recycle closes each connection's descriptor once it is let go, and one still held keeps its number, so nothing its holder does later reaches a socket that took the number.
ReleaseAcceptedTcpConnections A flush the kernel has ends through its own send completion while the loop runs, and the abort makes it end at once, zero-copy notifications included.
ReturnReleasedTcpConnections A let-go connection recycles through Recycle during the drain, its buffer ring unregistered there, into the pool the teardown frees.
CompleteParkedOps, CancelAndWait, IORING_REGISTER_SYNC_CANCEL, io_uring_sync_cancel_reg, ENOENT With the ring up, a parked op ends through its own CQE after an async cancel (CancelClientOps). Nothing waits on the kernel synchronously.
The per-tick CancelClientOps ticker Nothing reaches the op table after the first pass: an op submitted since Stop() throws, and one handed over before it is completed as cancelled when taken off the queue.
ReturnToPool One caller left (Recycle): inlined.

Kept, because they still guard what comes after the drain: DrainStoppedPosts, DrainStoppedOps, HandOff's ObjectDisposedException (now from Stop() on), the switches' PostsClosed check, ScheduleContinuation and TlsEncryptingPipeWriter's use of it (#580), and UdpSendTo's direct send, since the first pass sends QUIC's farewells and closes the UDP sockets right after.

Not taken from #625: its try/catch around the yield in HttpClientPool.AcquireAsync. A connection handed to a waiting request as the reactor stops reaches it: every post made on the reactor is drained before the loop's stop check, the first pass's included, so the yield runs on the reactor, the request takes the connection, and its refused send lets it go and closes it.

Across src/: 294 lines added, 197 removed (core 203 and 170, clients 91 and 27). The first version was 105 added, 164 removed; most of what was added since is the clients' stop hooks (PgPool and RingPool had no Dispose) and docs.

Behaviour a host would notice

  • Every client started on the reactor is disposed at the stop, before anything closes: HttpClientPool, RingHttpClient, PgPool and the AssetReader pool. A request waiting on one fails with ObjectDisposedException, a pool's idle connections close, and RingHttpClient's TLS context is freed. Before, a request waiting behind a pool's reconnect backoff (httpclient: a request waiting behind HttpClientPool's reconnect backoff is never ended once its reactor stops #584), or queued behind a connection handed over in the stop's pass (httpclient: a connection handed to a waiting request in the pass where its reactor stops is never closed, and the requests behind it wait forever #610), waited forever, and the idle connections and the TLS context were never freed.
  • A disposed HttpClientPool or RingHttpClient throws ObjectDisposedException, where it threw HttpClientException (pool for host:port is disposed, client is disposed). A handler that catches HttpClientException to answer a 502 no longer catches a request made on a disposed pool.
  • Clients get a reset. The stop aborts every plain TCP connection it finds, so its client sees a reset (an idle one gets the shutdown's FIN just before it), and nothing queued is sent. A kTLS connection is only shut down: its client sees a FIN, and what it queued may still go out. On main the stop submitted the SQ before closing the sockets, so queued bytes went out, and the close then sent a FIN, or a reset over unread bytes.
  • A ring op submitted from Stop() on throws ObjectDisposedException, on the reactor thread as from another, so a loop that waits again after a cancelled wait ends at once. On main one handed over from another thread threw only once the teardown had ended, and one submitted on the reactor from a completion the stop gave was never completed. TcpConnection.ReadFileAsync can throw it too, between Stop() and the first pass, and its docs say so. A PgConnection.ConnectAsync refused that way closes its socket and frees its buffers.
  • A TLS client handshake on a freed TlsClientContext throws ObjectDisposedException and closes its socket, where it called SSL_new on freed memory: a RingHttpClient the stop disposes frees its context while a connect of its can still complete in the same pass.
  • New public API: Reactor.AddStopper(Action); PgPool is IDisposable; RingPool<T> is IDisposable, and a resource given back to a disposed pool is disposed.
  • A stop waits for its handlers, about a second at most. Run returns once every connection has been let go and every ring op has completed, a pass or two for handlers the stop wakes. A handler busy elsewhere holds it up to the deadline. Before, Run returned at once.
  • Posts and switches made during the drain run on the reactor, as while running, so a handler finishing on another thread is served: Kestrel's pipe continuations and its DecRef post, a TLS flush handed to the reactor, an httpclient request's yield. Before, a switch's continuation went to the thread pool and threw there, and any other post was dropped.
  • Tickers keep running during the drain: pool sweeps, TLS handshake deadlines, the TCP sweep.
  • A connection held past the deadline keeps its descriptor until the process exits, where the stop closed it before, and the number could then go to another socket under a handler still using it. Its memory leaked before too. In shared mode the receive slab (RecvSlots x RecvBufferSize, 128 MiB by default) is then kept as well; before, it was freed under that connection's reads.
  • From the stop on an accept the kernel completed is closed, no accept is armed again on a listener's number, and a datagram reaching a UDP socket the stop closed is dropped rather than handed to OnDatagram.
  • A buffer return from another thread that finds its queue full after the stop is dropped, where it spun forever.
  • On a fault, nothing is drained: the clients stop, the listeners close and the connections are aborted, but a connection still open, its descriptor included, stays allocated until the process exits, and with any of them live so does the shared receive slab. Before, the fault path freed the ones whose handler had let go.
  • Removed public API: Native.IORING_REGISTER_SYNC_CANCEL, Native.io_uring_sync_cancel_reg, Ring.SqPending. Nothing in the repo used them but the code removed here.

The deadline

A fixed second (StopDrainMs), with no new option and no Stop parameter. A handler the stop wakes lets go within a pass or two, so the deadline only bounds one busy elsewhere, and a second stays inside the joins hosts already make: Kestrel's listener joins each reactor for 5 s, the test harness for 2 s. The loop's wait is bounded by the ticker, so the deadline is seen within 250 ms of passing.

Tests

Each was run with its part reverted and fails with the message shown; restored, it passes.

New this round, E2E Core/StopDrainTests.cs:

Test Part reverted Fails with
a zero-copy response its client never reads ends at the stop: the flush returns, the handler lets go, the stop ends at once and the write slab is freed the disconnect, connect(fd, AF_UNSPEC) the handler's flush never returned
the stoppers run on the reactor thread before its connections close, and one that throws leaves the rest to run running the stoppers expected [on the reactor True, its connection closed False; the next one], got []
the stoppers run after the connections close got [on the reactor True, its connection closed True; the next one]
the try/catch around each stopper got [on the reactor True, its connection closed False], and the reactor died of a stopper that throws
a ring op handed over from another thread before the stop, and taken after its cancel, ends cancelled completing it as cancelled in DrainRemoteOps a wait handed over before the stop, and taken after its cancel, never ended
a datagram queued in the pass that stops the reactor is sent before the stop closes its socket the first pass's submit the reply queued in the pass that stopped the reactor was never sent

Changed this round, E2E:

Test Change Part reverted Fails with
a ring op parked at the stop ends at once, and one its completion submits again throws ObjectDisposedException (was "... and the one its completion submits again ends too") the second wait now throws the refusal on the reactor thread the handler's wait parked at the stop, or the one after it, never ended
the first pass's cancel the same message
a handler that never lets go ends the stop at its deadline, ... (shared ring, incremental) two assertions added: the held connection's number still names its own socket after the stop, and a ring op handed over from another thread during the drain throws descriptors closed at the stop (close(fd) after the first pass's shutdown) after the stop fd 55 names nothing, not the socket:[20827467] its handler still holds
HandOff refusing only once the teardown has ended, as on main expected ObjectDisposedException (Reactor), but nothing was thrown

The stop now waits about a millisecond for the zero-copy response: 0 to 1 ms over 10 rounds, where the test allows 125 ms.

In the harness, the datagram and QUIC hosts' TCP handler now lets go of the readiness probe's connection. Held, it made about 300 of the UDP, QUIC and h3 tests' stops run the full second, so that half of E2E took about 13 min against 7 on main, and those tests exercised the deadline rather than the teardown.

New, Http HttpClientStopTests.cs, from #625 and #596:

Test Part reverted Fails with
a connection handed to a waiting request as its reactor stops is let go, and the requests queued behind it end the pool's stopper expected [second: ObjectDisposedException, third: ObjectDisposedException, origin connections closed: 1 of 1], got [second: ObjectDisposedException, third: never ended, origin connections closed: 1 of 1]
the refusal on the reactor thread got [second: answered 200, third: ObjectDisposedException, origin connections closed: 1 of 1]
a pool's idle connections are closed by its reactor's stop the pool's stopper got [idle 2, origin connections closed: 0 of 2]
a request a handler makes as its reactor stops is refused, and never reaches the origin the pool's stopper got [ObjectDisposedException: Cannot access a disposed object. Object name: 'ioxide.Reactor'.; ...]: refused by the ring rather than by the pool. On the stoppers' own commit, before the refusal: got [answered 200; origin requests 2, connections closed: 0 of 1]
ring client: its reactor's stop closes its origins' idle connections and frees its TLS context the client's stopper got [200 200, origins 2; origin connections closed: 0 of 1, TLS context freed: False]
ring client: a request made once it is disposed fails with ObjectDisposedException HttpClientException in PoolFor expected [ObjectDisposedException], got [HttpClientException]
httpclient h1, and ring client: a request waiting out the reconnect backoff ends when its reactor stops (#584) the stopper still waiting 2 s after its reactor stopped
the first pass's drain of what the stoppers posted ObjectDisposedException: The reactor has stopped. Object name: 'Reactor'.: the request ended on the thread pool, after the teardown
HttpClientException for a disposed pool HttpClientException: pool for 127.0.0.1:38601 is disposed
tls client: a connect that completes after its TLS context was freed is refused, and closes its socket the disposed-context check the test process dies of a segmentation fault

Changed, Http HttpClientConnectTests, "a connect its reactor's stop cancels, and the one to the next address it goes on to, both end with the stop": unchanged, but what ends the second connect is now the refusal, not the per-tick cancel. Without the refusal: the connect is still waiting 3 s after its reactor was stopped.

New, Pg PgStopTests.cs, against fake servers, so they run without postgres:

Test Part reverted Fails with
a reactor's stop closes each of its pools' connections, and fails the queries waiting for one with ObjectDisposedException the pool's stopper expected [waiting: ObjectDisposedException, connections closed: 1], got [waiting: still waiting, connections closed: 0]
a disposed pool closes a connection whose open completes after, and opens no more the disposed check after an open got [accepted 1, closed 0, connections 1]
the sweep's disposed check got [accepted 4, closed 4, connections 0]
a connection opened on a stopping reactor fails with ObjectDisposedException and leaves no socket open its deadline submitted before the try in ConnectAsync, as on main expected [ObjectDisposedException, sockets left open: 0], got [ObjectDisposedException, sockets left open: 8]

New, File FileTests.cs:

Test Part reverted Fails with
a reactor's stop disposes its reader pools: an idle reader is freed, a renter waiting and a rent made after fail with ObjectDisposedException, and a reader given back after is freed the pool's stopper got [idle freed: False, waiting: still waiting, given back after the stop freed: False, rent after: still waiting]
RingPool.RentAsync's disposed check got [..., rent after: still waiting]
RingPool.Return's disposed branch got [..., given back after the stop freed: False, ...]

New, Redis StackExchangeTransportTests.cs:

Test Part reverted Fails with
a connect through the tunnel of a stopped reactor is refused with ObjectDisposedException and leaves no socket open (runs without a server) disposing the socket of a refused connect expected [ObjectDisposedException, sockets left open: 0], got [ObjectDisposedException, sockets left open: 8]
a reactor's stop closes the connections its multiplexer has on the ring (needs a server) the first pass's cancel expected [connections before the stop 2, after it 0], got [connections before the stop 2, after it 2]

Removed, because what they guarded can no longer happen once the stop never closes a held descriptor:

  • StopParkedTests, "a send the last pass left queued ends before its connection's number is freed, and is not sent" and "a send a handler woken by the stop queues on another connection ends before that one's number is freed". After the abort a connection's number names an unconnected socket, so getpeername no longer tells which socket it is; the deadline test's new assertion checks the number by its inode instead.
  • TcpTimeoutTests, "a handler that lets go after its reactor stopped ..." and "a ShutdownWrite after its reactor stopped ...", with their ReleaseAfterStop helper.

From the first version, unchanged and still failing as shown with their part reverted, E2E Core/StopDrainTests.cs:

Test Part reverted Fails with
a handler holding its connection on another thread is waited for, and only then is everything freed (shared ring, incremental) the drain (the first pass ends the loop) the stop ended while a handler still held its connection
a handler that never lets go ends the stop at its deadline, with its connection and the receive memory it can reach left allocated, and what it calls after fails fast (shared ring, incremental) the deadline the stop never ended, with a handler that never lets go
the slab kept while a connection is held (freed unconditionally) the stop freed the receive memory a connection its handler still holds points into
reads from another thread during the stop and after it get the bytes queued before it, intact, from receive memory still allocated (open at the stop; its client closed before the stop) the slab kept the receive memory the read points into was freed at the stop, under a connection its handler still holds
the drain the stop ended with a handler still holding its connection
what a pipe reader took before the stop and still holds stays intact, in receive memory still allocated the slab kept the receive memory the read points into was freed at the stop, under a connection its handler still holds
buffers a handler gives back from another thread during the stop are taken the drain 0 of the 32 buffers given back during the stop were taken
from the stop on its listener refuses connections, while a handler still holds one listeners closed at the first pass (moved to Teardown) a connect during the stop was taken
no accept is armed again on the number of a listener the stop closed (by its accept's last completion) the accept's stop check the stopped reactor armed an accept on its closed listener's number and took the connection of the socket that took it
no accept is armed again on the number of a listener the stop closed (by the sweep, as an accept paused before the stop) clearing the paused accepts the same message
a datagram that reaches a socket the stop closed is dropped, not handed to OnDatagram _udpFds[i] = -1 in CloseUdpFds 8 datagrams reached OnDatagram after the stop closed their socket

And Unit StoppedReturnTests, from #619: a buffer return that finds its queue full waits while the reactor runs, and leaves when it stops (shared ring, incremental). Without the drop in the spin: a return waiting on the full queue still spins after the stop, where nothing will drain it.

Looped: E2E StopDrainTests and StopParkedTests, 3 rounds on the final head, 78 of 78; Http HttpClientStopTests and HttpClientConnectTests, 5 rounds at 5fc919b, 75 of 75.

Suites

Built and run pinned to CPUs 0-3, 6-7 and 14-31, one suite at a time, on 00ff024:

  • Unit: 265 passed, 0 failed, 0 skipped.
  • E2E: 743 passed, 0 failed, 0 skipped, in 11 min (16.5 min at 5fc919b, before the harness change).
  • Chaos: 105 passed, 0 failed, 0 skipped.
  • Tls: 269 passed, 0 failed, 0 skipped, with the tls module loaded, so the kTLS tests ran.
  • Http: 113 passed, 0 failed, 0 skipped.
  • File: 14 passed, 0 failed, 0 skipped.
  • Pg: 10 passed, 19 skipped without a server (the 3 new ones need none). At 5fc919b, against postgres 18 in a container: 28 passed, 0 failed, 0 skipped.
  • Redis: 1 passed, 7 skipped, 1 pending without a server. At 5fc919b, against redis 7 in a container: 8 passed, 0 failed, 0 skipped, 1 pending.

CI on 00ff024, x64 and arm64 alike, with its postgres and redis sidecars: Unit 265, E2E 736 (7 skipped: the QUIC and h3 tests that need a validating client), Http 113, Pg 29, Redis 8 (1 pending), Tls 269, File 14, Chaos 105, none failed.

Bench

Playground/Tcp/Raw, 2 reactors pinned to CPUs 14-15, wrk -t8 -c64 on CPUs 16-27, keep-alive, 2 s warm-up then 10 s, origin/main and this branch alternating, 5 pairs:

req/s (5 runs) mean server CPU per request
origin/main 652,388 / 652,634 / 680,565 / 644,802 / 669,968 660,071 3.032 us
this branch 647,600 / 650,570 / 648,675 / 672,305 / 660,397 655,909 3.051 us

-0.6% req/s and +0.6% CPU per request, inside the box's 5% noise (two pairs swap order). Nothing on a TCP read, write or flush changed; a client's ring op reads the stop flag once.

Overlaps

git merge-tree --write-tree origin/<branch> HEAD for each.

…ns on until they have been let go or a second has passed, and then frees what nothing holds (#634)

The stop had one path per thing a handler could still hold once the loop had exited: posts, parked ops, TLS services, flushes from other threads, and then connections, buffer returns and the ops a cancel's completion submits, each a bug of its own. Now the stop has one path.

On its first pass the reactor shuts every listener and connection down, as the sweep ends a connection it condemns, evicts its QUIC connections, closes its UDP sockets, and cancels every op in its table. The loop then runs on as usual, posts, switches, returns and completions included, so each connection lets go through its own completions and its handler's release, as it would while running. It ends once no connection is left unrecycled and the op table is empty, or a second after the stop began. Teardown then frees everything in one place. A connection a handler still holds past the second stays allocated, its descriptor shut down but not closed, and so does the shared receive slab its queued items and slices point into.

From the stop on, an accept is closed rather than served or armed again, a paused accept is not re-armed on a number the stop closed, an op submitted while the stop drains is cancelled within a tick, and a datagram on a socket the stop closed is dropped. A buffer return that finds its queue full after the stop is dropped instead of spinning on a queue nothing drains.

Removed, dead with the drain: the teardown's MarkAcceptedTcpConnectionsClosed, SubmitQueued, CloseAcceptedTcpSockets, ReleaseAcceptedTcpConnections, ReturnReleasedTcpConnections, CompleteParkedOps and CancelAndWait, the IORING_REGISTER_SYNC_CANCEL constant and io_uring_sync_cancel_reg, Ring.SqPending, ENOENT, and ReturnToPool, now inlined into Recycle.
…rted on the reactor, so their idle connections close and the requests waiting on them fail with ObjectDisposedException (#634)

Reactor.AddStopper(Action) registers a callback the stop runs once, on the reactor thread, at the start of its first pass, before the listeners and connections close: one list, since the service registry keeps one entry per type and a reactor can run several pools of one type. A stopper that throws is reported and the next one runs. A loop that faulted runs them in Teardown, through the same CloseAll. The first pass then drains the posts that its stoppers and closes made, since they come after that pass's own drain and the loop would otherwise park, or end, before running them.

HttpClientPool.Start, RingHttpClient.Start, PgPool.Start and AssetReader.CreatePool register their Dispose. The pools close their idle connections, RingHttpClient frees its TLS context, and the requests waiting on a connection, and any made after, fail with ObjectDisposedException, where httpclient threw HttpClientException. With the loop still running, a connection handed to a waiting request as the reactor stops is taken by that request's yield and let go when it ends.

PgPool gets a Dispose: the connections close, the slow path's waiters fail with ObjectDisposedException, a connection whose open completes after is closed, and the sweep opens no more. RingPool gets one: idle resources are disposed, and so is each one given back after, and a waiting or later rent fails with ObjectDisposedException.

A TLS client handshake on a freed TlsClientContext is refused with ObjectDisposedException and closes its socket, rather than calling SSL_new on freed memory: a RingHttpClient disposed by the stop frees its context while a connect of its can still complete in the same pass.
…keeps its descriptor, and four tests written around the stop closing descriptors go (#634)

The stop never closes a descriptor a handler still holds, so what those tests guarded against, a release or a send reaching a socket that took a closed number, can no longer happen. Removed: StopParkedTests "a send the last pass left queued ends before its connection's number is freed, and is not sent" and "a send a handler woken by the stop queues on another connection ends before that one's number is freed", and TcpTimeoutTests "a handler that lets go after its reactor stopped" and "a ShutdownWrite after its reactor stopped", with their ReleaseAfterStop helper.

In their place, the deadline test ("a handler that never lets go") checks that after the stop the held connection's number still names its own socket: the stop neither closed it nor let it be handed out again.
…a reset, nothing queued is sent, and every send in flight ends at once, zero-copy ones included (#634)

On the stop's first pass each connection in the table is shut down as before, and then disconnected with connect(fd, AF_UNSPEC), the kernel's tcp_disconnect: the peer gets a reset, the send queue is dropped, and every send on the socket completes at once. A zero-copy send's notification comes only once the kernel lets go of its pages, which after a shutdown alone waits for a client that may never read, and held its handler's flush, the connection and the stop until the deadline.

A connection the sweep had already condemned is disconnected too. The descriptor stays the connection's own until it is let go, as before.
…throws ObjectDisposedException, so the stop cancels each op once and the ticker that cancelled them again each tick goes (#634)

A client op or timeout submitted on the reactor thread once Stop() has been called throws ObjectDisposedException, as one handed over from another thread already did after the teardown; a handover now throws from the stop on too. One handed over before the stop and taken off the queue after it completes with -ECANCELED rather than reaching the kernel past the stop's one cancel. So the first pass's cancel is the last, and nothing is left for a per-tick cancel to find: a wait that is armed again after the stop cancelled it, as a timer loop does, now ends at once instead of holding the stop to its deadline.

ioxide.redis.stackexchange: a connect the stopped reactor refuses closes the socket it opened. StackExchange.Redis reconnects through the tunnel for as long as its multiplexer lives, and each refused attempt left a socket open.

The docs of Stop, IRingHost and ReadFileAsync say so. The #609 test now expects the wait submitted again from the stop's cancel to throw, and the hand-over test the second request's send to be refused.
…d that a connection still open then stays allocated, its descriptor included, until the process exits (#634)

The teardown after a fault runs the same close as a stop's first pass, its stoppers, listeners and connection aborts, and frees what nothing holds. It does not run the loop on, so no completion lets a connection still open go: it stays allocated with its descriptor and the receive memory it can reach.
…multiplexer has on its ring (#634)

The transport registers no stopper: each one always has a receive on the ring, so the stop's cancel reaches every one, and the transport closes through its own receive-failure path, on the reactor thread. Needs a redis server, so it skips without one. Without the stop's cancel the server still sees both connections after the stop.
…s any socket, so no entry reaches whatever takes a number it closed, and the stop's docs say a kTLS connection is shut down rather than aborted (#634)

The pass's own drains can queue entries on a UDP socket or a listener, a QUIC send or an accept armed again, and the pass then closes both. Left queued past the closes, those entries were submitted after them, against a closed number (EBADF, "udp send error: -9") or against whatever socket another thread opened on it. They are now submitted first, until the kernel takes no more, since a failed prep ends a submit early. A connection's descriptor is not closed by the stop, so its entries needed none of this.

connect(AF_UNSPEC) fails with EOPNOTSUPP on a kTLS socket, which then ends by the shutdown alone: its client sees a FIN, and what it already queued may still go out. It never sends zero-copy, so the shutdown is enough to end its sends. Stop()'s docs and the comment on CloseTcpConnections say so.
…rees its buffers when its deadline is refused (#634)

ConnectAsync submitted its open deadline before the try, and from the stop on that submit throws ObjectDisposedException on the reactor thread too, so a pool's sweep or a handler the stop wakes that opened a connection left its socket and its buffers behind. The submit is now inside the try, whose catch disposes the connection.
…nnection, so their stops end at once rather than at the deadline, and a stop test that repeated another goes (#634)

Their TCP handler returned without DecRef, so the probe's connection was held for good and each of those reactors' stops, about 300 in E2E, ran the full second and exercised the deadline rather than the teardown. The RingTimer loop test failed on the same two reverts as the test for a ring op parked at the stop, which covers its wait-again too.
@MDA2AV
MDA2AV merged commit 042e909 into main Nov 15, 2026
2 checks passed
@MDA2AV
MDA2AV deleted the fix-634 branch November 15, 2026 16:40
MDA2AV added a commit that referenced this pull request Nov 15, 2026
All twelve csproj Version elements, 0.26.563 to 0.27.637. Carries the 24 PRs from #564 to #636. Package validation against 0.26.563 finds breaks in ioxide alone: Ring.Dispose(bool), Ring.SqPending, Native.IORING_REGISTER_SYNC_CANCEL and Native.io_uring_sync_cancel_reg are removed. The other ten published packages are clean.

Suites on this tree: Unit 306, E2E 797, Http 113, Chaos 105, File 14, Tls 269 (kTLS included), Pg 10 (19 skipped, no server here), Redis 1 (7 skipped and 1 pending, no server here).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment