Repository navigation
tcp: a connection whose handler let go gives its buffers back when a recv runs short, and the read timeout ends it even with no recv running - #49
Merged
Conversation
…recv runs short, and the read timeout ends it even with no recv running (#21) A recv parked on -ENOBUFS can't see the peer's close, and the sweep's shutdown has no completion to end, while the buffers come back only at the teardown those completions run. Nobody will read such a connection, so what it holds goes back, and the sweep runs the teardown itself. In incremental mode the connection kept its descriptor and group id for good; on the shared ring, the default, a few kilobytes of one-byte sends stopped every connection on the proactor. A port of MDA2AV/ioxide#546, with its tests.
…els without io_uring fix a68ed2df7213, not by kronos With a client's data and its reset both pending when the recv's poll runs, such a kernel takes both wakeups in one pass, returns the data and waits for a wakeup that never comes. The client still reads to the end before closing, so it closes with a FIN and the test passes on either kernel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A port of MDA2AV/ioxide#546. Fixes #21.
What was wrong
A connection is recycled only once its handler and the proactor have both let go, and the proactor lets go when its recv ends. A recv needs a buffer even to report the peer's close. So once a handler had gone and the buffers had run out, three things waited on each other:
The read timeout didn't break the loop.
sweepClosecallsshutdown()and leaves the teardown to the recv's completion, but a recv parked on-ENOBUFShas no completion coming, andsweepClosedstops the sweep from looking again.It takes no misbehaving handler, only one that returns before reading everything its client sends.
maxConnections, or stop a proactor on the shared ring #21): the connection keeps its descriptor and its buffer group id for good, so clients can use upmaxConnections.maxConnections, or stop a proactor on the shared ring #21 as first filed): the same state, with every connection drawing on one set of buffers. On one proactor at the default sizes, 66 clients whose handlers returned at once sent 64 one-byte messages each, 4,224 bytes in all. A new client was never answered, and every connection stayed open.The fix
Both parts are needed, as on ioxide.
-ENOBUFS, and in the sweep whenever any recv is parked,returnReleasedBuffersgives back what the connection holds: what its reader took and what is queued, the way a handler would have returned them. The recv then re-arms and reads on to the peer's close.readTimeoutMs = 0, and it frees the ring when it runs out rather than at the timeout.closeFromRecv, which still waits for a send that is with the kernel. This is the sweep patch from a connection whose buffers run out after its handler has gone is never closed: clients can use upmaxConnections, or stop a proactor on the shared ring #21. It covers a let-go connection with nothing to hand back, because others hold the ring.drainRecvnow hands each item back throughreturnBuffer, so it works on a connection's own ring too. At recycle that changes nothing, because the ring is reset whole there. A normal recv completion runs no new code: the new code runs on-ENOBUFSand in the sweep.decRef()'s doc now says that once a handler has let go, what is queued and what a reader holds can go back to the kernel at any point, so a handler that carries on must leave them alone. That was already true once its client closed. Now it can also happen when a recv runs short.Not changed:
sendTimeoutMs(shared mode: a few connections parked in a flush can drain the buffer pool and stall every recv on the proactor #22).tryGetItemand never returns them still breaks that method's contract. In incremental mode the read timeout ends its connection, and withreadTimeoutMs = 0it stays. On the shared ring those buffers stay lost, as the method's doc says.API
Nothing added or removed. Internal:
TcpConnection.queuedRecvs, for the tests.Tests
Ports of ioxide's, in
commonTestso they run on both runtimes, apart from TLS, which is native like the other TLS tests.LetGoDriveris what the wider ones share: clients that outlive their handlers empty the rings and then carry on. Afterwards a new client must be served and the connections must end. Each case runs on the shared ring and in incremental mode, with the clients leaving or staying, with and without a read timeout.IncrementalDryRingTestLetGoBufferTestLetGoHandlerStyleTestTlsLetGoBufferTestTestSocketgainssetSendTimeoutandsetNoDelay.Against main's source: 23 of the 33 new tests fail. The 10 that pass are:
They stay as regression tests. The failures name what's wrong, for example:
Each part taken out alone, against the 33 new tests:
-ENOBUFSa read left parked when its handler let go strands nothing it takes laterafter let-go connections gave their buffers back connections on them are served intactdrainRecvreturning by mode (as before: it skipped a connection's own ring)With the change:
commonTest.tlsmodule, so CI's run is what covers them.Found while porting: a kernel bug, not kron's
A client's reset that follows its data closely can go unnoticed: the recv returns the data, and nothing completes for the reset. With the read timeout off, the connection then stays open. Closing with the server's bytes unread is enough to send such a reset.
This is io_uring's, fixed upstream by a68ed2df7213, "io_uring/poll: fix multishot recv missing EOF on wakeup race", first in 7.0 and marked for stable. Before it, the multishot poll could take the data's wakeup and the reset's in one pass, return only the data, and then wait for a wakeup that never came. The fix runs the recv once more when it took several wakeups and the socket reports a hangup, as it does after a reset, and that recv gets
-ECONNRESET. There is nothing to change in kron.maintoo, in either ring mode. The read timeout still ends such a connection.TlsLetGoBufferTest's client reads to the end before closing, so it closes with a FIN and passes with or without the fix.🤖 Generated with Claude Code