Skip to content

tcp: a connection whose handler let go gives its buffers back when a recv runs short, and the read timeout ends it even with no recv running - #49

Merged
MDA2AV merged 2 commits into
mainfrom
fix/let-go-buffers
Nov 8, 2026
Merged

MDA2AV merged 2 commits into
mainfrom
fix/let-go-buffers

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

A port of MDA2AV/ioxide#546. Fixes #21.

What was wrong

A connection is recycled only once its handler and the proactor have both let go, and the proactor lets go when its recv ends. A recv needs a buffer even to report the peer's close. So once a handler had gone and the buffers had run out, three things waited on each other:

  • Buffers come back only at teardown, since nobody is left to read them.
  • Teardown needs the recv to end.
  • The recv needs a buffer to end.

The read timeout didn't break the loop. sweepClose calls shutdown() and leaves the teardown to the recv's completion, but a recv parked on -ENOBUFS has no completion coming, and sweepClosed stops the sweep from looking again.

It takes no misbehaving handler, only one that returns before reading everything its client sends.

The fix

Both parts are needed, as on ioxide.

  • Hand back: when a let-go connection's recv hits -ENOBUFS, and in the sweep whenever any recv is parked, returnReleasedBuffers gives back what the connection holds: what its reader took and what is queued, the way a handler would have returned them. The recv then re-arms and reads on to the peer's close.
    • This also works with readTimeoutMs = 0, and it frees the ring when it runs out rather than at the timeout.
    • The reader is asked, not claimed the way recycle claims it, because a read left parked can still take more.
  • The sweep finishes the job: a connection the sweep condemned whose recv is parked goes straight to closeFromRecv, which still waits for a send that is with the kernel. This is the sweep patch from a connection whose buffers run out after its handler has gone is never closed: clients can use up maxConnections, or stop a proactor on the shared ring #21. It covers a let-go connection with nothing to hand back, because others hold the ring.

drainRecv now hands each item back through returnBuffer, so it works on a connection's own ring too. At recycle that changes nothing, because the ring is reset whole there. A normal recv completion runs no new code: the new code runs on -ENOBUFS and in the sweep.

decRef()'s doc now says that once a handler has let go, what is queued and what a reader holds can go back to the kernel at any point, so a handler that carries on must leave them alone. That was already true once its client closed. Now it can also happen when a recv runs short.

Not changed:

API

Nothing added or removed. Internal: TcpConnection.queuedRecvs, for the tests.

Tests

Ports of ioxide's, in commonTest so they run on both runtimes, apart from TLS, which is native like the other TLS tests. LetGoDriver is what the wider ones share: clients that outlive their handlers empty the rings and then carry on. Afterwards a new client must be served and the connections must end. Each case runs on the shared ring and in incremental mode, with the clients leaving or staying, with and without a read timeout.

Test What lets go
IncrementalDryRingTest #21's two tests, and a ring that runs dry after the handler let go or before, held raw or by a reader, with and without a read timeout
LetGoBufferTest Exact ring accounting, at the default sizes too, a reader holding, a read left parked, a ring others hold, a flush left in flight, a handler the send timeout wakes, rounds showing every buffer comes back exactly once with echoes intact, and an upload past the ring
LetGoHandlerStyleTest Raw reads, a throw, a pipe reader holding, completed or parked, and a duplex pipe
TlsLetGoBufferTest Answered through the pipes or the raw session, a failed or timed-out handshake, and kTLS TX and TX + RX

TestSocket gains setSendTimeout and setNoDelay.

Against main's source: 23 of the 33 new tests fail. The 10 that pass are:

  • Controls: 4.
  • Send-timeout cases: 2, where the failed send ends the connection anyway.
  • Timed-out handshake: 1, where something else ends the connection too.
  • kTLS: 3, which skip on this machine.

They stay as regression tests. The failures name what's wrong, for example:

shared ring - connections whose handlers let go holding every buffer are closed and a new client is served:
    the new client was never answered, with 6 connections still open
a connection whose ring ran dry after its handler let go is closed once its client leaves:
    the connection is still open after its client left
refused uploads do not use up the connection cap:
    the next client was shed, every group id taken by clients long gone

Each part taken out alone, against the 33 new tests:

Taken out Tests that fail
The hand-back on -ENOBUFS 3
The hand-back in the sweep 6
The sweep's teardown of a parked recv 2
Asking the reader 3
Asking the reader without claiming it (claimed instead) 1: a read left parked when its handler let go strands nothing it takes later
Draining the queue 16
Returning each buffer once (returned twice instead) 1: after let-go connections gave their buffers back connections on them are served intact
drainRecv returning by mode (as before: it skipped a connection's own ring) 12

With the change:

  • Native: 313 tests passed, the 280 there were and the 33 new ones.
  • JVM: 89 tests passed, the 63 there were and the 26 new ones in commonTest.
  • kTLS: the tests skip on this machine, which has no tls module, so CI's run is what covers them.
  • Stability: the new tests passed 3 runs in a row.

Found while porting: a kernel bug, not kron's

A client's reset that follows its data closely can go unnoticed: the recv returns the data, and nothing completes for the reset. With the read timeout off, the connection then stays open. Closing with the server's bytes unread is enough to send such a reset.

This is io_uring's, fixed upstream by a68ed2df7213, "io_uring/poll: fix multishot recv missing EOF on wakeup race", first in 7.0 and marked for stable. Before it, the multishot poll could take the data's wakeup and the reset's in one pass, return only the data, and then wait for a wakeup that never came. The fix runs the recv once more when it took several wakeups and the socket reports a hangup, as it does after a reset, and that recv gets -ECONNRESET. There is nothing to change in kron.

  • Kernels without the fix: Linux 6.14, where this was found, is one. There it happens on main too, in either ring mode. The read timeout still ends such a connection.
  • CI: runs 7.0, which has the fix. The probe wasn't run there.
  • Tests: TlsLetGoBufferTest's client reads to the end before closing, so it closes with a FIN and passes with or without the fix.

🤖 Generated with Claude Code

…recv runs short, and the read timeout ends it even with no recv running (#21)

A recv parked on -ENOBUFS can't see the peer's close, and the sweep's shutdown has no completion to end, while the
buffers come back only at the teardown those completions run. Nobody will read such a connection, so what it holds
goes back, and the sweep runs the teardown itself. In incremental mode the connection kept its descriptor and group id
for good; on the shared ring, the default, a few kilobytes of one-byte sends stopped every connection on the proactor.

A port of MDA2AV/ioxide#546, with its tests.
…els without io_uring fix a68ed2df7213, not by kronos

With a client's data and its reset both pending when the recv's poll runs, such a kernel takes both wakeups in one
pass, returns the data and waits for a wakeup that never comes. The client still reads to the end before closing, so
it closes with a FIN and the test passes on either kernel.
@MDA2AV
MDA2AV merged commit 24f1a03 into main Nov 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

a connection whose buffers run out after its handler has gone is never closed: clients can use up maxConnections, or stop a proactor on the shared ring

1 participant