Skip to content

tcp: the listeners carry sendTimeoutMs as TCP_USER_TIMEOUT, so a stream of small flushes to a peer that stopped reading or vanished ends too - #51

Merged
MDA2AV merged 1 commit into
mainfrom
fix/tcp-user-timeout
Nov 8, 2026
Merged

MDA2AV merged 1 commit into
mainfrom
fix/tcp-user-timeout

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Follows #50, which rewrote the same sendTimeoutMs doc. The same gap on ioxide is MDA2AV/ioxide#551.

What was wrong

A flush small enough to fit in the socket's send buffer completes whatever the peer does. So a server that keeps flushing a little at a time, to a peer that has stopped reading or is gone, was never caught:

  • The send timeout only fires on a flush that's still out, and each of these finishes at once.
  • The read timeout runs only while a read waits, and each flush restarts it anyway.

The connection stayed until the send buffer filled, which for a slow stream can take hours. For a peer that vanished without a FIN or reset, it stayed until TCP gave up retransmitting, about 15 minutes by default. Meanwhile it held its descriptor, its connection and slab, in incremental mode a group id out of maxConnections, and a handler producing data for nobody. SSE, websocket pushes and feeds have this shape.

The fix

Sockets.listen sets TCP_USER_TIMEOUT = sendTimeoutMs on each listener, next to TCP_NODELAY, and accepted sockets inherit it the same way. At 0 it sets nothing. The kernel then aborts a connection whose sent data has gone unacknowledged, or whose window has stayed shut with data waiting, for that long. The abort arrives as -ETIMEDOUT on the recv kron keeps armed, or on a send, and both already end a connection, so no other code changes.

  • A peer that vanished while we were sending is closed about sendTimeoutMs after it stopped acknowledging.
  • A peer that stopped reading is closed once its window has stayed shut for sendTimeoutMs. That starts only when its own receive buffer is full, since its kernel acknowledges until then.
  • A slow peer that keeps reading is left alone: its acknowledgements and window updates reset the kernel's clock.
  • The cost: a peer whose network drops for longer than sendTimeoutMs while we're sending is cut, where TCP would have waited about 15 minutes for it to come back.

It replaces neither clock. The read timeout still covers a live peer that goes quiet. The whole-flush deadline still covers a peer that reads a little now and then to hold a flush open, which the kernel's clock lets through because progress resets it. The sendTimeoutMs doc now says what the option adds.

Tests

TcpUserTimeoutTest is in commonTest, so it runs on both runtimes. The handler flushes 100 bytes every 50 ms to a client with a 4 KiB receive buffer, with readTimeoutMs = 0:

Test Client sendTimeoutMs Result
a client that stops reading while small flushes still fit is closed at the send timeout doesn't read 1000 closed after about 3.96 s
control - with the send timeout at 0 the same client stays open doesn't read 0 open after 6 s
control - a client that keeps reading through the same stream is left alone reads everything 1000 open after 3 s, every byte received

Without the setsockopt, the first test fails and both controls pass:

kotlin.AssertionError: still open after 12019 ms of flushes to a client not reading

No flush blocks in these tests, so the sweep's deadline never comes into play. A peer that vanishes can't be faked on loopback without root, so that case rests on tcp(7).

With the change:

  • Native: 283 tests passed, the 280 there were and the 3 new ones. The new tests passed 3 runs in a row.
  • JVM: 66 tests passed, the 63 there were and the 3 new ones.
  • kTLS: the tests skip on this machine, which has no tls module, so CI's run is what covers them.

…am of small flushes to a peer that stopped reading or vanished ends too

A flush small enough to fit in the socket's buffer completes whatever the peer does, so the sweep's whole-flush
deadline never sees a server that keeps flushing a little at a time stall, and the flushes keep restarting the read
clock. A peer that stopped reading held such a connection until the send buffer filled, and one that vanished until
TCP gave up retransmitting, about 15 minutes. The kernel now aborts it once sent data has gone unacknowledged, or the
peer's window has stayed shut with data waiting, for sendTimeoutMs. Accepted sockets inherit the option from the
listener, as they inherit TCP_NODELAY, and the abort ends the connection through the recv's error path.

The same gap on ioxide is MDA2AV/ioxide#551.
@MDA2AV
MDA2AV changed the base branch from docs/send-timeout to main November 8, 2026 16:27
@MDA2AV
MDA2AV merged commit e38c1d1 into main Nov 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant