Repository navigation
tcp: the listeners carry sendTimeoutMs as TCP_USER_TIMEOUT, so a stream of small flushes to a peer that stopped reading or vanished ends too - #51
Merged
Conversation
…am of small flushes to a peer that stopped reading or vanished ends too A flush small enough to fit in the socket's buffer completes whatever the peer does, so the sweep's whole-flush deadline never sees a server that keeps flushing a little at a time stall, and the flushes keep restarting the read clock. A peer that stopped reading held such a connection until the send buffer filled, and one that vanished until TCP gave up retransmitting, about 15 minutes. The kernel now aborts it once sent data has gone unacknowledged, or the peer's window has stayed shut with data waiting, for sendTimeoutMs. Accepted sockets inherit the option from the listener, as they inherit TCP_NODELAY, and the abort ends the connection through the recv's error path. The same gap on ioxide is MDA2AV/ioxide#551.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #50, which rewrote the same
sendTimeoutMsdoc. The same gap on ioxide is MDA2AV/ioxide#551.What was wrong
A flush small enough to fit in the socket's send buffer completes whatever the peer does. So a server that keeps flushing a little at a time, to a peer that has stopped reading or is gone, was never caught:
The connection stayed until the send buffer filled, which for a slow stream can take hours. For a peer that vanished without a FIN or reset, it stayed until TCP gave up retransmitting, about 15 minutes by default. Meanwhile it held its descriptor, its connection and slab, in incremental mode a group id out of
maxConnections, and a handler producing data for nobody. SSE, websocket pushes and feeds have this shape.The fix
Sockets.listensetsTCP_USER_TIMEOUT = sendTimeoutMson each listener, next toTCP_NODELAY, and accepted sockets inherit it the same way. At 0 it sets nothing. The kernel then aborts a connection whose sent data has gone unacknowledged, or whose window has stayed shut with data waiting, for that long. The abort arrives as-ETIMEDOUTon the recv kron keeps armed, or on a send, and both already end a connection, so no other code changes.sendTimeoutMsafter it stopped acknowledging.sendTimeoutMs. That starts only when its own receive buffer is full, since its kernel acknowledges until then.sendTimeoutMswhile we're sending is cut, where TCP would have waited about 15 minutes for it to come back.It replaces neither clock. The read timeout still covers a live peer that goes quiet. The whole-flush deadline still covers a peer that reads a little now and then to hold a flush open, which the kernel's clock lets through because progress resets it. The
sendTimeoutMsdoc now says what the option adds.Tests
TcpUserTimeoutTestis incommonTest, so it runs on both runtimes. The handler flushes 100 bytes every 50 ms to a client with a 4 KiB receive buffer, withreadTimeoutMs = 0:sendTimeoutMsa client that stops reading while small flushes still fit is closed at the send timeoutcontrol - with the send timeout at 0 the same client stays opencontrol - a client that keeps reading through the same stream is left aloneWithout the setsockopt, the first test fails and both controls pass:
No flush blocks in these tests, so the sweep's deadline never comes into play. A peer that vanishes can't be faked on loopback without root, so that case rests on tcp(7).
With the change:
tlsmodule, so CI's run is what covers them.