Skip to content

Set TCP_NODELAY on actor node sockets - #206

Merged
resetius merged 1 commit into
masterfrom
actors-socket-options
Nov 13, 2026
Merged

resetius merged 1 commit into
masterfrom
actors-socket-options

Conversation

@resetius

Copy link
Copy Markdown
Owner

Actor nodes exchange many small messages (acks, stream tails). Nagle's algorithm plus delayed ACK can hold a small write behind unacked data, which adds latency on a window round trip. Nothing in coroio set TCP_NODELAY on actor sockets.

What changed:

  • TSocket::SetNoDelay(): best-effort TCP_NODELAY (POSIX and Windows). It returns false instead of throwing, so a connection already reset by the peer or a non-TCP socket stays usable.
  • TNode calls it on the outbound socket after the socket factory creates it, before connect.
  • TActorSystem::InboundServe calls it on each accepted socket.
  • Test test_node_sets_tcp_nodelay checks the option on a connected node socket. The accepted side is not observable from outside the actor system, so it has no test.

Not changed: SO_SNDBUF/SO_RCVBUF. I tried 4 MB on both ends and saw no gain (below). SO_KEEPALIVE is already set for every socket in TSocketBase::Setup.

Measurements (QumirDB, 4 nodes x 4 threads, loopback macOS, with large chunks and a byte window; ABBA order, all results verified):

Workload stock TCP_NODELAY NODELAY + 4 MB buffers
TPC-DS q95 SF100, execute, mean of 4 runs 11.32 s 11.32 s 11.33 s
TPC-H SF100, 22 queries, sum, mean of 2 passes 45.33 s 45.42 s 45.67 s

There is no measurable change on loopback; differences are within 0.5% noise. The option is the standard setting for a message-oriented transport and should matter on a real network or with small windows.

Risk: low. With Nagle off, tiny writes go out as separate packets; the actor transport already batches into large buffers before writing.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Linux)

333 tests   333 ✅  24s ⏱️
  4 suites    0 💤
  4 files      0 ❌

Results for commit 58a7158.

@github-actions

Copy link
Copy Markdown

Unit Test Results (MacOS)

254 tests   254 ✅  34s ⏱️
  4 suites    0 💤
  4 files      0 ❌

Results for commit 58a7158.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Windows)

175 tests   175 ✅  43s ⏱️
  4 suites    0 💤
  4 files      0 ❌

Results for commit 58a7158.

@resetius
resetius merged commit 85489f0 into master Nov 13, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant