Skip to content

Write at once on TUring when the write does not block - #213

Draft
resetius wants to merge 1 commit into
masterfrom
uring-direct-write
Draft

resetius wants to merge 1 commit into
masterfrom
uring-direct-write

Conversation

@resetius

@resetius resetius commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Experiment 3 of the TUring write-path series (draft, for measurement; the owner decides).

Problem

On TUring every WriteSome() is its own SQE, CQE and coroutine wake-up, even when the socket or pipe has room and the write would finish at once. The profile on rpi5 (single N=1024, master 85489f0) puts the write path at ~41% of the uring cycles per hop; ~660 cycles/hop of it are uring-only: the CQE post and request free, the write's share of SQE prep and fget, and the user side.

Change

TUring::Write() and TUring::Send() first try the write directly:

  • Send(): send(fd, buf, n, MSG_DONTWAIT | MSG_NOSIGNAL). IORING_OP_SEND never raises SIGPIPE either.
  • Write(): pwritev2(fd, iov, 1, -1, RWF_NOWAIT). RWF_NOWAIT gives EAGAIN on a full pipe even if the pipe is blocking, so the loop never blocks. A regular file returns EOPNOTSUPP (ext4 and tmpfs, checked on rpi5). Then the descriptor is written through the ring until the poller closes it, so a file write costs no extra syscall.
  • Bytes written, including a partial write, or an error other than EAGAIN/EINTR/EOPNOTSUPP: the operation is complete. op.Handle is reset, the result is in op.Result, and await_suspend() of TPollerDrivenSocket::WriteSome() / TPollerDrivenFileHandle::WriteSome() returns false, so the coroutine goes on without suspending. TIOCp never resets the handle here, so it does not change.
  • EAGAIN, EINTR or EOPNOTSUPP: the write goes through the ring as before.
  • Ordering: while a write on the descriptor is still in the ring (queued or submitted), a later write does not try the direct path; it is queued behind it. Writes are counted per descriptor (TOperationState::Write).
  • Errors are the ring's: a direct send() after a reset returns ECONNRESET first, then EPIPE. On master the ring gives the same; the new test passes on master too.
  • A partial write is returned as is, like the readiness pollers and like IORING_OP_SEND without MSG_WAITALL. The rest is not posted to the ring.

Tests

New, in tests/tests.cpp:

  • test_uring_write_at_once: socket and pipe writes finish inside WriteSome().
  • test_uring_write_behind_queued_write: a write queued in the ring after EAGAIN, then room comes, then a 1-byte write. The 1-byte write must queue behind, and the stream must be the queued bytes, then b.
  • test_uring_write_stream_order: 4 MiB of counters through a 4 KiB SO_SNDBUF, with direct, partial and ring writes mixed, arrives in order.
  • test_uring_write_at_once_errors: EPIPE without SIGPIPE from Send(); ECONNRESET, then EPIPE, after an RST.
  • test_uring_write_at_once_after_close: close while a write is pending gives -ECANCELED. Neither the pending-write count nor the regular-file mark stays on the number for the next file.
  • test_uring_write and test_uring_write_resume are updated: the first now checks the direct path, the second fills the pipe so the write still goes through the ring and resumes the coroutine.

Mutations, each caught by its test: ignore pending writes → behind_queued_write; no write-count decrement → after_close; no mark reset on close → after_close; no MSG_NOSIGNAL → errors; ring on any error → errors; full size as the result → stream_order.

A/B (rpi5)

rpi5 (Cortex-A76, Linux 7.3.0-1003-raspi, liburing 2.15, GCC 15.3), Release, examples/bench on taskset -c 3, run with run_ab.sh. The configs alternate in each repetition. Values are the median (min-max) of 7 repetitions for single N=1024 and of 5 for N=16 and chain100 (-n 1024 -a 100). Cycles/hop = cycles:u + cycles:k of -w 100000 minus -w 100, divided by 25 * 99900 hops. µs/hop = p50 of a run / 100001.

shape path µs/hop cycles/hop user kernel total / master
single N=1024 uring master 2.110 (2.106-2.142) 556 4540 5092 (5054-5163) 1.000
single N=1024 uring this PR 2.157 (2.141-2.175) 622 4577 5195 (5140-5247) 1.020
single N=1024 epoll master / this PR 1.678 / 1.674 415 / 406 3617 / 3615 4027 / 4021 1.000 / 0.999
single N=16 uring master 1.385 (1.368-1.393) 396 2925 3329 (3280-3351) 1.000
single N=16 uring this PR 1.441 (1.425-1.443) 426 3030 3456 (3419-3464) 1.038
single N=16 epoll master / this PR 1.223 / 1.215 344 / 340 2594 / 2580 2938 / 2915 1.000 / 0.992
chain100 uring master 1.759 (1.737-1.771) 330 3914 4240 (4172-4299) 1.000
chain100 uring this PR 1.948 (1.936-1.963) 371 4315 4674 (4638-4774) 1.102
chain100 epoll master / this PR 1.430 / 1.450 307 / 311 3119 / 3170 3426 / 3480 1.000 / 1.016

On top of the deferred task work (#214): 0.970 / 0.976 / 1.049 of master in run 1, against 0.933 / 0.898 / 1.014 for #214 alone. Run 2, with the final #214: 0.978 / 0.992 / 1.052 against 0.933 / 0.903 / 0.983.

Result: slower on rpi5 (+2% / +4% / +10%):

  • A write()/send() syscall per write costs more here (~500 cycles of syscall entry/exit on this kernel) than the write's SQE and CQE inside the shared io_uring_enter.
  • It is worst on chain100, where uring batches 100 writes per enter and the direct path pays a syscall for each.
  • Not worth keeping on this host; x86 was not measured.

kqueue (Mac, M5 Pro, 7 repetitions): master 0.622 / 0.615 / 0.463 µs/hop, the combination with this change 0.622 / 0.614 / 0.463 (1.000 / 0.998 / 1.001). The readiness pollers do not use this code.

Checks

  • rpi5 (Linux 7.3, liburing 2.15, GCC 15.3), Debug + ASan as in CI: ctest 4/4; test_uring* 23/23.
  • The new error, ordering and stream tests also pass on master; they check semantics, not the direct path (except test_uring_write_at_once*).
  • macOS (Apple clang, Debug + ASan): ctest 4/4.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Linux)

338 tests  +5   338 ✅ +5   24s ⏱️ -1s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 7ae2be3. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (MacOS)

254 tests  ±0   254 ✅ ±0   32s ⏱️ ±0s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 7ae2be3. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Windows)

175 tests  ±0   175 ✅ ±0   43s ⏱️ -1s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 7ae2be3. ± Comparison against base commit 85489f0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant