Skip to content

Read ahead on TUring with multishot reads into a buffer ring for ReadBuffer() - #210

Draft
resetius wants to merge 1 commit into
read-buffer-commonfrom
read-buffer-uring
Draft

resetius wants to merge 1 commit into
read-buffer-commonfrom
read-buffer-uring

Conversation

@resetius

@resetius resetius commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Depends on #209 (stacked: the diff of this PR is the second commit only).

TUring reads ahead for ReadBuffer(): the first wait on a descriptor arms one multishot request, and the kernel picks the buffers from a provided buffer ring.

  • Sockets: IORING_OP_RECV multishot. Pipes and other pollable files: IORING_OP_READ_MULTISHOT. Regular files (multishot answers -EBADFD): one buffer-select read at a time.
  • The ring holds the TReadBufferConfig::Count buffers of the pool (io_uring_setup_buf_ring, set up at the first ReadBuffer()). ~TReadBuffer puts the buffer back into the ring.
  • -ENOBUFS (ring empty): the descriptor waits and is armed again when a buffer comes back.
  • PerSocketLimit: at that many queued leases the multishot is cancelled. It is armed again when the reader waits on an empty queue, and the rest waits in the socket. The limit is checked as completions arrive, so one wake-up that reads a burst can go over it.
  • Close: the armed multishot counts as an operation of the descriptor (the counter of Skip the uring cancel by descriptor when the descriptor has no operations #187), so the close goes as cancel + close in a hard link. A waiter gets ECANCELED, as ReadSome() does today. user_data carries (fd, generation), so completions of a closed descriptor give their buffers back to the ring and never reach a socket that reuses the number.
  • A coroutine destroyed while waiting only drops its waiter: nothing goes to the kernel (today: ASYNC_CANCEL and submit_and_wait).
  • ReadSome() adapter: after ReadBuffer() on a socket, ReadSome() copies from the same queue of leases (one memcpy), so the order of bytes is kept and no single-shot read runs beside the multishot. The switch is a flag in the socket object; a socket that never called ReadBuffer() takes the old path plus one branch on that flag. The readiness backends are not touched.
  • One reader per descriptor (assert).
  • Requires Linux 6.7 for READ_MULTISHOT (the project minimum is 7.0) and liburing 2.5.

Pool slots get one more cache line each: the ring hands the buffers out in turn, and 16 KiB apart they all fell into the same cache sets (N=16: L3 refills 1.05 -> 0.04 per hop, -6% cycles).

examples/bench gets -r some|buffer|adapter: ReadSome() as before (the same reader coroutine as on master), ReadBuffer(), or ReadSome() after one ReadBuffer() switched the pipe to the leases.

Tests

rpi5, Linux 7.3, Debug + ASan, head fa03c20:

  • ctest 4/4, test_read_buffer_* 35/35.
  • The 4 uring tests each fail under their mutation: no fd operation count for the multishot, no generation check, no per-socket limit, no re-arm after -ENOBUFS.
  • macOS: ReadBuffer tests 24/24.
  • Stacked PRs get no CI here.

A/B

rpi5 (Cortex-A76, Linux 7.3.0-1003-raspi, liburing 2.15, GCC 15.3, Release, taskset -c 3). master 85489f0 vs this PR fa03c20.

  • examples/bench -w 100000: 25 runs per invocation, µs/hop = p50 / hops.
  • Cycles/hop: perf stat cycles:u/cycles:k of -w 100000 minus -w 100, per hop.
  • Median (min-max) over the repetitions; configurations alternate.

single N=1024, uring, 7 repetitions:

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
uring, master 2.170 (2.154-2.198) 579 4681 5268 (5172-5339) 1.000 12.4
uring, #210 ReadSome 2.186 (2.143-2.203) 609 4725 5344 (5144-5392) 1.014 12.4
uring, #210 ReadBuffer 2.233 (2.225-2.240) 876 4475 5351 (5304-5434) 1.016 15.7
uring, #210 adapter 2.213 (2.189-2.244) 895 4433 5338 (5233-5536) 1.013 15.5

single N=1024, epoll, 7 repetitions (ReadBuffer() on epoll pays one more read() with EAGAIN per hop than the bench's ReadSomeYield(), because it tries the read first like ReadSome()):

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
epoll, master 1.701 (1.688-1.721) 413 3698 4106 (4042-4218) 1.000 7.2
epoll, #209 1.691 (1.686-1.708) 401 3673 4073 (4045-4191) 0.992 7.2
epoll, #210 ReadSome 1.711 (1.700-1.721) 430 3700 4120 (4079-4224) 1.004 7.5
epoll, #210 ReadBuffer 2.212 (2.210-2.225) 735 4593 5333 (5305-5382) 1.299 12.1
epoll, #210 adapter 1.702 (1.694-1.716) 423 3682 4116 (4090-4158) 1.003 7.6

single N=16, 5 repetitions:

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
uring, master 1.389 (1.365-1.391) 397 2941 3340 (3272-3344) 1.000 0.0
uring, #210 ReadBuffer 1.288 (1.275-1.301) 548 2541 3089 (3062-3126) 0.925 0.1
uring, #210 adapter 1.302 (1.300-1.308) 571 2556 3132 (3120-3143) 0.938 0.1
epoll, master 1.220 (1.217-1.231) 341 2584 2925 (2923-2965) 0.876 0.0

chain100 N=1024 (-a 100), 5 repetitions:

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
uring, master 1.801 (1.789-1.802) 338 3969 4303 (4267-4417) 1.000 14.6
uring, #210 ReadBuffer 1.643 (1.611-1.677) 479 3461 3943 (3850-4080) 0.916 16.4
uring, #210 adapter 1.583 (1.575-1.616) 396 3443 3834 (3793-3965) 0.891 17.0
epoll, master 1.464 (1.427-1.479) 321 3189 3512 (3419-3648) 0.816 9.3

Actor bench (ping_actors, 10 nodes, inflight 1024, 10M messages, still on ReadSome()), msg/s:

msg/s, median min-max repetitions
uring, master 1 022 060 967 437-1 041 730 7
uring, #210 1 036 910 1 018 850-1 083 020 7

Result

  • At single N=1024 the multishot read saves 206 kernel cycles per hop (-4.4%), and the lease bookkeeping in user space costs 297 more. The total is 1.016x master, within the spread. This is not the -15..25% estimated in the design.
    • With multishot the kernel still runs the whole read in task work after each poll wake-up (io_rw_init_file, aa_file_perm, the copy). Only the SQE, the first attempt with -EAGAIN and the apoll arm go away, and buffer selection is added.
  • With a small working set or with batching the read-ahead pays: N=16 -7.5% (kernel -13.6%), chain100 -8.4% (ReadBuffer()) and -10.9% (adapter).
  • epoll stays ahead in all three shapes.
  • Recommendation: do not make the ReadSome() adapter the TUring default. It costs 16 MiB per poller and gains nothing at N=1024. ReadBuffer() is opt-in. The user-side cost is the subject of a separate follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant