Skip to content

Cut the user-side cost of TUring read-ahead: one-line per-fd state, batched ring returns, frame prefetch - #212

Draft
resetius wants to merge 1 commit into
read-buffer-uringfrom
read-buffer-uring-opt
Draft

resetius wants to merge 1 commit into
read-buffer-uringfrom
read-buffer-uring-opt

Conversation

@resetius

@resetius resetius commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Depends on #210 (stacked: the diff of this PR is the last commit only). TUring only: it does not touch IOCP (#211) or the readiness backends.

At single N=1024 the read-ahead of #210 saved kernel cycles but lost them in user space. A perf record -e cycles:u profile showed cache misses, not instructions:

  • the per-descriptor state line in OnBuffered (55% of its samples on the first load);
  • the separately allocated queue of leases;
  • the waiter's coroutine frame, loaded only at resume (63% of WakeupReadyHandles on the indirect call).

Changes:

  • One-line per-fd state. TBufferedFd is alignas(64). The first queued buffer is kept inline as (id, size, offset), with a vector only for the rest. A TReadBuffer is made only when ReadBuffer() takes it; the ReadSome() adapter copies from the raw buffer and gives it straight back.
  • Batched ring returns. Buffers back from leases are queued and put into the ring with one tail update per Wait(), instead of one io_uring_buf_ring_advance per lease. A count of the buffers in the ring decides, after -ENOBUFS, whether to re-arm at once (buffers came back meanwhile) or to wait for the next flush.
  • Frame prefetch. When a completion wakes a waiter, its coroutine frame is prefetched, because it is resumed after all completions are taken.

Tests

rpi5, Linux 7.3, Debug + ASan, head 44f8d58:

  • ctest 4/4; test_read_buffer_* 35/35, three runs in a row.
  • The first version of the batched returns lost a re-arm after -ENOBUFS; test_read_buffer_consume(TUring) caught it.
  • Mutations: each fails its test. No fd operation count, no generation check, no per-socket limit, no re-arm when the returned buffers reach the ring, no check of the ring count after -ENOBUFS.

A/B

rpi5 (Cortex-A76, Linux 7.3.0-1003-raspi, liburing 2.15, GCC 15.3, Release, taskset -c 3), 5 alternating repetitions, median (min-max). master 85489f0, #210 fa03c20, this PR 44f8d58. Method as in #210: cycles/hop from perf stat of -w 100000 minus -w 100.

single N=1024:

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
uring, master ReadSome 2.159 (2.155-2.190) 567 4665 5233 (5185-5256) 1.000 12.0
uring, #210 ReadBuffer 2.228 (2.214-2.249) 882 4530 5409 (5387-5459) 1.034 16.2
uring, this PR ReadBuffer 2.112 (2.085-2.121) 815 4245 5062 (5016-5086) 0.967 14.0
uring, this PR adapter 2.081 (2.065-2.107) 797 4205 5003 (4960-5120) 0.956 13.8
uring, this PR ReadSome 2.202 (2.193-2.235) 593 4688 5291 (5254-5369) 1.011 12.3

Plain ReadSome() on epoll (no code change; regression check):

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
epoll, master 1.696 (1.678-1.700) 420 3682 4105 (4057-4165) 1.000 7.3
epoll, this PR ReadSome 1.703 (1.677-1.710) 417 3668 4094 (4027-4114) 0.997 7.4

The uring ReadSome() row of this PR (1.011) is the bench binary, not the library. #210 measured its library with master's bench.cpp at 1.00; the reader coroutine is the same.

single N=16:

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
uring, master ReadSome 1.385 (1.368-1.389) 397 2923 3319 (3286-3337) 1.000 0.0
uring, #210 ReadBuffer 1.278 (1.266-1.299) 538 2531 3070 (3042-3130) 0.925 0.1
uring, this PR ReadBuffer 1.242 (1.223-1.248) 514 2461 2975 (2941-2999) 0.896 0.0
uring, this PR adapter 1.249 (1.231-1.260) 534 2464 3001 (2958-3030) 0.904 0.1
epoll, master 1.219 (1.215-1.238) 340 2586 2931 (2920-2975) 0.883 0.0

chain100 N=1024 (-a 100):

path µs/hop cycles/hop user kernel total total / base L3 refills/hop
uring, master ReadSome 1.813 (1.786-1.820) 337 4019 4351 (4314-4452) 1.000 15.2
uring, #210 ReadBuffer 1.624 (1.618-1.632) 483 3494 3989 (3890-4016) 0.917 17.0
uring, this PR ReadBuffer 1.580 (1.552-1.598) 439 3367 3807 (3725-3825) 0.875 15.4
uring, this PR adapter 1.501 (1.474-1.509) 339 3270 3605 (3589-3712) 0.829 14.7
epoll, master 1.446 (1.431-1.458) 307 3159 3466 (3420-3500) 0.797 8.9

Result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant