Skip to content

Defer uring task work to Wait with SINGLE_ISSUER and DEFER_TASKRUN, without spinning on a stale TASKRUN flag - #214

Draft
resetius wants to merge 6 commits into
masterfrom
uring-defer-taskrun-v2
Draft

resetius wants to merge 6 commits into
masterfrom
uring-defer-taskrun-v2

Conversation

@resetius

@resetius resetius commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Experiment 2 of the TUring write-path series (draft, for measurement; the owner decides).

This is #188 with #191 stacked on it, rebased onto master 85489f0. #188 was closed on 2026-11-06 ("not merging") for these reasons:

  • the gain was only on single N <= 128;
  • it adds a thread-affinity contract, a deferred-task-work drain loop and R_DISABLED;
  • 2 of 11 actor-bench runs hung, and the cause was not found.

The owner asked to remeasure it with user/kernel cycles per hop, alone and together with the direct write (experiment 3), and to chase the hangs.

Change (as in #188 + #191)

  • The ring is set up with SINGLE_ISSUER | DEFER_TASKRUN | TASKRUN_FLAG | R_DISABLED.
  • The kernel keeps task work in a ring-local list and runs it in io_uring_enter with GETEVENTS, which Wait() makes anyway.
  • The ring is enabled and bound to the thread that first waits, or first submits a full SQ.
  • Wait() and the destructor drain the deferred work while IORING_SQ_TASKRUN is set.

New in this PR:

  • Fix of the actor-bench hang (the 2-of-11 hangs of Defer uring task work to Wait with SINGLE_ISSUER and DEFER_TASKRUN #188; see below): RunTaskWork() stops when io_uring_get_events() posted no completion.
  • If the kernel refuses these flags with -EINVAL (before 6.1), the ring falls back to COOP_TASKRUN | TASKRUN_FLAG (5.19), and then to plain task work. There is no thread binding in either fallback.
  • test_uring_wait_timeout_with_deferred_completion: a wait with a timeout returns 0 at the timeout, and returns 1 early when a write in another thread completes a pending read, whose task work is deferred to the wait.

Tests

  • Fallbacks were checked by forcing -EINVAL with an unknown flag bit on the first, and then also on the second, flag set. The whole tests binary passes except test_uring_destroyed_after_failed_wait, which expects SINGLE_ISSUER's EEXIST in another thread.
  • Setting IORING_SETUP_SQ_AFF instead fails late, after the ring memory is charged. On rpi5 (RLIMIT_MEMLOCK 8 MiB) that runs out of locked memory after ~30 rings while the freed rings wait for their exit work. A real old kernel rejects unknown flags before it allocates anything, so this is not a problem of the fallback.

The actor-bench hang

On rpi5, Linux 7.3, the local actor bench (4 nodes, inflight 1024, 10M messages) hung in about 1 of 9 runs: 5 hangs in ~45 runs, Release and RelWithDebInfo. Each hang looked the same:

  • One node spins at 100% CPU inside Wait(), in RunTaskWork() → io_uring_get_events(). The other nodes sleep in io_cqring_wait.
  • Its ring has sq_flags == IORING_SQ_TASKRUN and an empty CQ.
  • perf stat on the spinning process over 1 s: ~3.06M io_uring_enter calls, 0 io_uring:io_uring_local_work_run and 0 io_uring:io_uring_task_add. So no deferred work is pending, but the flag is set.
  • ss shows ~16 KB unread on one of its inbound sockets: InboundConnection waits in co_await Poller->Yield(), and timers never run because Wait() never returns. The other nodes wait for its messages.
  • Pausing the spinner (SIGSTOP for 1 s) does not help, and no other task is runnable, so this is not a starved producer.

Cause: IORING_SQ_TASKRUN is a hint.

  • io_req_local_work_add() (Linux 7.3, io_uring/tw.c) sets it after the request is already queued, so the owner thread can run that work, and clear the flag at the start of __io_run_local_work(), before the producer sets it.
  • After that only the next local-work run clears it, and io_cqring_wait() starts one only when the queue is not empty.
  • The drain loop of Defer uring task work to Wait with SINGLE_ISSUER and DEFER_TASKRUN #188 (while (RunTaskWork()), also in the destructor) treated the flag as exact and spun forever.

Fix: a drain step counts only if io_uring_get_events() moved the CQ tail. Otherwise the drain stops and the next Wait() runs what is left.

test_uring_wait_with_stale_taskrun_flag sets the flag by hand, through a second mapping of the ring, while nothing is pending. Wait({0, 0}) must return at once. With the old RunTaskWork() it spins until an alarm clears the flag 3 s later, and the test fails.

After the fix: 40 of 40 runs completed, 0 hangs.

A/B (rpi5)

rpi5 (Cortex-A76, Linux 7.3.0-1003-raspi, liburing 2.15, GCC 15.3), Release, examples/bench on taskset -c 3, run with run_ab.sh. The configs alternate in each repetition. Values are the median (min-max) of 7 repetitions for single N=1024 and of 5 for N=16 and chain100 (-n 1024 -a 100). Cycles/hop = cycles:u + cycles:k of -w 100000 minus -w 100, divided by 25 * 99900 hops. µs/hop = p50 of a run / 100001.

Run 1 (head bdc1ebf, before the stale-flag fix, which does not touch the hot path); run 2 with the final head is below.

shape path µs/hop cycles/hop user kernel total / master
single N=1024 uring master 2.110 (2.106-2.142) 556 4540 5092 (5054-5163) 1.000
single N=1024 uring this PR 1.974 (1.958-2.008) 546 4216 4753 (4686-4822) 0.933
single N=1024 epoll master / this PR 1.678 / 1.688 415 / 431 3617 / 3634 4027 / 4067 1.000 / 1.010
single N=16 uring master 1.385 (1.368-1.393) 396 2925 3329 (3280-3351) 1.000
single N=16 uring this PR 1.244 (1.241-1.270) 391 2600 2991 (2976-3044) 0.898
single N=16 epoll master / this PR 1.223 / 1.225 344 / 350 2594 / 2594 2938 / 2944 1.000 / 1.002
chain100 uring master 1.759 (1.737-1.771) 330 3914 4240 (4172-4299) 1.000
chain100 uring this PR 1.795 (1.779-1.808) 354 3939 4301 (4261-4329) 1.014
chain100 epoll master / this PR 1.430 / 1.450 307 / 315 3119 / 3182 3426 / 3497 1.000 / 1.021
  • single: -6.7% at N=1024 and -10.2% at N=16. At N=16 uring is within 1.8% of epoll (2991 vs 2938 cycles).
  • chain100: no change (+1.4%), within the spread.
  • epoll code does not change; its 1.01-1.02 is binary layout (all heads show it on chain100).

Run 2: master 85489f0, #214 608f9e8 (deferred task work), #215 51879eb (fixed files), #214 + #215 dbf3d2b, #214 + #213 eebf0d1 (direct write). Same method; heads alternate in each repetition.

shape path µs/hop cycles/hop user kernel total / master
single N=1024 uring master 2.114 (2.088-2.166) 562 4530 5078 (5015-5163) 1.000
single N=1024 uring #214 1.972 (1.954-1.987) 555 4176 4737 (4685-4949) 0.933
single N=1024 uring #215 2.090 (2.058-2.107) 574 4447 5022 (4913-5146) 0.989
single N=1024 uring #214 + #215 1.964 (1.915-1.976) 564 4167 4732 (4595-4875) 0.932
single N=1024 uring #214 + #213 2.067 (2.029-2.100) 597 4373 4967 (4871-5111) 0.978
single N=1024 epoll master 1.661 (1.652-1.692) 408 3585 3992 (3965-4081) 1.000
single N=1024 epoll #214 / #215 / #214 + #215 / #214 + #213 1.669 / 1.677 / 1.680 / 1.661 423 / 403 / 425 / 402 3627 / 3643 / 3617 / 3609 4048 / 4046 / 4036 / 4001 1.014 / 1.013 / 1.011 / 1.002
single N=16 uring master 1.372 (1.366-1.393) 395 2907 3297 (3281-3357) 1.000
single N=16 uring #214 1.241 (1.239-1.258) 391 2589 2978 (2970-3024) 0.903
single N=16 uring #215 1.316 (1.302-1.322) 408 2755 3165 (3125-3176) 0.960
single N=16 uring #214 + #215 1.205 (1.200-1.219) 401 2494 2889 (2888-2928) 0.876
single N=16 uring #214 + #213 1.364 (1.350-1.368) 425 2844 3271 (3239-3287) 0.992
single N=16 epoll master 1.219 (1.214-1.222) 343 2583 2926 (2919-2936) 1.000
single N=16 epoll #214 / #215 / #214 + #215 / #214 + #213 1.221 / 1.217 / 1.231 / 1.218 349 / 339 / 348 / 340 2582 / 2579 / 2605 / 2585 2931 / 2921 / 2954 / 2927 1.002 / 0.998 / 1.009 / 1.000
chain100 uring master 1.719 (1.710-1.747) 331 3872 4203 (4112-4290) 1.000
chain100 uring #214 1.726 (1.703-1.763) 352 3778 4130 (4087-4312) 0.983
chain100 uring #215 1.625 (1.618-1.660) 331 3604 3933 (3866-4073) 0.936
chain100 uring #214 + #215 1.649 (1.630-1.669) 360 3577 3936 (3922-4103) 0.937
chain100 uring #214 + #213 1.824 (1.798-1.884) 340 4083 4420 (4375-4563) 1.052
chain100 epoll master 1.414 (1.407-1.435) 312 3103 3415 (3380-3520) 1.000
chain100 epoll #214 / #215 / #214 + #215 / #214 + #213 1.428 / 1.423 / 1.451 / 1.425 314 / 311 / 325 / 299 3128 / 3122 / 3169 / 3120 3433 / 3434 / 3494 / 3421 1.005 / 1.006 / 1.023 / 1.002

Local actor bench (ping_actors, 4 nodes, inflight 1024, 10M messages, 5 repetitions, timeout 60 s per run), msg/s, median (min-max), no hangs:

head uring epoll
master 3 140 060 (2 834 870-3 161 820) 2 638 130 (2 607 150-2 703 770)
#214 3 126 840 (2 912 330-3 202 830) 2 622 220 (2 555 710-2 692 790)
#215 3 052 120 (3 007 500-3 161 370) 2 609 410 (2 539 140-2 642 180)
#214 + #215 3 119 500 (2 883 150-3 191 260) 2 609 590 (2 579 490-2 681 700)
#214 + #213 3 061 730 (2 945 120-3 126 350) 2 618 180 (2 562 770-2 760 790)

The actor bench does not tell the heads apart: they are all within each other's spread.

Run 2 for this PR: 0.933 / 0.903 / 0.983 of master, the same as run 1.

Checks

  • rpi5 (Linux 7.3, liburing 2.15, GCC 15.3), Debug + ASan as in CI: ctest 4/4; test_uring* 22/22.
  • macOS (Debug + ASan): ctest 4/4.

With DEFER_TASKRUN a linked close is deferred task work once its cancel
completes. io_uring_submit() enters without GETEVENTS, so the ring's exit
cancelled the closes prepared by a Wait() that threw and leaked the
descriptors. The destructor now enters with GETEVENTS and runs the rest of
the deferred work with the same helper as Wait().
With PR 187 only descriptors with operations get the linked close, so the
test left two deferred items and passed without the drain in Wait(). With a
read on each of 40 handles and a 256-entry queue the cancelled reads and the
closes are more than one wait runs; the test fails if Wait() does not drain.
@github-actions

Copy link
Copy Markdown

Unit Test Results (Linux)

337 tests  +4   337 ✅ +4   23s ⏱️ ±0s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 608f9e8. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (MacOS)

254 tests  ±0   254 ✅ ±0   32s ⏱️ ±0s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 608f9e8. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Windows)

175 tests  ±0   175 ✅ ±0   43s ⏱️ -1s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 608f9e8. ± Comparison against base commit 85489f0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant