Skip to content

reactor: once it has stopped, neither the thread that ran it nor a new thread given that one's managed id is taken for the reactor's (#608) - #622

Merged
MDA2AV merged 1 commit into
mainfrom
fix-608
Nov 15, 2026
Merged

MDA2AV merged 1 commit into
mainfrom
fix-608

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #608.
Touches core: src/ioxide/Reactor/Reactor.Runner.cs (the end of Teardown) and src/ioxide/Reactor/Reactor.Post.cs (XML docs only), because whether a caller is on the reactor is the reactor's own state, read by every satellite through OnReactorThread and Reactor.Current: only the reactor can take it back when its thread stops being the reactor's.

What was wrong. BindReactorThread records the reactor thread's managed id and sets t_current, and nothing took either back at the stop. Every test of whether a caller is on the reactor compares that id: OnReactorThread (which the TLS pipes, the http2, http3, nghttp2 and nghttp3 OffReactor tests, httpclient, pg and Kestrel's OnReactor branch on), ScheduleOnReactor, SubmitClientOp, SubmitTimeout, EnqueueReturnQ, EnqueueRecycle, EnqueueFlush, EnqueueReturnQIncremental and UdpSendTo. After a stop two kinds of thread still passed, and took the reactor's own path against a reactor whose ring the teardown had unmapped and whose buffers it had freed:

  • the thread that ran the reactor, if it goes on running after Run returns (a host's own thread, a pool thread): the id and Reactor.Current both still matched;
  • a new thread, once the reactor thread's Thread object has been collected: CoreCLR hands a dead thread's managed id to a new thread.

Reproduced at main 19aec5c with a standalone program, on a reactor with no transport, run on a thread, stopped and joined:

  • On the thread that ran it, after Run returned: OnReactorThread true, Reactor.Current the stopped reactor, and ReactorSwitch.Yield never resumed (3 of 3).
  • On a new thread started after a GC: it was given the stopped reactor thread's id every time (3 of 3), OnReactorThread was true, and ReactorSwitch.Yield never resumed.
  • SubmitTimeout from either: the process aborted with AccessViolationException in Ring.GetSqe, which read the SQ head from the ring the teardown unmapped (2 of 2 each).

The other paths from reading: EnqueueRecycle runs Recycle there, which submits a cancel to the same ring, EnqueueReturnQ writes through the buffer ring pointer the teardown freed and set to null, and UdpSendTo allocates its send blocks again and sends on a descriptor number the stop closed.

The change. The end of Teardown, still on the reactor's thread, sets the recorded id to -1 and clears t_current. A thread's id is its own while it runs, and Teardown runs on the reactor's thread before Run returns (on every path past the ring's creation, see What to weigh), so from then on no thread is taken for the reactor's: not the one that ran it, nor a new one given its id later. -1 is no thread's id, and it is not 0, which SubmitClientOp, SubmitTimeout and UdpSendTo read as "not started", so every caller takes the path any other thread already took after a stop and gets the stopped reactor's answer. The checks themselves are unchanged.

Why the id stays. The issue proposed comparing t_current with the reactor. I built that first, and it works (E2E 732 passed, 0 failed), but it costs every check: a thread static outside CoreLib is reached through the thread's array of static bases (a bounds test, two loads and a null test), where the managed id is read at a fixed offset. Tier-1 code, from DOTNET_JitDisasm:

main t_current compared
OnReactorThread 47 bytes 81 bytes
ScheduleOnReactor 255 276
EnqueueReturnQ 304 332
EnqueueFlush 291 312
main:      call <tls>; add rax, 16; mov eax, [rax+0x10]; cmp eax, [rbx+0x1E0]; jne
t_current: call <tls>; cmp dword [rax], 4; jle <helper>; mov rdi, [rax+8]; mov rax, [rdi+0x20];
           test rax, rax; je <helper>; cmp [rax+0x10], rbx; jne

Invalidating the id gives the same guarantee on every path through Teardown, and with it the tier-1 code of those four methods is byte-identical to main's (addresses aside), so there was nothing for a wrk A/B to measure. A pinned microbenchmark of the check through a non-inlined call could not tell the t_current variant from main on this box either (2.0 to 3.4 ns per call for both, with other suites running).

What a caller notices. Only after a stop, and only on the two kinds of thread above, which now behave as every other thread already did:

  • OnReactorThread is false, and on the thread that ran it Reactor.Current is null. The XML docs of both now say so.
  • ReactorSwitch.Yield(r) throws ObjectDisposedException instead of never resuming. await new ReactorSwitch(r) already threw it on main; it now arrives through the thread pool instead of synchronously.
  • SubmitConnect, SubmitSend, SubmitRecv, SubmitRead, SubmitWrite and SubmitTimeout throw ObjectDisposedException instead of reaching the unmapped ring.
  • UdpSendTo throws the InvalidOperationException it throws on any other thread.
  • ScheduleOnReactor drops the post, as for any other thread (one posted the way a switch posts runs on the thread pool), and EnqueueReturnQ, EnqueueFlush and EnqueueRecycle take their queue paths.
  • The satellites' checks take their off-reactor side there: the TLS pipes, the http2, http3, nghttp2 and nghttp3 OffReactor tests, the Dispose of httpclient's pool and client, pg's queries, pool pick and Dispose, and Kestrel's OnReactor.
  • So a Dispose of httpclient's pool or client, or of a pg connection, made from the thread that ran the reactor after Run returned is now dropped, as one from any other thread already was: it posts itself, and a stopped reactor drops a plain post. It used to run inline there.

Tests. In tests/Ioxide.Tests.E2E/Core/AffinityTests.cs, both on a reactor with no transport:

  • affinity: once Run has returned, the thread that ran the reactor is not its reactor's, so a yield or a ring op from it fails as from any other. The thread that ran the reactor, once Run returns, checks OnReactorThread and Reactor.Current, then yields to the reactor and submits a timeout to it.
    • At main, and with the id reset reverted: the thread that ran it passes OnReactorThread for the stopped reactor
    • With only the t_current clear reverted: Reactor.Current on the thread that ran it is still the stopped reactor
  • affinity: a new thread the runtime gives a stopped reactor thread's managed id is not the reactor's. Runs and stops a reactor on a thread it then drops, collects, and starts threads, each held so its id is not handed out again, until one is given the dead thread's id. At most 4096, and a control assertion fails the test if none is.
    • At main, and with the id reset reverted: a new thread given its id 33 passes OnReactorThread for the stopped reactor
    • With only the t_current clear reverted it passes, as it should: a new thread's t_current is null.

Both make the yield and the ring op only from a thread neither check takes for the reactor's, since on the reactor's path the ring op aborts the process. The revert runs were of AffinityTests alone, out of the built E2E suite (12 passed and 2 failed with the id reset reverted, or both parts; 13 and 1 with the t_current clear reverted).

Suites. Release, pinned to the safe CPUs, one at a time:

  • Unit: 263 passed, 0 failed
  • E2E: 732 passed, 0 failed (built before the last edit, which only rewords a comment in BindReactorThread)
  • Tls: 269 passed, 0 failed, no skips (this box has the tls module)
  • Http: 104 passed, 0 failed
  • File: 13 passed, 0 failed
  • Chaos: 105 passed, 0 failed
  • Pg: 7 passed, 19 skipped, and Redis: 6 skipped, 1 pending (main's), for want of their servers here

The build adds no warnings.

Overlaps.

What to weigh.

  • Left alone: when the kernel refuses the ring, Run returns before Teardown, so there the thread that ran it still passes (checked on this branch). Nothing was mapped or freed, so its path ends in a NullReferenceException from a ring op rather than in freed memory. That path also never closes the posts: a ReactorSwitch.Yield from another thread never resumes, with or without this. Taking the binding back there alone would make a switch from that thread wait forever too, so it needs its posts closed as well, which is a change of its own.
  • _reactorThreadId now has three values: 0 before Run, the thread's id while it runs, -1 after Teardown. One test already sets it to -1 to mean that every caller is another thread (AffinityTests' handoff wake count).
  • The developer docs' description of the hand-off fast path (CurrentManagedThreadId == reactorThreadId) is still accurate, so it is unchanged.

…w thread given that one's managed id is taken for the reactor's (#608)

BindReactorThread records the reactor thread's managed id and sets t_current, and nothing took either back at the stop. Every test of whether a caller is on the reactor compares that id: OnReactorThread, ScheduleOnReactor, SubmitClientOp, SubmitTimeout, the Enqueue fast paths and UdpSendTo. So after a stop two kinds of thread took the reactor's own path against a reactor whose ring the teardown had unmapped: the thread that ran it, if it went on running after Run returned, and any new thread the runtime gave the same id once that thread's Thread object was collected. From either, a ring op read the SQ head from the unmapped ring and the process aborted with AccessViolationException, and a ReactorSwitch.Yield was queued where nothing drains it and never resumed. The thread that ran it also still had the stopped reactor as Reactor.Current.

The end of Teardown, still on the reactor's thread, now sets the recorded id to -1 and clears t_current. A thread's id is its own while it runs, and Teardown runs on the reactor's thread before Run returns, so from then on no thread is taken for the reactor's. -1 is no thread's id, and it is not 0, which the submit and UDP checks read as not started, so every caller hands off and gets the stopped reactor's answer. The checks compile as before. A ring the kernel refuses still returns from Run before Teardown, and that path is unchanged.

Comparing t_current with the reactor instead costs each check: a thread static outside CoreLib is reached through the thread's array of static bases, a bounds test, two loads and a null test, where the managed id is read at a fixed offset. OnReactorThread compiled to 81 bytes against 47, and ScheduleOnReactor, EnqueueReturnQ and EnqueueFlush each 21 to 28 bytes longer.

AffinityTests (E2E): once Run has returned, the thread that ran a reactor is not taken for its reactor's, and neither is a new thread the runtime gives the stopped reactor thread's id; from both, a yield and a ring op throw ObjectDisposedException.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reactor: the reactor-thread checks compare managed thread ids, which the runtime reuses, so after a stop a new thread can pass for the reactor's

1 participant