Skip to content

http2: a RunAsync begun off the reactor reads only once its SETTINGS are out, so a request already in is served on the reactor (#614) - #627

Merged
MDA2AV merged 1 commit into
mainfrom
fix-614
Nov 15, 2026
Merged

MDA2AV merged 1 commit into
mainfrom
fix-614

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Fixes #614.

What was wrong. A RunAsync begun on the thread pool knows no reactor: Reactor.Current is null there, and the loop learns its reactor from the first read that completes on it. Before #593 the loop awaited its SETTINGS flush, which a TCP pipe completes on the reactor, so its first read was made from the reactor. #593 starts that flush detached, as it does each pass's flush, so the first read was made from the pool. A client need not wait for the server's preface (RFC 9113 3.4): when the client's preface and request were already in, that read completed at once on the pool, and the pass ran there with no reactor known. OffReactor was then false, so the handler's WriteHeaders, flushes and body reads ran inline on its own pool thread while the SETTINGS flush resumed on the reactor. Stage, the queue and the pump are reactor-only and share _flushing, _queued and the pipe writer unlocked, so frames were torn. The issue's hypothesis is right.

Reproduced at main 19aec5c, not only read, with a scratch copy of the late-start test that also records where the handler was dispatched (bench/.work/repro614 in the worktree, built and run pinned):

  • The test as it is: the handler was dispatched off the reactor in 14 of 60 rounds, and 1 round failed visibly, "the transport was only called from its reactor (3 calls from elsewhere)".
  • RunAsync held on the pool for 100 ms before it begins, as the issue suggested: dispatched off the reactor in 20 of 20 rounds, 1 failure (PROTOCOL_ERROR).
  • The client's HEADERS and first DATA left unread in the TCP pipe until RunAsync begins on the pool: dispatched off the reactor in 320 of 320 rounds, 76 failures: FRAME_SIZE_ERROR, PROTOCOL_ERROR, "Error while copying content to a stream", "N calls from elsewhere", a truncated echo, and echoes of the right length whose bytes differ from offset 16384 or 32768 on.
  • The nghttp2 twin, which awaits its preface flush, held the same way: dispatched on the reactor in 20 of 20.

What changes. Http2Connection.RunCoreAsync: begun off the reactor (_reactor null), the loop awaits its SETTINGS flush (FlushAsync), as it did before #593 and as Nghttp2Connection does, so its first read is made where that flush completed: over a TCP pipe, the reactor. Begun on the reactor, it still starts that flush and reads on behind it (FlushPassAsync). With the fix, the same scratch: the test as it is, 1,000 of 1,000 rounds dispatched on the reactor on each stack, none failed; the request held, 999 of 1,000 for http2 (the one left is under "What is left").

Behaviour a caller would notice. A RunAsync begun off a reactor thread, which includes one over a pipe with no reactor at all, makes its first read only once its SETTINGS flush has completed, as before #593. Over a transport that completes a flush only once its peer reads, that first read waits for it. Begun on the reactor, nothing changes.

Tests. OffReactorH2FixTests: the two "h2 RunAsync begun on the thread pool" tests (http2, nghttp2) also assert that the connection made no read before its SETTINGS flush completed; the test's transport counts the reads made before its first flush is done. With the fix reverted, the http2 one fails in 10 of 10 rounds of the file: the connection read only once its SETTINGS were out (1 reads made before that flush completed), or 2 when the harness's listen probe read early too. The nghttp2 twin passes either way. With the fix, 80 of 80 over the same 10 rounds. The check is on that order rather than on where the handler was dispatched: a dispatch can still land off the reactor in the window below, so a test on it would be flaky.

Suites (Release, pinned to the safe CPUs): Unit 263 passed, 0 failed; E2E 730 passed, 0 failed; Chaos 105 passed, 0 failed; Tls 269 passed, 0 failed.

What is left. Learning the reactor from the thread a completion runs on cannot be made certain from an IDuplexPipe. When the reactor finishes the SETTINGS send before the pool thread has awaited it, the flush is seen complete on the pool, and a first read that completes at once there runs the pass off the reactor as before. Two ways in showed up: the eventfd write that hands the reactor the flush wakes it, and it can preempt a pool thread on its CPU; and a process's first such connection, most likely because its pool thread is still compiling the await path. The nghttp2 twin has had this window all along. With the request held: http2 with this fix, 1 of 1,000 rounds, the first one; nghttp2 at main, 19 of 1,000, 15 of them with the pool thread on the reactor's CPU. That run had the pool thread there in 58 rounds against 1 in the http2 run, so the rate follows the box's load. With the test's own timing it was 0 of 1,000 on each stack: after a lost race the first read usually still parks, and completes on the reactor. Closing it needs the connection told its reactor before its first pass, for example by the TcpConnection constructor taking the connection's reactor (internal to the core today) and an IDuplexPipe overload taking one, or by a way for a pipe to name its reactor. Each is an API decision and would cover nghttp2 as well, so it is not in this PR.

Overlaps. #603 (fix-592) and #604 (fix-588) change Http2Connection.cs in DispatchBuffered, CompleteBufferedAsync and RetireStream; this changes the start of RunCoreAsync only. git merge-tree --write-tree origin/fix-<n> HEAD is clean (exit 0) against fix-588, fix-592 and every other open PR branch, fix-581 to fix-592, so either order merges as is. No open PR touches OffReactorH2FixTests.cs. Of the issues in progress, #615 points at Http2Connection.cs near RetireStream, away from this change.

For the reviewer.

  • The awaited path also covers a RunAsync with no reactor at all, so the Unit tests over StrictClient now take it for the preface; all 263 pass unchanged.
  • The SETTINGS are still written from the pool, before the connection can know its reactor, as the test's transport has always allowed.

…are out, so a request already in is served on the reactor (#614)

A RunAsync begun on the thread pool knows no reactor and learns it from the first read that completes there. Before #593 the loop awaited its SETTINGS flush, which a TCP pipe completes on the reactor, so that first read was made from the reactor. #593 started the preface flush detached, as each pass's flush now is, so the first read was made from the pool, and completed there at once when the client's preface and request were already in: a client need not wait for the server's preface (RFC 9113 3.4). That pass then ran on the pool with no reactor known, so the handler's writes, flushes and body reads ran inline on its own pool thread while the SETTINGS flush resumed on the reactor. Stage, the queue and the pump are reactor-only, so frames were torn: the client's FRAME_SIZE_ERROR and PROTOCOL_ERROR, and echoes that came back with other bytes.

A RunAsync begun off the reactor awaits its SETTINGS flush again, as the nghttp2 twin does. Begun on the reactor, the loop still reads on behind that flush.

OffReactorH2FixTests: the two "h2 RunAsync begun on the thread pool" tests also check that the connection made no read before its SETTINGS flush completed. With the fix reverted the http2 one fails in 10 of 10 rounds, "the connection read only once its SETTINGS were out (1 reads made before that flush completed)"; the nghttp2 twin passes either way.
@MDA2AV
MDA2AV merged commit 22f5d04 into main Nov 15, 2026
2 checks passed
@MDA2AV
MDA2AV deleted the fix-614 branch November 15, 2026 16:40
@MDA2AV MDA2AV mentioned this pull request Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests/http2: "h2 RunAsync begun on the thread pool (http2)" fails now and then, once with FRAME_SIZE_ERROR from the server

1 participant