Skip to content

tests/http2: "h2 RunAsync begun on the thread pool (http2)" fails now and then, once with FRAME_SIZE_ERROR from the server #614

Closed

Description

@MDA2AV

"h2 RunAsync begun on the thread pool (http2): an upload read and echoed from the pool arrives whole, served on the reactor" (OffReactorH2FixTests.cs:71-73) failed in 2 of the 24 full E2E logs kept from today's #581-#592 batch, all on trees with #593, and once more in a review run. It passed 170 rounds run alone, and its nghttp2 twin passed in every run.

Likely cause (a hypothesis, from reading). #593 turned the preface's await FlushAsync() into await FlushPassAsync(), which starts the pump detached and returns at once (Http2Connection.cs:194-196, Http2Connection.Write.cs:95-113). The old await parked the loop on the SETTINGS flush, which a TCP pipe completes on the reactor (TcpConnectionPipeWriter.cs:76-78), so the first read was always issued from there; the nghttp2 twin still works that way (Nghttp2Connection.RunBuffered.cs:48-56). A RunAsync begun on the pool now issues its first read from the pool:

The class is reactor-only (Http2Connection.cs:28). Stage, the queue and the pump share _flushing, _queued and the pipe writer with no lock, the pump empties the queue by resetting _queuedUsed, and a frame's header and payload are separate Stage calls (Http2Connection.Write.cs:47-74, :153-155, :409-417). Two threads staging at once can put another frame between a header and its payload, or drop queued bytes; the client then reads echoed upload bytes as a frame header, whose length is far past its 16,384 limit: FRAME_SIZE_ERROR. The test's transport counts only calls made after a read completed on the reactor (OffReactorH2FixTests.cs:328-342), so a first pass on the pool goes unseen, and handler calls still running inline after that are the "calls from elsewhere". On the reactor alone the read-during-flush path keeps stage order; I found no single-threaded way for it to tear a frame.

Fix. Confirm it first: hold RunAsync on the pool until the client's preface has arrived (a short delay before serve in StartH2, OffReactorH2FixTests.cs:234-244), so the first read completes there. If the test then fails, let no pass run while _reactor is null, for example by awaiting the preface flush while the reactor is unknown, as before #593 and as the nghttp2 twin does.

Seen by #583's implementer, locally and in #600's CI, and by #604's reviewer. The cause is from reading at main 19aec5c; not reproduced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingseverity:mediumPerf cliff / hardening gap

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions