Repository navigation
Conversation
…top sending the rest (#613) A handler that takes an Nghttp2BodyReader runs at its request's HEADERS and can answer before the body has arrived: a 401, a 413, a redirect. Its response then ended with END_STREAM alone, and nghttp2 sends no reset of its own, so the stream stayed half-closed (local) and every DATA frame the client went on sending was credited and dropped: a 64 MiB HttpClient POST answered with a 413 its handler never read still uploaded all 64 MiB. RFC 9113 8.1 lets a server that has sent a complete response ask the client to stop with RST_STREAM(NO_ERROR) after its END_STREAM; ioxide.http2 has done so since #582. The shim now does it. nghttp2 sends a queued RST_STREAM ahead of DATA still queued, so the reset is submitted from an on_frame_send callback once a HEADERS or DATA frame carrying the response's END_STREAM has gone out, and only while the request is still arriving (nghttp2_session_get_stream_remote_close is 0). It leaves in the same drain, right behind the END_STREAM. The callback is registered on server sessions with manual flow control only: those are the sessions whose handlers are dispatched before their request ends, and the only ones whose on_stream_close reports a NO_ERROR close after the response's END_STREAM, which is how the managed side lets go of the request. DATA the client had in flight is then ignored by nghttp2 and credited on the connection only, never on the closed stream. Both libioxide_nghttp2.so are rebuilt with scripts/build-nghttp2-native.sh at its pinned nghttp2 (500edc92, 1.70.90) and default WORK: linux-x64 on the host with GCC 13.3.0, linux-arm64 in ubuntu:24.04 with gcc-13-aarch64-linux-gnu 13.3.0. Each toolchain first rebuilt main's shim byte for byte, and two builds of this one are identical. Each library differs from main's by the new callback and the grown ih2_new; NEEDED, the GLIBC 2.17 floor and the 15 ih2_ exports are unchanged. Tests: - Unit Nghttp2EarlyAnswerTests, on the wire: a buffered answer, a bodyless one (END_STREAM on its HEADERS), a streamed one, the 500 of a handler that faulted, and an answer given behind a flush in flight are each followed by one RST_STREAM(NO_ERROR) after their END_STREAM, as the stream's last frame. Controls: a request ended by its last DATA, by trailers, by an empty DATA, or by its HEADERS is not reset. Guards: an answer parked on credit is reset only after its END_STREAM and arrives whole; an upload the peer reset gets nothing back, buffered or streamed; a handler that fails after its headers is reset once, with INTERNAL_ERROR; and DATA in flight after the reset draws no frame on its stream and is credited on the connection. - E2E Nghttp2EarlyAnswerTests, with .NET's HttpClient over h2c posting 64 MiB from a StreamContent to a handler that answers 413 unread, streamed, buffered and bodyless: the answer arrives whole and the upload stops, at 64 KiB in a scratch run. Control: an upload read to its end is sent whole and answered. - Chaos: H2cClient's Upload reports an answer followed by RST_STREAM(NO_ERROR) as Stopped, where it returned Reset with code 0. The unread-uploads test accepts that end on its two unread paths. Against main's library, the five reset tests and the two guards that need the reset fail with "the answer ended with its request still arriving, and no RST_STREAM asked the client to stop uploading", and the three E2E stop tests with "the client uploaded 67108864 of 67108864 bytes", while the controls and the other guards pass. With the END_STREAM condition dropped, the reset follows the response's HEADERS and cuts its body off: six tests fail. With the fix, the chaos test as it was fails: an unread upload ends Reset (error 0x0).
# Conflicts: # src/protocols/ioxide.nghttp2/runtimes/linux-arm64/native/libioxide_nghttp2.so # src/protocols/ioxide.nghttp2/runtimes/linux-x64/native/libioxide_nghttp2.so
Owner
Author
|
Merged main in after #594, which changed the same shim: |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #613.
What was wrong
Nghttp2BodyReaderis dispatched at its request's HEADERS, so it can answer before the body has arrived: a 401, a 413, a redirect.Reproduced at main (19aec5c) with a scratch program: .NET's HttpClient over h2c, through a proxy that counts frames, posts 64 MiB from a non-duplex
StreamContentto a body-reader handler that answers 413 without reading.The controls are the same on both: a body-reader handler that reads an 8 MiB upload to its end, and a handler without a body reader given 4 MiB, get the whole upload and no reset.
The fix
All in
src/protocols/ioxide.nghttp2/native/ioxide_nghttp2_shim.c, plus the two rebuilt libraries. No managed code and nothing insrc/ioxidechanges.on_frame_send. After a HEADERS or DATA frame carrying END_STREAM has gone out, ifnghttp2_session_get_stream_remote_close()is 0 (the request is still arriving), it callsnghttp2_submit_rst_stream(..., NGHTTP2_NO_ERROR).mem_sendrunson_frame_sendbefore it hands the END_STREAM frame back, so the reset leaves in the same drain, right behind it. An answer parked on the client's credit is reset only once its last DATA goes.RunAsyncwhose handler takes a body reader:on_stream_closereport a NO_ERROR close after our END_STREAM. That is how the request leaves_pending(CallbackStreamError); its handler has returned by then.Native libraries
runtimes/linux-x64andruntimes/linux-arm64are rebuilt withscripts/build-nghttp2-native.sh, unchanged, at its pinned nghttp2 ref (500edc92, 1.70.90) and its defaultWORK:RID=linux-arm64in a throwawayubuntu:24.04container:gcc-13-aarch64-linux-gnu13.3.0-6ubuntu2~24.04.1cross1,binutils-aarch64-linux-gnu2.42-4ubuntu2.10, cmake 3.28.3,DOCKER_CONFIGpointed at a scratch directory.5db0c664b48293584f36f7e485b7671230303a3a757b98822fd68168912b46e95c79c0a7366c03d17a4e238789107f65615a8f378344888fe7b12e93d3a28d4568a99fa8cf516d5c063e8c6fe4abfa804a96518117001dff22ad15cd0ad89d45d7dbe1957953dc4172429103f8cb7afed0dcb3708311b5364065cdb09b6284f1on_frame_send(0x57 bytes on x64, 0x60 on arm64) and a longerih2_new.ih2_symbols. Both built without a warning, and the shim compiles clean under-Wall -Wextrawith either compiler.Behaviour a caller would notice
Nghttp2BodyReaderthat answers before its request's END_STREAM now has its stream reset with RST_STREAM(NO_ERROR) right after the answer's END_STREAM. The client gets the whole answer and stops uploading. DATA and trailers it already had in flight are ignored. The request leaves the connection at that reset, not at the client's END_STREAM.Tests
New:
Nghttp2EarlyAnswerTests(Unit), on the wire over the strict test transport. The response's:statusis read with ioxide.http2's HPACK decoder.a buffered answer to an upload still arriving is followed by RST_STREAM(NO_ERROR)(END_STREAM on DATA)a bodyless answer, END_STREAM on its HEADERS, to an upload still arriving is followed by RST_STREAM(NO_ERROR)a streamed answer to an upload still arriving is followed by RST_STREAM(NO_ERROR)the 500 of a handler that faulted with its upload still arriving is followed by RST_STREAM(NO_ERROR)(parked, so written by its own drain)with a flush in flight, the reset still follows END_STREAM, oncea buffered answer parked on credit is reset only after its END_STREAM, and arrives whole. The upload goes on, and is credited, while the answer waits.a buffered / streamed answer to an upload the peer reset gets nothing back(RFC 9113 5.4.2)a handler that fails after its headers, its upload still arriving, is reset once, with INTERNAL_ERRORDATA in flight after the reset comes back on the connection, never on its closed stream. A connection window's worth sent after the reset draws no frame on stream 1. Then stream 3's body fits only if that DATA was credited on the connection.New:
Nghttp2EarlyAnswerTests(E2E), with .NET's HttpClient over h2c posting 64 MiB from aStreamContentto a handler that answers 413 unread:Changed, Chaos:
H2cClient.Uploadreports HEADERS followed by RST_STREAM(NO_ERROR) as a newUploadEnd.Stopped, where it used to returnResetwith code 0. No caller expected that.nghttp2 streamed request (...): uploads a handler never reads give their connection credit backsends its unread uploads to handlers that answer without reading. Those uploads can now be stopped at their answer, so the test acceptsStoppedon its two unread paths./readmust still be answered whole.Reverting the fix.
the answer ended with its request still arriving, and no RST_STREAM asked the client to stop uploading. The controls and the other guards pass. E2E: the three stop tests fail withthe client uploaded 67108864 of 67108864 bytes to a handler that read none of them: nothing asked it to stop, and the control passes. That run used a scratch runner that compiles just this file, since the E2E suite takes no filterthe answer never endedfor the buffered, streamed, flush-in-flight and in-flight-DATA tests,expected [16384], got [0]for the answer parked on credit, andstream 1 was reset with [0x0], where the stack's own reset is one 0x2for the handler failing after its headersWith the fix, main's version of the chaos test fails on both flavors:
/unread: upload 5 of 5, 160 KiB already sent, ended Reset (error 0x0) after 36793 of 40960 bytes/unread: upload 3 of 5, 80 KiB already sent, ended Reset (error 0x0) after 16384 of 40960 bytesThe uploads before those ended before their answer, which is why the new version accepts either end. The new version passes against main's library and with the fix.
Suites
With the fix, pinned to the safe CPUs:
Overlaps
git merge-tree --write-treeof this branch with #594 to #604 and the open PRs from #622 to #633:nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594 (nghttp2: a request doesn't say whether it has a body, because the shim reports the end of the headers without END_STREAM #581) and nghttp2: the connection's receive window opens to 2^31-1 behind the SETTINGS, so a connection's request bodies in flight are its streams times their window (#612) #632 (nghttp2: the connection's receive window stays at 65,535, so a connection has at most 64 KiB of request body in flight across its streams #612) change this shim too and rebuild the same two libraries. Only
libioxide_nghttp2.soconflicts, for both RIDs, as binaries. The shim merges cleanly with either one:on_end_headersandon_frame_recv.ih2_new's signature and SETTINGS andih2_server_new, and addsih2_abi.on_frame_sendand its registration.Whichever of the three merges later should merge main and rebuild both libraries from the merged shim (main's rebuilt byte for byte first), rather than pick a side.
nghttp2: the connection's receive window opens to 2^31-1 behind the SETTINGS, so a connection's request bodies in flight are its streams times their window (#612) #632 also changes the chaos file this PR touches. It resizes
UnreadUploadsGiveCreditBackto 100 KiB uploads past a 65,535-byte stream window, and opens the connection window to 2^31-1. The two merge cleanly, since this PR changes only that test's loop assertion. A stopped upload needs no more stream credit, so nghttp2: the connection's receive window opens to 2^31-1 behind the SETTINGS, so a connection's request bodies in flight are its streams times their window (#612) #632's 100 KiB unread uploads simply end at their stop.Program.cs. Every PR adds one registration line to
tests/Ioxide.Tests.Unit/Program.csortests/Ioxide.Tests.E2E/Program.cs, and this one merges with all of them. Its Unit line sits afterNghttp2ResetResponseTests, where it first conflicted with http2: DATA or HEADERS after a request's END_STREAM resets its stream with STREAM_CLOSED, and HEADERS without END_STREAM on a body-reader request with PROTOCOL_ERROR (#615) #626's.EarlyAnswerTests. http3: a response that ends before its request asks the client to stop sending the rest (#587) #601 and http2: an answer waiting on the client's credit has its unread upload credited, so a client that uploads everything first gets it (#588) #604 rework
EarlyAnswerTests.cs, so this PR's E2E tests are in a file of their own.Semantics. nghttp2, nghttp3, http3: a buffered handler that answers null gets a 500 on its own stream instead of ending the connection or going unanswered (#616) #628 (a null buffered answer gets a 500) touches
Nghttp2Connection.RunBuffered.cs, not these files. Its 500, given with an upload still arriving, is now followed by the reset like any other answer.The others merge cleanly and share at most a
Program.cswith this one.For the reviewer
nghttp2_submit_rst_streamqueues its reset withcontinue_without_stream = 0. A stream whose request already ended closes insidesession_after_frame_sent1, right after our END_STREAM. So nghttp2 drops the queued reset insession_prep_framebefore it leaves, and with the condition removed every test still passes.ih2_writeloop, before anything is read, so nothing can arrive in between. The bodyless tests cover that path.CallbackStreamErrornow also runs for our own NO_ERROR reset, after the answer is out. It aborts the reader, which credits once whatever the handler left unread, as its retire would have, and marks the completed writer gone (OnPeerReset). Its "the peer gave up" comment was already loose: it also hears nghttp2's own resets and the INTERNAL_ERROR ones.ih2_client_newsharesih2_new, but the callback is registered only for a server. Nothing managed calls the client.