Skip to content

nghttp3, http2, http3, nghttp2: the streamed writers stage what was written to the buffer handed out last, and refuse a hint or an Advance past what a block holds rather than wrap the sum (#617) - #631

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-617

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #617.

What was wrong

IBufferWriter<byte> has Advance apply to the buffer handed out last, and GetSpan and GetMemory return at least sizeHint bytes or throw. The streamed response writers broke both, and a third check had the same flaw as the second:

  • nghttp3: Advance after GetMemory then GetSpan. GetMemory hands out a managed scratch array and marks it outstanding, and Advance copies that scratch into the native staging block. GetSpan left the mark set, so GetMemory(16), GetSpan(3) filled with "abc", Advance(3) staged three bytes of the scratch over "abc". Reproduced at main: the client received [1, 13, 0].
  • All four writers: the room check summed in int. EnsureStaging computed needed = _staged + sizeHint. With bytes staged and a hint near int.MaxValue the sum wrapped negative, the check passed, and GetSpan or GetMemory returned the rest of the block instead of throwing. Reproduced at main with 16 bytes staged and a hint of int.MaxValue: http2, http3 and nghttp3 handed out 16368 bytes and nghttp2 4080. (nghttp3's GetMemory rents its scratch from the pool, which already refused.)
  • Found while fixing that: Advance's bounds check had the same int sum in http2, http3 and nghttp3 (_staged + count > capacity). Advance(int.MaxValue) after a span was accepted and left the staged count negative, reproduced in all three. On nghttp3 a flush then handed nghttp3 that count as a size_t length, which it refused with NGHTTP3_ERR_STREAM_DATA_OVERFLOW, failing the connection (reproduced). From reading, http2 then ends the stream on an empty body, dropping the bytes staged before, and http3 fails the response when its flush rents a negative length.

What changes

  • nghttp3 GetSpan clears the outstanding-Memory mark, so Advance stages the span.
  • EnsureStaging in all four writers checks room as (uint)_staged + (uint)sizeHint > (uint)capacity: both are non-negative ints, so the uint sum cannot wrap. Growth moves to a NoInlining GrowStaging, which throws OutOfMemoryException when the staged bytes and the hint pass what a block can hold: Array.MaxLength for the managed arrays (http2, http3, nghttp2), int.MaxValue for nghttp3's native block, as no span is longer.
  • Advance in http2, http3 and nghttp3 checks its bound the same way.

Left as it is: nghttp2's Advance has no bounds check at all (_staged += count). Adding one is new validation on its hot path rather than a fix of a check that exists, so it is not in this PR; its over-advance still surfaces later, from a span slice, as ArgumentOutOfRangeException.

Behaviour a caller would notice

  • nghttp3: GetMemory, then GetSpan, then Advance sends what was written into the span.
  • GetSpan or GetMemory with a hint that, beside what is staged, passes what a block can hold throws OutOfMemoryException, as ArrayBufferWriter<T> does, instead of returning less than the hint. Flushing first and asking again gets a fresh block.
  • Advance with a count that wrapped the sum throws InvalidOperationException, as any Advance past the span already did.
  • One edge, a caller's bug either way: on nghttp3 an Advance after GetMemory, with a count past the Memory and a sum past int.MaxValue, now throws OutOfMemoryException from the growth where it threw ArgumentOutOfRangeException from slicing the scratch.

The hot path (GetSpan, Advance)

GetSpan and Advance cost what they did:

  • The room check and Advance's bound add in uint instead of int, and the JIT emits the same add and compare with an unsigned branch. In the disassembly of a GetSpan(16)/Advance(16) loop on http2, the hot loop at main and on this branch is the same instructions in the same order, condition codes aside.
  • nghttp3's GetSpan gains one byte store, the one that clears the Memory mark.
  • Growth moves out of line, and has to. At main the tier-1 inliner, with PGO, inlined EnsureStaging whole into GetSpan, growth included (115 to 132 bytes of IL). A first cut of this fix with the guard and its message inline took it to about 200 bytes: it stopped being inlined, and a GetSpan plus Advance went from about 0.7 ns to 1.6 ns. With growth in a NoInlining GrowStaging what inlines is the check alone, 26 to 35 bytes of IL.

Measured with BenchmarkDotNet (in process, one P-core, one writer per process, main and this branch alternated), in ns per GetSpan(16) plus Advance(16) on a bare writer, and per 16-byte Write through IBufferWriter<byte>:

Writer GetSpan + Advance, main this branch Write, main this branch
http2 0.772, 0.773 0.773, 0.774 1.759, 1.781 1.762, 1.757
http3 0.793 to 0.817 (5 runs) 0.773 to 0.782 (3 runs) 1.789 to 1.818 (5 runs) 1.751 to 1.772 (5 runs)
nghttp2 0.702, 0.707 0.660, 0.663 1.668, 1.698 1.673, 1.677
nghttp3 0.923, 0.924 0.857, 0.895 2.047, 2.051 2.035, 2.062

Two more http3 runs of this branch were disturbed by other sessions' load on the box (standard deviations of 0.27 and 0.45 ns, iterations swinging from 0.8 to 3.4 ns within one process) and are left out of the table. Before this I timed a hand-rolled loop, which came out 12% slower on http2, whose loop is instruction for instruction main's, and 17% faster on nghttp3: that is where the loop lands rather than what it costs, and why the table above is BenchmarkDotNet's.

Tests

Each part of the fix was reverted on its own, its test run, and the file restored:

Part reverted Test Fails with
nghttp3 GetSpan leaves the Memory mark h3 body: a span asked for after a Memory is what Advance sends (E2E) body content differs at offset 0
nghttp3 room check summed in int h3 writer: GetSpan and GetMemory refuse a hint the staging block cannot hold beside what it has staged, rather than hand out less (E2E) expected [GetSpan refused, GetMemory refused], got [GetSpan handed out 16368 bytes, GetMemory refused]
nghttp3 GrowStaging without its guard same same
nghttp3 Advance bound summed in int h3 writer: an Advance past the span it was handed is refused, however large its count (E2E) expected [refused], got [advanced]
http3 room check summed in int http3 writer: GetSpan and GetMemory refuse a hint the staging array cannot hold beside what it has staged, rather than hand out less (E2E) expected [GetSpan refused, GetMemory refused], got [GetSpan handed out 16368 bytes, GetMemory handed out 16368 bytes]
http3 GrowStaging without its guard same same
http3 Advance bound summed in int http3 writer: an Advance past the span it was handed is refused, however large its count (E2E) expected [refused], got [advanced]
http2 room check summed in int h2 writer: GetSpan and GetMemory refuse a hint the staging array cannot hold beside what it has staged, rather than hand out less (Unit) expected [GetSpan refused, GetMemory refused], got [GetSpan handed out 16368 bytes, GetMemory handed out 16368 bytes]
http2 GrowStaging without its guard same same
http2 Advance bound summed in int h2 writer: an Advance past the span it was handed is refused, however large its count (Unit) expected [refused], got [advanced]
nghttp2 room check summed in int nghttp2 writer: GetSpan and GetMemory refuse a hint the staging array cannot hold beside what it has staged, rather than hand out less (Unit) expected [GetSpan refused, GetMemory refused], got [GetSpan handed out 4080 bytes, GetMemory handed out 4080 bytes]
nghttp2 GrowStaging without its guard same expected [GetSpan refused, GetMemory refused], got [GetSpan handed out 8176 bytes, GetMemory handed out 16368 bytes]

Each revert failed that one test and nothing else (Unit 265 of 266, the two touched E2E classes 14 of 15). The Memory in the first test is filled with # before the span is written, so the bytes sent instead of the span are the same whatever the pool's array held.

The probes live in the harness (BufferWriterProbes): each stages 16 bytes, reaches past the block, reports what the writer did, and the test then checks those 16 bytes are still what the response carries. The h2 and nghttp2 tests run in Unit against the in-process strict client; the h3 and http3 ones in E2E against a real QUIC server and the nghttp3 test client.

Suites

Built (ioxide.slnx, Release) and run pinned to the safe CPUs, one suite at a time:

  • Unit: 266 passed, 0 failed (three new).
  • E2E: 735 passed, 0 failed (five new). Run on the commit before a last amend that only moved where tests/Ioxide.Tests.Unit/Program.cs registers the new unit tests.
  • Chaos: 105 passed, 0 failed.
  • Tls, Http, File, Pg and Redis were not run: nothing they cover changed, and the harness only gains a class no other suite calls.

Overlaps

For the reviewer

  • The exception for a hint no block can hold is OutOfMemoryException, as ArrayBufferWriter<T> throws and as the core's write slab does when it cannot grow that far. ArgumentOutOfRangeException on the hint would be the other reading, but the hint alone can be fine: it is the hint beside what is staged that does not fit, and a flush makes room.
  • nghttp3's GetMemory hands out its scratch whatever is staged, so a Memory the block cannot take beside what is staged (more than a GiB staged) is refused at Advance, by the growth, not at GetMemory.
  • nghttp2's unchecked Advance is left as it is (above).
  • Nothing under src/ioxide changes.

…ritten to the buffer handed out last, and refuse a hint or an Advance past what a block holds rather than wrap the sum (#617)

IBufferWriter has Advance apply to the buffer handed out last, and GetSpan
and GetMemory return at least the hint or throw.

nghttp3: GetMemory marked its scratch as outstanding and GetSpan left the
mark, so GetMemory, GetSpan, Advance copied the scratch over the bytes
written into the span. GetSpan now ends an outstanding Memory.

All four writers checked room as _staged + hint in int. Beside staged bytes
a hint near int.MaxValue wrapped that sum negative, the check passed, and
GetSpan or GetMemory handed out the rest of the block instead of throwing.
The room check is now done in uint, where two non-negative ints cannot wrap,
and growth refuses a block past Array.MaxLength (int.MaxValue for nghttp3's
native block) with OutOfMemoryException.

Advance in http2, http3 and nghttp3 had the same int sum in its bounds
check, so a count near int.MaxValue was accepted and left the staged count
negative; it is checked in uint too.

Growth moves out of line, so the room check still inlines into GetSpan.
@MDA2AV

MDA2AV commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Closed: both cases need call patterns no handler uses (GetMemory then GetSpan with no Advance between them, and size hints near int.MaxValue).

@MDA2AV MDA2AV closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant