Skip to content

http2: a stream's window is credited a quarter of it at a time instead of once per DATA frame, so an upload costs one WINDOW_UPDATE per 64 KiB (#630) - #633

Merged
MDA2AV merged 4 commits into
mainfrom
fix-630
Nov 15, 2026

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #630.
Part of #588.

What was wrong

The connection's receive window is topped up once it falls below a quarter, but each stream's window was credited frame by frame (RFC 9113 6.9, 6.9.1):

  • a buffered body, and a streamed body's padding, as each DATA frame arrived;
  • a streamed body as its handler read each chunk, and outside a read pass each of those reads also started a flush of its own;
  • the upload behind a buffered answer parked on the client's credit, as each of its frames was dropped.

So an upload cost one WINDOW_UPDATE per DATA frame however small its frames were: 255 for a 4 MiB body in 16 KiB frames, 4,194 for the same body in 1,000-byte frames.

What changes

All in src/protocols/ioxide.http2; no core change.

  • A stream is credited once what it has consumed and not yet been credited reaches a step, in one WINDOW_UPDATE for all of it. Consumed is read, or for a buffered body and for padding, received. The check runs at every read and at every DATA frame that consumes on arrival.
  • What is owed comes from state the stream already keeps: what it was sent and not credited is its initial window less its ReceiveWindow, and from that the bytes still queued unread in its body reader are taken off. Those are never credited: the stream window is what bounds what a handler holds unread, as before.
  • The step is a quarter of InitialWindowSize, but at most 65,535 bytes and at least one byte. See below for why it is capped.
  • A request that has ended (END_STREAM) is owed nothing, as before, and a stream that is reset or has ended is credited nothing afterwards: what a stream is owed lives on its request, which a reset takes out of the stream table.
  • CreditBody's detached flush outside a pass happens only when a step was staged. A read short of a step writes nothing and flushes nothing.

New state: Http2BodyReader.Unread, the bytes its queue holds. Taking unread bytes off what is owed needs that sum at every read, and summing the queue each time goes quadratic with tiny frames. It is kept by Push, by each hand-over, and zeroed when the queue is dropped.

Removed

  • The credit for the upload behind a buffered answer parked on the client's credit. origin/main gave it one WINDOW_UPDATE per frame as each was dropped, and the first version of this PR kept a count per such stream to credit it in steps: a branch in RetireStream, CreditParkedUpload, _parkedUploads and its Remove in CompleteParkedAsync. Both are gone: HandleData returns for any stream outside the stream table and credits nothing.

In its place: RST_STREAM(NO_ERROR) after a parked answer's END_STREAM

  • An answer that ends at once is followed by RST_STREAM(NO_ERROR) while its request is still arriving, which asks the client to stop sending and leaves it the answer (RFC 9113 8.1). A buffered answer that parked on the client's credit was not, since its stream is retired with the rest still to go and the retire leaves it alone.
  • Now ParkRest reads whether the request is still arriving, before the retire takes the stream out of the stream table, and the parked rest resets the stream with NO_ERROR once its END_STREAM is staged, if the response is still live: not after the peer's reset, nor after one of the stack's own. No connection state.
  • One imprecision: a request END_STREAM that arrives while the answer is parked is not seen, so the reset can follow a stream already closed both ways. RFC 9113 5.1 has the peer ignore a RST_STREAM it receives in that state.

Why it never stalls

A client is blocked on a stream only when its bytes in flight there, sent and not yet credited, fill its view of the stream window. On the server those bytes are either unread, queued in the body reader, or consumed and not yet credited. Whenever the handler has read everything queued, the unread part is zero, so the uncredited part alone is everything in flight. If the client is blocked, that is at least its window, and so at least the step as long as the step fits inside every window the client can be counting. The read that emptied the queue, or for a buffered body or padding the frame that arrived, runs the check and sends the credit. A handler that stops reading stops the upload, which is the backpressure this reader is for. This is about a request still being read; the upload behind an answer whose handler is done with it is under Removed.

The windows a client can count:

  • the RFC's default 65,535 bytes, until it has applied the server's SETTINGS (RFC 9113 6.9.2);
  • InitialWindowSize once it has applied them, which can happen before the server has their acknowledgement (6.5.3), so the server cannot tell which of the two a client counts until then.

A quarter of InitialWindowSize always fits inside InitialWindowSize. But a quarter of the default 256 KiB is 65,536, one byte more than 65,535: a client that has not yet applied the SETTINGS would be blocked at 65,535 bytes in flight with 65,535 owed, one short of a step, and would wait until it applied them. Capping the step at 65,535 makes it fit both windows. The floor of one byte keeps a window below four bytes credited at all.

The cap means a window above 256 KiB is credited every 65,535 bytes rather than every quarter. The server could raise the step to a quarter once the SETTINGS are acknowledged (6.5.3), at the cost of a flag it does not keep today; this PR does not.

What a caller would notice

  • One WINDOW_UPDATE per step instead of one per DATA frame: with the default 256 KiB window, one per 65,535 bytes or more, each a whole step.
  • While credit is owed, a client keeps at least the window less one step to send, three quarters of it for any window up to 256 KiB. A client that has not yet applied the SETTINGS counts 65,535 and can use it all before its first credit.
  • A body that ends before it reaches a step is credited nothing, so a small upload gets no WINDOW_UPDATE at all. It used to get one per frame that arrived before its END_STREAM.
  • A client that uploads its whole body before it reads, given a buffered answer bigger than its stream window by a handler that did not read the upload, now waits until its own timeout, as it would with nginx or Go. A client that reads while it uploads still gets the whole answer, and is then sent RST_STREAM(NO_ERROR), so it stops uploading where it used to be left waiting once the answer had ended.
  • A read outside a pass flushes only when a step is due, so bytes a streamed writer staged meanwhile wait for that writer's own flush instead of leaving with the read's.
  • No public API changes; the InitialWindowSize docs describe the steps.

Tests

Each line gives what fails with its part reverted, the rest of the change in place.

Unit, new Http2CreditStepTests, on the wire with the strict client and a client that keeps its own windows:

  • An upload in 16 KiB frames and one in 1,000-byte frames, 4 MiB each, read as it arrives: the upload completes, and its stream gets one WINDOW_UPDATE per step at most, each a whole step. At origin/main: "256 DATA frames were credited by 255 WINDOW_UPDATEs of [16384, 16384, ...]" and "4195 DATA frames were credited by 4194 WINDOW_UPDATEs of [1000, 1000, ...]".
  • What a handler has not read is never credited: a handler reads one chunk of a full window and holds the rest, and nothing is credited until a step of it is read. With the unread bytes not taken off what is owed: "16384 of the 262144 bytes in its window were read and 262144 were credited".
  • Reads short of a step flush nothing, outside the pass too, with the handler's own headers staged meanwhile, and the read that completes the step is credited in one WINDOW_UPDATE. With CreditBody flushing whether or not a step was staged, and at origin/main: "49152 bytes were read short of a step and 1 flushes went out behind them".
  • A buffered body is credited in steps as it arrives, and MaxRequestBytes still bounds it: a body of exactly MaxRequestBytes is answered, one past it is reset with ENHANCE_YOUR_CALM, and DATA in flight after that reset is credited nothing. At origin/main: "64 DATA frames of a buffered body were credited by [16384, 16384, ...], not in whole steps". The bound passes there too.
  • No stall: a body reader that reads as data arrives, for InitialWindowSize 256 KiB (the default), 1 MiB, 65,536, 65,535, 16,384, 1,000, 100 and 3, with the client never applying the server's SETTINGS, applying them without acknowledging, acknowledging them before the upload, and applying them with one 65,535-byte window in flight and acknowledging after the next. Every upload completes. It passes at origin/main, which credits every frame. With the step uncapped: "InitialWindowSize 262144, SETTINGS Unseen: the upload stopped at 65535 of 787433 bytes, 65535 of them read, the client out of window", and the same at 1 MiB. With the step taken from the server's own window before the acknowledgement, a quarter of 65,535: "InitialWindowSize 1000, SETTINGS Applied: the upload stopped at 1000 of 197606 bytes", and the same for 100 and 3, in every state where the client applied the SETTINGS.
  • Three streams a step short of credit, one ending with END_STREAM, one reset by the peer and one reset by its handler, are credited nothing afterwards, though a frame arrives on each that would make the step. Without the ended check on reads: "credit went out after the end: 65536 to 1, which ended".

The clean close, unit in Http2EarlyAnswerTests and Http2EarlyAnswerGuardTests, and E2E in EarlyAnswerTests:

  • A buffered answer that waited on the client's credit, its request still arriving, is followed by RST_STREAM(NO_ERROR) once its END_STREAM is out, as the stream's last frame. Without the reset: "the answer ended with its request still arriving, and no RST_STREAM asked the client to stop uploading".
  • Control: one to a request that had ended before the answer parked, with a body reader and without, is not reset. Reading only whether the stream is still in the table: "a request that had ended was reset".
  • Guard: no reset back once the peer has reset the stream, nor a second one after the stack's own FLOW_CONTROL_ERROR, both while the answer's first flush is in flight. Without the liveness check: "stream 1 was reset after the peer had reset it".
  • E2E: .NET's HttpClient with a duplex content, which reads while it uploads, posting 64 MiB to a handler that answers a 256 KiB 413 unread: the answer arrives whole and the upload stops under 512 KiB. Without the reset: "the upload neither finished nor stopped, at 262144 bytes".

Removed with the special case: the two tests that had a client finish its upload behind a parked answer, and the one that read _parkedUploads. Http2EarlyAnswerTests' "an upload still arriving while a buffered answer is parked on credit is credited on its stream", which tested origin/main's per-frame credit, now checks that the upload is credited nothing and that a client crediting the answer gets it whole; with origin/main's credit put back it fails with "expected [0], got [65536]".

Existing tests that expected credit at once, each keeping what it proves:

  • Http2FlowControlTests "padding on a streamed body is credited" runs with a window whose step is the padded body's payload (4 x 1,457), and its padding-only frame now arrives once the handler has read the body and parked, so it is that arrival that credits. With the arrival check skipped for a streamed body's padding: "1457 flow-controlled bytes arrived and 0 were credited".
  • Http2FlowControlTests "padding on the buffered path is credited in full" uses the same window, and "a DATA frame past MaxRequestBytes resets its stream, and only the frames taken are credited" a 64 KiB window, whose step is its one frame taken.
  • Http2StreamedRequestTests "credit is returned on read, not on arrival" uses a 4,000-byte window, whose step of 1,000 is its two chunks, and "buffered dispatch still assembles the body" a 1,600-byte window, whose step of 400 is its first frame.
  • Http2ReceiveWindowTests "the frame loop goes on reading while a flush is stalled" sends a step, four frames, instead of two.

With the step taken from the server's own window before the acknowledgement, the first four of these fail as well ("expected [1000], got [0]", "expected [400], got [0]", "1457 flow-controlled bytes arrived and 0 were credited", "expected [1457], got [0]"). With the step uncapped, the existing "every byte comes back once, on the connection's top-up" fails: "stream 1 was sent 1610572675 bytes, read 1610572474 of them, and was credited 1090529176".

Playground/Http2/ManagedStreamedRequest no longer says each read hands back the credit for its chunk, and its docs pane is regenerated with scripts/gen-docs-panes.py.

Suites

Pinned to the safe CPUs, on this head, one suite at a time; other agents were running their own suites on the box meanwhile.

Bench

h2c on an i9-14900K, origin/main (19aec5c) against this branch at 40c9b8f; the commits after it change only an answer parked on credit, which none of these runs takes. The two builds were interleaved, alternating which goes first: the server taskset -c 2,3,6,7 (two P-cores), h2load taskset -c 16-31 (E-cores), 3 s of warm-up and 10 s measured, the server's CPU read from /proc/<pid>/stat over the measured window. Uploads POST to Playground/Http2/ManagedStreamedRequest, which reads its body as it arrives; the GET control is Playground/Http2/ManagedBuffered. Other agents were running their own test suites on the box throughout, load average 1 to 6, and the GET control swung from 4.6M to 6.7M req/s between runs of one binary. No request failed in any run.

Stream WINDOW_UPDATEs per upload, through nghttp -v -d (16 KiB DATA frames):

body origin/main this branch
64 KiB, timing-dependent: over 12 runs 2 to 4 0 or 1
1 MiB 64, the smallest 1 byte 16, the smallest 65,535
16 MiB 1,024 256

Throughput and server CPU per request, medians of the pairs:

cell origin/main this branch
POST 1 MiB, -t8 -c16 -m4, 1 reactor, saturated, 5 pairs 6,365 req/s, 157.4 us 6,363 req/s, 157.5 us
POST 64 KiB, -t8 -c32 -m8, 1 reactor, saturated, 5 pairs 94,946 req/s, 10.57 us 93,462 req/s, 10.74 us
the same, repeated 94,385 req/s, 10.64 us 93,426 req/s, 10.75 us
the same, origin/main against itself 95,651 req/s, 10.50 us 95,635 req/s, 10.50 us
POST 1 MiB at 6,000 req/s, 2 reactors, 4 pairs 184.6 us 184.4 us
POST 64 KiB at 80,000 req/s, 2 reactors, 4 pairs 12.01 us 11.92 us
POST 1 MiB, -t4 -c16 -m4, 2 reactors, 4 pairs 7,651 req/s, 219.8 us, 1.68 cores 7,668 req/s, 229.7 us, 1.76 cores
POST 64 KiB, -t4 -c32 -m8, 2 reactors, 4 pairs 120,625 req/s, 14.37 us, 1.73 cores 121,834 req/s, 12.32 us, 1.50 cores
GET, -t8 -c64 -m32, 2 reactors, 4 pairs 5,974,274 req/s, 0.335 us 5,870,366 req/s, 0.340 us

What it shows: the wire saving is 4 to 1 for 16 KiB frames, and a body that ends inside a step costs no WINDOW_UPDATE. Saturated 64 KiB uploads are about 1% slower on this branch, consistently: lower in 13 of 14 interleaved pairs, 9 of the 10 above and all 4 of a reviewer's re-run, which were lower by 0.7% to 1.8%. The cause is unknown. Otherwise the two builds are level: at a fixed request rate, which compares CPU at equal throughput, and in the saturated 1 MiB cell. The two-reactor upload rows ran at 1.5 to 1.9 of 2 cores, short of saturation; their 64 KiB row's -14% CPU did not reproduce at a fixed rate, and the 1 MiB row's +4.5% drifted with the load from round to round (pairs +5%, -2%, +11%, +2%).

Overlaps

Checked with git merge-tree --write-tree against each branch's head, all based on 19aec5c.

…d of once per DATA frame, so an upload costs one WINDOW_UPDATE per 64 KiB (#630)

A stream's receive window was credited frame by frame: a buffered body and a streamed body's padding
as each DATA frame arrived, a streamed body as its handler read each chunk, and the upload behind a
buffered answer parked on credit as each frame was dropped. Outside a read pass each read also made
a flush of its own.

A stream is now credited once what it has consumed and not yet been credited reaches a step, in one
WINDOW_UPDATE for all of it. The step is a quarter of InitialWindowSize, at most 65,535 bytes and at
least one: until the client has applied the server's SETTINGS it counts the default 65,535-byte
window (RFC 9113 6.9.2), and a quarter of the default 256 KiB is one byte more than that. Consumed is
what the stream was sent and not credited, from its ReceiveWindow, less what its body reader still
holds unread, which a new count on the reader keeps; unread bytes are never credited, so the window
still bounds what a handler holds. A request that has ended is owed nothing, as before, and a stream
that is reset or ended is credited nothing afterwards. A read outside a pass flushes only when a step
is due.

The upload behind a parked buffered answer has no request left to count against, so the connection
keeps a count per such stream, freed when the answer ends or the stream does. It counts what the
handler left unread too: with steps, an upload whose window the handler had nearly filled would
otherwise wait on credit that never comes.
…d uncredited, like any stream its handler is done with (#630)

The count kept for that upload is gone: the branch in RetireStream that started it, CreditParkedUpload,
_parkedUploads and its Remove in CompleteParkedAsync. So is the credit origin/main gave the same upload,
one WINDOW_UPDATE per frame dropped: HandleData returns for any stream outside the stream table.

A client that uploads its whole body before it reads, given a buffered answer bigger than its stream
window by a handler that did not read the upload, now waits until its own timeout. A client that reads
while it uploads still gets the whole answer.

Tests: the two that had a client finish its upload behind a parked answer, and the one that read the
count, tested only what is removed and are gone. Http2EarlyAnswerTests' parked test now checks that the
upload is credited nothing and that a client crediting the answer gets it whole; with origin/main's
per-frame credit put back it fails with "expected [0], got [65536]".
…T_STREAM(NO_ERROR) while its request is still arriving (#588)

A buffered answer bigger than the client's stream window parks the rest on the client's credit, and its
stream is retired with the rest still to go, so the retire does not reset it. An answer that ends at once
is followed by RST_STREAM(NO_ERROR) once its END_STREAM is out, which asks the client to stop sending a
request the server is done with and leaves it the answer (RFC 9113 8.1); a parked one was not. With that
upload no longer credited, a client that reads while it uploads got the whole answer and then had its
upload stall.

ParkRest now reads whether the request is still arriving, before the retire takes the stream out of the
stream table, and the parked rest resets the stream with NO_ERROR once its END_STREAM is staged, if the
response is still live: not after the peer's reset, nor after one of the stack's own. No connection state.

One imprecision: a request END_STREAM that arrives while the answer is parked is not seen, so the reset can
follow a stream already closed both ways; RFC 9113 5.1 has the peer ignore a RST_STREAM in that state.

Tests:
- Unit: a buffered answer that waited on credit, its request still arriving, is followed by
  RST_STREAM(NO_ERROR) after its END_STREAM, as the stream's last frame. Without the reset: "the answer ended
  with its request still arriving, and no RST_STREAM asked the client to stop uploading".
- Unit control: a request that had ended before its answer parked, with a body reader and without, gets no
  reset. Reading only whether the stream is in the table: "a request that had ended was reset".
- Unit guard: no reset back once the peer has reset the stream, nor a second after the stack's own
  FLOW_CONTROL_ERROR, both while the answer's first flush is in flight. Without the liveness check: "stream 1
  was reset after the peer had reset it".
- E2E: .NET's HttpClient with a duplex content, which reads while it uploads, posting 64 MiB to a handler that
  answers a 256 KiB 413 unread: the answer arrives whole and the upload stops under 512 KiB. Without the
  reset: "the upload neither finished nor stopped, at 262144 bytes".
# Conflicts:
#	tests/Ioxide.Tests.Unit/Http2EarlyAnswerTests.cs
@MDA2AV
MDA2AV merged commit 7c6a98e into main Nov 15, 2026
2 checks passed
@MDA2AV
MDA2AV deleted the fix-630 branch November 15, 2026 16:41
@MDA2AV MDA2AV mentioned this pull request Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

http2: each stream's receive window is credited with one WINDOW_UPDATE per DATA frame

1 participant