Skip to content

Update ioxide to 0.27.637, serialize websocket writes by default, and refuse them once the handler has ended - #953

Draft
MDA2AV wants to merge 19 commits into
mainfrom
update-ioxide-0.20.396
Draft

MDA2AV wants to merge 19 commits into
mainfrom
update-ioxide-0.20.396

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Bumps all six ioxide package references from 0.15.251 to 0.27.637, adapts the engine to each release, and fixes ten open issues against the engine and the websockets module. One commit per area, plus two for the Sonar findings. Rebased onto main, whose own Expect: 100-continue handling (#965) replaces the branch's (#928, moved to #958 before), so its two commits are gone, and so is the compliance concern fix made on top of them.

ioxide 0.16 to 0.20 (#929, #917)

  • IdleTimeoutMs is ReadTimeoutMs on both transports now, and it means something else: the clock only runs while the server waits on its client. A slow handler keeps its connection; an idle keep-alive connection or a silent websocket does not. The engine's own option follows, as ReadTimeoutMs on TcpTransportOptions and QuicTransportOptions, and the old IdleTimeoutMs is gone, with no alias. The docs are rewritten for the new meaning - including HTTP/2, whose frame loop keeps a read parked while handlers work, so there a response slower than the timeout is still cut.
  • SendTimeoutMs is a deadline for one flush now rather than for the whole response; documented.
  • Both stay off by default, as decided in Update ioxide to 0.14.239 and pin the new TCP timeouts #916: this listener carries upgraded connections as well as keep-alive HTTP. The docs say how each relates to a websocket's ping interval (Decide the ioxide TCP timeout defaults for websocket workloads #917).
  • The raw shutdown(2) calls and their LibraryImport are gone. DecRef sends the FIN itself, after any send still in flight, and never on a descriptor the reactor has already closed.
  • TLS on the TCP transports is terminated in BoringSSL now, which ioxide bundles; docs, log messages and samples say so.
  • Ioxide engine: adapt to the next ioxide release (IdleTimeoutMs rename, drop the raw shutdown calls) #929's optional third point needs nothing: ioxide went back to a RecvQueueEntries default of 64 in 0.17.340, which is what the engine uses.

ioxide 0.21 to 0.26.563 (#971, #975)

  • Request bodies stream. HTTP/2 and HTTP/3 run through ioxide's one RunAsync, whose handler takes the body reader, so a request's body reaches the handler as it arrives instead of being assembled first.
  • Back to the reactor after the handler (Ioxide engine: return to the reactor after the handler, since ioxide 0.24 no longer does it #971). Since 0.24 a reactor has no SynchronizationContext, so a handler that awaits something other than ioxide comes back on the thread that completed it, where the response and the thread-statics it reads (IoxideReactor.Current, the request pools) don't belong. Both drivers take Reactor.Current when they start and await a ReactorSwitch right after the handler. On the reactor, the usual case, that's one thread check.
  • Static files straight into the write buffer (Ioxide engine: static files under kernel TLS (and cleartext) can use ReadFileAsync and skip the copy through an AssetReader #975). IIoxideFileSink in GenHTTP.Api.Ioxide, which the HTTP/1.1 sink offers where the writer is the connection's own write buffer: in cleartext, or with kernel TLS. IoxideAssetContent then reads the file off the ring straight into that buffer after the headers, 12 KiB per flush, with no AssetReader and no copy. TLS in BoringSSL, HTTP/2 and HTTP/3 keep the AssetReader path.
  • Certificate rotation once per port. The engine hands every reactor the same TlsOptions per port, and since 0.26 the reactors started with one TlsOptions share its contexts. So a session ticket resumes on any reactor of the port, and ReloadCertificates rotates each port through one service. One call per reactor only built the same set again, which 0.26 keeps until the stop.
  • Docs. SendTimeoutMs is also the listener's TCP_USER_TIMEOUT since 0.26, and TxKernelTls fails the start without the kernel's tls module rather than every connection after its handshake.
  • Nothing else the engine uses changed. ioxide 0.26.563 has no API break against 0.25.482, and 0.25's removals name nothing the engine uses.

ioxide 0.27.637 (#984)

0.27 closes the ioxide issues filed from this pull request's review, MDA2AV/ioxide#566 to MDA2AV/ioxide#573. The engine builds against it unchanged, and takes up three of its additions:

HTTP/2 and HTTP/3 streamed responses (#930, #932)

Reactor start (#933)

Reactor.OnFault is set. A reactor that cannot start - a ring the kernel refuses, a port already taken, a throw from OnStart - fails StartAsync with its reason, and the reactors that did start are stopped again instead of serving one short. Every reactor is constructed before any thread starts, so a configuration ioxide refuses fails before anything runs. A reactor that faults after it started is logged as critical; one that fails while the start is failing already is logged at debug, so a taken port does not produce a critical line per core.

Websocket writes (#914, #959)

  • Serialized by default. A connection serializes each write together with its flush - including the pong and close frames the default OnPing and OnClose handlers send from the message pump, which an application's own lock never covers. DisableWriteSynchronization() on the builders opts out, for a connection only one thread ever writes. An uncontended SemaphoreSlim.WaitAsync completes synchronously, so the default costs next to nothing.
  • Refused once the handler has ended. DisposeAsync marks the connection closed under the write lock, so a write in flight finishes first, and every later write throws ConnectionClosedException, which a background writer started from OnConnected should stop on. The docs on ISocketConnection say so.

On the ioxide engine both matter for more than garbled frames. With four writers and no lock the test host crashed with native memory corruption in 2 of 16 runs, and in none of 16 with the writes serialized. And ioxide pools its connections: on one reactor, a late write to a closed connection landed in the stream of the client that took its place, in 3 of 3 runs, and is refused with this change.

Websocket protocol checks

Autobahn's fuzzing client failed 111 of its 244 cases against the imperative echo handler HTTP Arena runs. Apart from the frame size, each failure was a rule of RFC 6455 the reader did not check. The reader now reports them as error frames, with new FrameErrorType values, and FrameError.CloseStatus gives the status code to close the connection with:

What the client sends Error Close
A frame with a reserved bit set and no extension negotiated (5.2) InvalidReservedBits 1002
A continuation frame with no message to continue, or a new message started inside a fragmented one (5.4) InvalidContinuation 1002
A close frame with a one-byte payload, or with a status code that may not be sent (5.5.1, 7.4) InvalidCloseFrame 1002
A text message or close reason that is not valid UTF-8 (8.1) InvalidUtf8 1007

TryReadFrame returned the frames of a fragmented message one at a time. A message interrupted by a ping also lost its final fragment: TryReadFrame returned that fragment on its own, and the next ReadFrameAsync waited for it. TryReadFrame now leaves both cases to ReadFrameAsync, which reassembles the message. It does not decode those frames itself, because decoding unmasks the payload in place.

MaxFrameSize(int) on the websocket builders sets the largest payload a frame may carry. The default stays 16 KiB, the limit the connection had hard-coded.

What users will notice: a client that breaks one of these rules now gets an error frame where its data used to be passed on. The new FrameErrorType values come after the existing ones, so their numbers do not change. In manual continuation mode, checking that a fragmented text message is valid UTF-8 is up to the handler (documented on HandleContinuationFramesManually); unfragmented text messages are still checked.

Hot path: Frame.Decode now runs Utf8.IsValid (vectorized) over every unfragmented text frame. Binary and control frames are unaffected. Benchmarks/ has no websocket benchmark, so the cost is not measured here.

Chunked request bodies

ChunkedFraming.CopyChunk read the length of the chunk it had just copied after advancing the reader past it. A ReadOnlySequence computes its length from its segments, and a reader may reset those once they are handed back, so a chunk that started in a later segment of the buffer could report a length larger than the buffer it was copied into. Over the ioxide engine's TLS connections, this cut chunked bodies short: an echoed 64 KiB chunked body came back 12 bytes short in 18 runs out of 20, and the connection faulted. The length is now taken first. The bug came with the body consolidation of #964, so 11.2.0 and 11.3.0 have it too, but there it stayed latent: with ioxide 0.15.251 the same requests over TLS never hit it (0 of 60). It showed with this branch's ioxide update, whose TLS reader fills its buffer differently. ChunkedFramingTests.TestReadWholeChunkStartingInLaterSegment reproduces it with a plain Pipe, so any engine can meet it when a chunk starts in a later segment of the buffer.

Samples

The ioxide samples with a TCP listener set both ReadTimeoutMs and SendTimeoutMs, a minute each, so they show both clocks: a client that sends nothing is let go, and so is one that stops reading. Both stay off by default in the engine. Http3Sample binds no TCP listener; it already set QUIC's read timeout, and QUIC has no flush timeout of its own.

Verification

  • The solution builds clean, and each commit builds on its own.
  • Acceptance suite on the rebased branch, net11.0: 2,949 passed, 0 failed, 4 skipped. net10.0: 2,946 passed, 0 failed, 4 skipped. The skips are the client certificate tests.
  • New tests in StreamedResponseEndTests, ReactorStartupTests, ReadTimeoutTests, SynchronizedWriteTests, Http3Tests (a request body streamed over HTTP/3), ReactorReturnTests (HTTP/1.1 and HTTP/2), CertificateTests (20 connections over 4 reactors after a rotation) and LateWriteTests (all three engines, plus the reused ioxide connection). Each was run with its fix disabled and fails without it, apart from the pre-header 500 case, which is a control.
  • The large static file test goes through the new file sink: with the sink broken, it fails.
  • Websocket protocol checks: the 147 websocket tests pass on net10.0 and net11.0 with all three engines, and the two new TryReadFrame tests fail without the fix. Autobahn (cases 1-7 and 10) against the HTTP Arena genhttp-ioxide entry built from this branch: 238 of 244, up from 133 with 11.3.0. The six left are the 64 KiB frames, above the 16 KiB default.
  • With the websocket checks, the chunked fix and its tests: acceptance suite net11.0 2,981 passed, 0 failed, 4 skipped; net10.0 2,978 passed, 0 failed, 4 skipped. The new RecyclingPipeReader overwrites and resets every segment it is advanced past, so a framing that reads released memory fails deterministically. Without the chunked fix, TlsBodyTests.TestChunkedBodiesOverHttp1 (ioxide, TLS), the chunked TestLargeBodyFromRecyclingReader and TestReadWholeChunkStartingInLaterSegment fail.
  • ioxide 0.27.637: the solution builds clean, and each commit builds on its own. Acceptance suite net11.0: 2,984 passed, 0 failed, 4 skipped. net10.0, ioxide engine only: 1,364 passed, 0 failed, 3 skipped. TestFaultReachingIoxideIsLoggedOverHttp2/3, the reset tests' stderr check, TestBodyOfUnknownLengthIsRead and TestHttp3StreamsTheRequestBody, which now sends its body without a content-length, each fail without their fix.

Follow-ups

  • GenHTTP.Website: write serialization is the default now, DisableWriteSynchronization, and the timeout guidance for websockets.
  • The engine binds the wildcard whatever address an endpoint asks for, so a loopback endpoint is reachable from other machines. ioxide 0.26's TcpOptions.Address and QuicOptions.Address make that fixable.
  • HTTP/3 client certificates: ioxide exposes the QUIC peer certificate now, so ICertificateValidator.Validate could run over HTTP/3 as well; until then the startup warning still applies.

Closes #914, closes #917, closes #929, closes #930, closes #932, closes #933, closes #959, closes #971, closes #975, closes #984

@MDA2AV
MDA2AV marked this pull request as draft October 24, 2026 22:02
@MDA2AV MDA2AV changed the title Update ioxide to 0.20.396, answer Expect: 100-continue, and let websockets serialize writes Update ioxide to 0.20.396 and let websockets serialize writes Oct 26, 2026
@MDA2AV MDA2AV changed the title Update ioxide to 0.20.396 and let websockets serialize writes Update ioxide to 0.26.563, serialize websocket writes by default, and refuse them once the handler has ended Nov 11, 2026
- Rename IdleTimeoutMs to ReadTimeoutMs on the TCP and QUIC options, and
  document what the read timeout and the send timeout bound now (#929, #917)
- Drop the raw shutdown(2) calls: DecRef sends the FIN itself, after any
  send in flight (#929)
- Take the FlushResult the HTTP/2 and HTTP/3 writers return, and end a
  streamed body with ConnectionClosedException once the peer is gone, so a
  server-sent event source notices the client left (#932)
- Rethrow a fault that happens after the headers are out, so ioxide resets
  the stream instead of ending a truncated body cleanly (#930)
- TLS on the TCP transports is terminated in BoringSSL now; update the docs
  and samples
Set Reactor.OnFault, so a reactor that cannot start - a ring the kernel
refuses, a port already taken, a throw from OnStart - fails StartAsync with
its reason instead of ending the process from the reactor's thread. The other
reactors are stopped again, rather than serving one reactor short. A reactor
that faults after it started is logged as critical (#933).
SynchronizeWrites() on the builder makes the connection serialize every
write and its flush, including the pong and close frames the default OnPing
and OnClose handlers send from the message pump - which an application's own
write lock never covers. Off by default; the docs on OnPing, OnClose and
ISocketConnection now say which writes need serializing (#914).
Both drivers ran the handler, wrote the head and the body and handled a
fault the same way, line for line, apart from the ioxide writer they held.
That part is now StreamedDriver.DispatchAsync, driven through a small
IStreamedWriter adapter per protocol; each driver keeps only the mapping of
its own request.
IdleTimeoutMs stays as an obsolete alias on both, so code that sets it
keeps compiling.
…nd serve static files straight into the write buffer

- ioxide 0.26.563, from 0.20.396: HTTP/2 and HTTP/3 run through the one RunAsync whose handler takes the body reader, so request bodies stream. Nothing else the engine uses changed.
- #971: since ioxide 0.24 a reactor has no SynchronizationContext, so a handler that awaits something other than ioxide comes back on the thread that completed it. Both drivers take the reactor when they start and await a ReactorSwitch right after the handler, so the response, and the thread-statics it reads, are on the reactor again.
- #975: IIoxideFileSink in GenHTTP.Api.Ioxide. HTTP/1.1 offers it where the writer is the connection's own write buffer, in cleartext or with kernel TLS: a static file is read off the ring straight into that buffer after the headers, 12 KiB per flush, with no AssetReader and no copy. TLS in BoringSSL, HTTP/2 and HTTP/3 keep the AssetReader path.
- Certificate rotation: with ioxide 0.26 the reactors started with one TlsOptions share its contexts, so ReloadCertificates rotates each port through one service. One call per reactor only built the same set again.
- Docs: SendTimeoutMs is also the listener's TCP_USER_TIMEOUT, and TxKernelTls fails the start without the kernel's tls module.
…nization opts a connection out

A connection written from several threads, as the reactive flavour exists for, raced the pong and close frames the default OnPing and OnClose handlers write from the message pump, which no lock of the application's covers. On ioxide that corrupts the connection's buffer. An uncontended SemaphoreSlim.WaitAsync completes synchronously, so serializing is the default now; a connection only one thread ever writes can opt out. (#959)
A task started from OnConnected, a broadcaster or a timer, still held the connection after the client had gone and the pump had returned, and its writes still went to the response sink of a connection the engine was done with. On ioxide, whose connections are pooled, they landed in the stream of the client that took its place. DisposeAsync now marks the connection closed under the write lock, so a write in flight finishes first, and every later write throws ConnectionClosedException, which a background writer should stop on. (#959)
Each sample with a TCP listener sets ReadTimeoutMs and SendTimeoutMs to a minute, both off by default in the engine: a client that sends nothing is let go, and so is one that stops reading. Http3Sample binds no TCP listener; its QUIC read timeout was set already, and QUIC has no flush timeout of its own.
…Frame from splitting messages

- A frame with a reserved bit set, a close frame with a one-byte payload or a status code that
  may not be sent, and a close reason or text message that is not valid UTF-8 are reported as
  errors, as are a continuation frame with no message to continue and a new message started
  inside a fragmented one (new FrameErrorType values).
- FrameError.CloseStatus gives the status code to close the connection with for an error.
- TryReadFrame leaves a fragmented message, and one interrupted by a control frame, to
  ReadFrameAsync, which reassembles it. It returned the fragments one by one before, and a
  message interrupted by a ping lost its final fragment.
- MaxFrameSize(int) on the websocket builders sets the largest frame payload a client may
  send (16 KiB, as before).
…ast it

ChunkedFraming.CopyChunk read the length of the chunk it had copied after
handing the reader's buffer back with AdvanceTo. A ReadOnlySequence computes
its length from its segments, and a reader may reset and reuse those once
advanced past: a chunk starting in a later segment of the buffer then
reported a wrong length, larger than the buffer it was copied into. Over the
ioxide engine's TLS connections this cut an echoed chunked body short and
faulted the connection.
…its buffers, and over TLS

- RecyclingPipeReader serves a body a few segments at a time and overwrites,
  resets and unlinks every segment it is advanced past, so a framing that
  still touches memory it handed back fails instead of getting away with it.
  ChunkedFraming and ContentLengthFraming read 256 KiB through it across
  segment and read sizes, read 4 MiB to the end, and skip 1 MiB unread.
- StreamBodyTests: a megabyte in chunks of uneven sizes, streamed straight
  back while it is still being read, on every engine.
- TlsBodyTests (ioxide): bodies streamed back over TLS, chunked over
  HTTP/1.1 and with a Content-Length over HTTP/2, 64 KiB to 1 MiB, five
  times each.

Without the fix of 28b0cd1, TestChunkedBodiesOverHttp1, the chunked
TestLargeBodyFromRecyclingReader and TestReadWholeChunkStartingInLaterSegment
fail.
MDA2AV added a commit to MDA2AV/HttpArena that referenced this pull request Nov 16, 2026
…t/GenHTTP#953 (#1783)

* genhttp-ioxide: build GenHTTP from the branch of Kaliumhexacyanoferrat/GenHTTP#953

GenHTTP is cloned at e3974794, the head of #953, and referenced as
projects rather than the 11.3.0 packages; the commit is pinned because
the bench box builds on cached layers. ioxide.pg and ioxide.timer move
to 0.26.563 with the branch's ioxide. The echo handler closes a rejected
frame with FrameError.CloseStatus, and frames of up to 64 KiB are
accepted (MaxFrameSize).

* genhttp-ioxide: pin GenHTTP to 28b0cd10, which fixes chunked request bodies over TLS
MDA2AV added a commit to MDA2AV/HttpArena that referenced this pull request Nov 16, 2026
…nHTTP#953

5967f5c6b, one commit past 28b0cd107. It adds acceptance tests only:
Engine/Ioxide and the modules this entry builds are unchanged, and still
on ioxide 0.26.563.
All five ioxide packages move from 0.26.563 to 0.27.637. The release closes
MDA2AV/ioxide#566 to #573, which were filed from this pull request's review:
a file read under WriteOverflow=Segmented no longer goes out ahead of the
head written before it, HTTP/2 checks a request's content-length against its
DATA and grows its connection window, a stream that ends early stops the
client's upload, a TLS flush after the reactor stopped returns, a failed
reactor start gives back its ring, and the HTTP/2 and HTTP/3 stacks gain a
request body flag, ResetStreamAsync and OnHandlerFault. The engine builds
against it unchanged; the commits after this one adopt the new APIs.
…rver's logging instead of ioxide's stderr

A response that failed after its head was ended by rethrowing, the only way
ioxide used to have to reset a stream. ioxide then printed the fault to
stderr too, so every such fault was reported twice, and the second report
could be neither routed nor silenced. The shared streamed dispatch now ends
the stream with the writers' ResetStreamAsync from ioxide 0.27, which sends
the INTERNAL_ERROR (HTTP/2) or H3_INTERNAL_ERROR (HTTP/3) the rethrow did.

What ioxide still catches itself, a fault in the dispatch's own error path or
the stack failing to finish a response once its handler has returned, goes
to OnHandlerFault, which logs it through the server's logging as "Failed to
complete {Protocol} request". HTTP/2 connections share one Http2Options,
built with the server. (#984)

StreamedResponseEndTests: the reset tests assert that the fault does not
reach stderr, and TestFaultReachingIoxideIsLoggedOverHttp2/3 have a log sink
refuse the dispatch's own report, so the fault escapes to ioxide and has to
arrive in the server's logging instead. Without the change, the reset tests
and the two new ones fail.
… the handler

A stream counted as having a body only when it carried a content-length,
which neither protocol requires. A body of unknown length, as HttpClient
streams one it cannot measure and fetch sends one from a ReadableStream,
arrives without it, so the handler was given no body at all. StreamedRequest
now takes the request body flag ioxide 0.27 reads off the HEADERS, and still
counts a content-length, so an empty body announced by one stays a body, as
it is over HTTP/1.1.

StreamedRequestBodyTests.TestBodyOfUnknownLengthIsRead echoes a body sent
without a content-length over HTTP/2, and Http3Tests.TestHttp3StreamsTheRequestBody
now sends its body that way. Both fail without the change.
@MDA2AV MDA2AV changed the title Update ioxide to 0.26.563, serialize websocket writes by default, and refuse them once the handler has ended Update ioxide to 0.27.637, serialize websocket writes by default, and refuse them once the handler has ended Nov 16, 2026
Playground/Program.cs takes main's version: #970 moved the general samples
to the website and deleted FunctionalSample, which 8d1bae3 had switched the
playground to. The ioxide samples stay, so their timeouts from a5d6314 are
kept.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment