Skip to content

proactor, tcp, tls: a stop ends what waits on the proactor, so a parked handler unwinds, and a call from another thread after it returns instead of waiting forever (#29) - #53

Merged
MDA2AV merged 6 commits into
mainfrom
fix/stop-parked
Nov 11, 2026

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Closes #29.

A stop now ends what was waiting on the proactor, instead of leaving it parked forever. This follows MDA2AV/ioxide#555, ioxide's fix for the same bug, except where kron's code needs something different.

What was wrong

Teardown closed the sockets of live connections and left the connections themselves alone. Nothing resumed a handler parked in read(), flush() or a RingTimer wait, so its finally never ran. After the stop, a call from another thread posted to a queue nothing would drain again, and waited forever.

What changed

  • Reads. Before any socket closes, teardown marks every live connection closed. A parked read returns closed, and its handler unwinds right there on the proactor thread, while the socket number is still its own. A flush started after that, on any connection, sends nothing.
  • What is queued when the stop comes. The last pass's sends, and a send a woken handler queued on a connection still open, go to the kernel next, while every socket they name is open. Only then do the sockets close. Left in the SQ, a later submit (a growing op table) would send them to whatever had taken the number, and no cancel finds what the kernel never saw.
    • This PR's first version submitted first but marked each connection only as its socket closed. A woken handler's send to a neighbour still open stayed queued, and the new test for it fails there: 28 bytes reach the socket that took the number.
  • Flushes. In kron a flush belongs to the kernel the moment it parks: submitFlush claims it in the same call, so marking the connection closed releases none of them. Each one ends once IORING_REGISTER_SYNC_CANCEL, keyed by the send's tag, says the kernel is done with the send.
    • That call waits up to 1 s for a send running in an io-wq worker. If the kernel can't confirm in time, or the SQ could not be emptied, the flush stays parked, since the kernel may still be reading its slab.
    • Teardown calls completeFlush(), not finishSend(). With closeAfterSend set, finishSend() would recycle the connection and close its socket number a second time.
  • RingTimer waits still parked complete with -ECANCELED (RingTimer.ECANCELED). The kernel copies a timeout's deadline when it takes the SQE, so these need no cancel, unlike ioxide's ring ops that carry buffers. A handler that starts another wait from there stays parked.
  • Calls from another thread after the stop. Kron has no ReactorSwitch: every call from another thread goes through post.
    • Posts close at the end of the teardown, with a volatile flag and no gate. A post from another thread that finds the flag set drains the queue itself, one task taken at a time under a lock only ever taken once posts are closed, since MpscQueue has a single consumer.
    • A call its caller waits on is posted as a Handoff. The drain runs those, and drops every other post, as ioxide drops its own.
    • read() returns closed, and flush() drops what was staged, both as on any closed connection.
    • The TCP and TLS pipes' read() and flush() return completed. The TLS pipe's do so without reaching the session, whose BoringSSL is the proactor's.
    • RingTimer.delay returns ECANCELED, and a raw submitTimeout's completion gets -ECANCELED.
    • Proactor.post drops the task, and so the fire-and-forget calls (decRef and the like) are dropped too.
  • Memory. Once every wake is done, each live connection whose handler has let go goes back to the pool as a recycle would: its own buffer ring unregistered while the ring is open, cleared, and freed right there if the pool is full. The teardown's last step frees the pool.
    • A connection whose handler is still busy elsewhere at the stop, or whose send the kernel may still be reading, keeps its memory. A write into it would land in freed memory, and clear() would reopen it to a read from another thread.
    • It needs no new state: the first version's freeing took a second gate, a queue of late releases and a flag reset on every recycle.
  • Smaller changes the fix needed:
    • closeListeners marks the listeners closed. Otherwise a handler woken by the teardown that calls stopAccepting would close their numbers again.
    • The proactor's thread stops counting as the proactor's once run() returns.

On the hot path, a post from another thread reads one volatile flag after its enqueue. The check that tells a handoff from a post runs only once the proactor has stopped, and posts from the proactor thread itself are unchanged. Everything else is in the teardown.

Tests

  • StopParkedTest is in commonTest, so it runs on both runtimes. It has eleven tests:
    • A handler parked in a read, in a RingTimer wait, or in a flush its client holds up runs its finally, and its connection is freed.
    • In both receive modes, the connections of handlers parked in a read, a held-up flush or a RingTimer wait, and of one that let go, have their slab freed, and their own buffer ring in incremental mode. A connection whose handler is busy outside kronos at the stop keeps its slab.
    • A send the last pass left queued, and one a handler woken by the stop queues on another connection, reach their own clients. A socket put on the freed descriptor number (dup2) receives nothing when the ring is entered.
    • A read, a flush, a pipe's read and flush, a RingTimer wait, a raw submitTimeout and a post, each made from another thread after the stop, return or drop as their docs say.
  • PoolFullTest recycles two connections into a pool of one: the second has its slab freed right there, while the proactor runs.
  • TlsStopParkedTest (native) makes the after-the-stop calls through TlsDuplexPipe.
  • IoUringLayoutTest checks the new opcode and the sync-cancel struct against the kernel header.

Taking out each part of this version fails its own tests (native):

Without Fails
submitting what is queued before the sockets close both queued-send tests
marking every connection before closing any the send a woken handler queues on another connection
running handoffs after the stop the read, flush, pipe and RingTimer calls from another thread
the TLS pipe's checks after the stop TlsStopParkedTest: BoringSSL refuses the other thread
returning let-go connections to the pool at stop the parked tests' and the let-go test's freed slabs
freeing on a full pool PoolFullTest: neither connection freed
the let-go check before returning the held connection's slab freed under it, and reads from another thread after the stop hang

Marking connections closed, ending flushes and completing parked waits are unchanged from the first version, where taking out each one failed its own test.

With the change:

  • Native: all 331 tests passed: the 318 from before plus the 13 new ones.
  • JVM: all 105 passed: the 93 from before plus the 12 new ones.
  • kTLS: the kTLS tests print "skipped" locally, since this machine has no tls module.

…ed handler unwinds, and a call from another thread after it returns instead of waiting forever (#29)

The teardown marks each live connection closed before its socket goes, which wakes a parked read. A flush, which in
kronos is the kernel's the moment it parks, ends once IORING_REGISTER_SYNC_CANCEL by its tag says the kernel is done
with the send, a second at most; one it cannot vouch for stays parked, its slab kept. A RingTimer wait still parked
completes with -ECANCELED. What the last pass prepared goes to the kernel first, while every socket it names is open.

A post from another thread is refused once the stop begins, and the ones taken run in the teardown. Each handoff of
kronos's own answers a refusal as a closed connection would: a read returns closed, a flush drops what was staged, a
pipe's read and flush return completed, a RingTimer wait returns ECANCELED, and submitTimeout throws. Proactor.post
drops the task.

The stop lets go of each live connection, freeing it once its handler has let go too. A handler that lets go later,
from another thread, frees it then: at the teardown's end, or where it is once the teardown is over.
…ingTimer wait, and to calls from another thread after it (#29)

Each handler runs its finally and its connection is freed; a read, a flush, a pipe's read and flush, a RingTimer
wait, a raw submitTimeout and a post made from another thread after the stop each return, throw or drop as their docs
say; and a handler that lets go after the stop frees its connection. TlsStopParkedTest makes the calls through
TlsDuplexPipe. All 8 fail on main.
…r by a handler it wakes, reaches no other socket, and a stop no longer frees live connections (#29)
…ore a socket closes and sends what is queued first, and posts close last, with no gate on a post from another thread (#29)

A send a handler woken by the stop queued on a connection still open stayed in the SQ when its socket closed, and a later submit (a growing op table) sent it to whatever had taken the number. Every connection is now marked closed first, then the SQ is submitted, then the sockets close.

Posts close at the end of the teardown with a volatile flag. A post from another thread that finds it set drains the queue itself, one task taken at a time under a lock only ever taken then: a Handoff (a read, a flush or a wait its caller waits on) runs and answers as on a closed connection, and any other post is dropped. The gate's compare-and-set and decrement on every post from another thread are gone, and so is the freeing of live connections at stop, which keep their memory.
…one a handler still holds, and a recycle into a full pool frees the connection right there (#29)
…to the pool, which its last step frees (#29)

Once every wake is done, each live connection whose handler has let go goes back to the pool as a recycle would: its own buffer ring unregistered while the ring is open, cleared, and freed right there if the pool is full. The teardown's freeMemory then frees the pool. One a handler still holds, or whose send the kernel may still be reading, keeps its memory: a write into it would land in freed memory, and clear() would reopen it to a read from another thread.
@MDA2AV
MDA2AV merged commit 1130b10 into main Nov 11, 2026
3 checks passed
MDA2AV added a commit that referenced this pull request Nov 11, 2026
…ptions (#54) count their services, so the last to stop frees the context

With #54 every proactor started with one TlsOptions instance shares its context, so the first proactor to stop could no longer free it under the rest. TlsOptions also keeps the ALPN list the context's callback reads and a count of the services on it, under the lock #54 builds it under: start raises the count, free lowers it, and the one that reaches zero frees the context and the list and clears them, so a start after that builds another. Each service still frees its own kTLS handoff buffer, and teardown frees the services after #53's drain, last of all.

Tests: TlsResumptionTest runs one-proactor servers on one port through a stop, a later start, and a start after the last stopped; the heap test runs two proactors sharing one options instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

a stop abandons handlers parked on the proactor, and calls from other threads never return

1 participant