Skip to content

Destroy the TNode drainer and connector before the buffers they use - #222

Merged
resetius merged 1 commit into
masterfrom
fix-tnode-drainer-destruction-order
Nov 19, 2026
Merged

resetius merged 1 commit into
masterfrom
fix-tnode-drainer-destruction-order

Conversation

@resetius

Copy link
Copy Markdown
Owner

Fixes #220

TNode declared SendBuffer after Drainer, so ~TNode freed the buffer before the suspended write was cancelled. On io_uring the kernel can read the freed memory until the cancel in ~TAwaitable returns.

The fix is an explicit ~TNode that destroys Drainer and Connector first, so correctness does not depend on member declaration order. The destructor has a comment saying why. Member order is unchanged to keep the diff minimal.

Test: test_node_destroyed_during_blocked_write destroys a node whose 64 MB write is blocked on a peer that never reads. It is meant to run under ASan; note that ASan cannot see kernel reads, so on io_uring it only exercises the teardown path rather than detecting the original race.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Linux)

334 tests  +1   334 ✅ +1   25s ⏱️ +2s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 36b381d. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (MacOS)

255 tests  +1   255 ✅ +1   34s ⏱️ +2s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 36b381d. ± Comparison against base commit 85489f0.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Windows)

176 tests  +1   176 ✅ +1   44s ⏱️ ±0s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 36b381d. ± Comparison against base commit 85489f0.

@resetius

Copy link
Copy Markdown
Owner Author

Independent review of head 36b381d591405c1f3bfcb357d7761b1cf9b97b30. Verdict: the fix is correct and complete. No P1/P2 findings, two P3 notes.

What I checked

  • TNode frees SendBuffer before cancelling the write that reads it #220 and the diff. On master, members are destroyed in reverse declaration order: SendBuffer first, then Socket, then Connector, then Drainer. So both the buffer and the socket were gone before the pending write was cancelled. ~TNode() (coroio/actors/node.hpp:88-91) now resets Drainer and then Connector before any member is destroyed. A suspended WriteSome is cancelled in ~TAwaitable while SendBuffer and Socket are still alive. TUring::Cancel(TOperation&) returns only after the kernel has finished with the SQE.
  • Coroutine teardown. TFuture's move-assign destroys the old frame. Destroying DoDrain's frame also destroys the TByteWriter::Write child frame and its WriteSome awaitable. The connector frame cancels Connect/Sleep the same way.
  • Other pollers. IOCP had the same hazard: an overlapped send uses the buffer until CancelIoEx completes, and TIOCp::Cancel(TOperation&) waits for the packet. The same change fixes it. On kqueue, epoll, poll and select nothing reads the buffer while the coroutine is suspended, so the old order was harmless there. The new order also removes the event before the socket is closed.
  • Build and full ctest with ASan (Debug, as in CI). macOS (kqueue): all passed. rpi5 aarch64, Linux 7.3 (epoll, plus the uring tests): all passed. test_actors 33/33 on both.
  • I also ran the new test with the poller switched locally: TPoll/TSelect on macOS and TUring/TPoll on Linux. It passes on all of them.
  • Does the test fail without the fix? No. With master's node.hpp it passes on every backend, including TUring under ASan.
  • CI: every job on this head is green (Linux, macOS, Windows).

Findings

  • P3: tests/test_actors.cpp:1293-1295 and the PR body. The test runs only on TDefaultPoller: epoll on Linux, kqueue on macOS, IOCP on Windows. It never runs on io_uring, so "run under ASan on every poller" and "on io_uring it only exercises the teardown path" do not match what the test does. It also passes on master, so it does not guard the destruction order. A deterministic check is possible without the kernel. Use a poller wrapper whose socket has a WriteSome that never completes and reads its buffer in the awaitable's destructor, the way io_uring's kernel may. I tried this locally on macOS with ASan. With master's node.hpp it reports a heap-use-after-free in that destructor (~TNode -> ~TFuture -> TByteWriter::Write frame). With this PR it runs clean:
    struct TReadOnCancelPoller : TDefaultPoller {
        struct TSocket : TDefaultPoller::TSocket {
            using TDefaultPoller::TSocket::TSocket;
            TSocket(TDefaultPoller::TSocket&& s) : TDefaultPoller::TSocket(std::move(s)) { }
            auto WriteSome(const void* buf, size_t size) {
                struct TAwaitable {
                    ~TAwaitable() {
                        if (Handle) {
                            volatile char c = static_cast<const volatile char*>(Buf)[Size - 1];
                            (void)c;
                        }
                    }
                    bool await_ready() { return false; }
                    void await_suspend(std::coroutine_handle<> h) { Handle = h; }
                    ssize_t await_resume() { return 0; }
                    const void* Buf; size_t Size; std::coroutine_handle<> Handle = {};
                };
                return TAwaitable{buf, size};
            }
        };
    };
    // TNode<TReadOnCancelPoller> with the resolver on static_cast<TDefaultPoller&>(poller) and a
    // socket factory wrapping TDefaultPoller::TSocket; Acquire/Commit 4096 bytes, Drain() + Step()
    // for ~300 ms against a listening socket, then let the node go out of scope.
  • Out of scope, pre-existing and not changed by this PR: if ~TNode runs while the connector waits in TResolver::Resolve (a DNS host name, not an IP literal), the destroyed frame's handle stays in TResolver::WaitingAddrs (coroio/dns/resolver.cpp:279). ResumeWaiters resumes it later (resolver.cpp:227). This is worth a separate issue.

@resetius resetius added the agent-reviewed An independent review agent checked this PR; the verdict is in a PR comment label Nov 19, 2026
@resetius
resetius merged commit c80694f into master Nov 19, 2026
7 checks passed
@resetius
resetius deleted the fix-tnode-drainer-destruction-order branch November 19, 2026 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-reviewed An independent review agent checked this PR; the verdict is in a PR comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TNode frees SendBuffer before cancelling the write that reads it

1 participant