Skip to content

test: fix flaky cleanup in http2 test - #65701

Open
pimterry wants to merge 1 commit into
nodejs:mainfrom
pimterry:fix-flaky-h2-deadlock-test
Open

test: fix flaky cleanup in http2 test#65701
pimterry wants to merge 1 commit into
nodejs:mainfrom
pimterry:fix-flaky-h2-deadlock-test

Conversation

@pimterry

@pimterry pimterry commented Sep 1, 2026

Copy link
Copy Markdown
Member

This new HTTP/2 test just flaked today & yesterday: a ECONNRESET on the server side.

Looking at this closely, I think this is very likely a race in the test teardown. The test simply destroyed the client and closed the server: it never ended the request that was in progress, and didn't cleanly close the client session. I think this triggers a race, where the server sends a GOAWAY to shut down the client, gets a ECONNRESET back if the client is always fully gone, and then throws that on the incomplete request.

This change now ends the request properly, and closes the client session and server cleanly, so everything should do the proper GOAWAY shutdown dance before anything is closed, guaranteeing no unexpected resets.

It's hard to validate (only failing on macos15-x64, and with low frequency, stress-test on main didn't reproduce) but I think doing proper cleanup here is good in any case, and doing clean shutdown throughout the test should make an unexpected reset structurally impossible.

Signed-off-by: Tim Perry <pimterry@gmail.com>
@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to Node.js core tests and test infrastructure. labels Sep 1, 2026
@panva panva added the flaky-test Issues and PRs involving tests that fail intermittently in CI. label Sep 1, 2026
@panva panva added author ready PRs with CI started, the required approvals, and no outstanding review comments. request-ci Add this label to start a Jenkins CI on a PR. labels Sep 1, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Sep 1, 2026
@nodejs-github-bot

This comment has been minimized.

@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.06%. Comparing base (971cc1c) to head (f7fb8fd).
⚠️ Report is 25 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #65701      +/-   ##
==========================================
+ Coverage   90.05%   90.06%   +0.01%     
==========================================
  Files         754      754              
  Lines      255759   256395     +636     
  Branches    48329    48505     +176     
==========================================
+ Hits       230328   230929     +601     
- Misses      16542    16578      +36     
+ Partials     8889     8888       -1     

see 49 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@panva

panva commented Sep 1, 2026

Copy link
Copy Markdown
Member

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@pimterry

pimterry commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Thanks @panva! I have just cancelled and re-run the PR test though, it's now here: https://ci.nodejs.org/job/node-stress-single-test/881/. The main test linked there is the 2nd I've tried, after the first passed, and it only fails because the test run count is bumped up to 10,000.

New PR stress test now does 10,000 runs for the PR as well to make it a fair fight.

@panva

panva commented Sep 1, 2026

Copy link
Copy Markdown
Member

New PR stress test now does 10,000 runs for the PR as well to make it a fair fight.

The main one was clearly failing starting from about 500 onwards, letting it run for 5 more hours was a waste of a host we could use better elsewhere. If run count 10000 is needed to let the flake come through fine, but we don't need to actually let it finish all 10000

@pimterry

pimterry commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

https://ci.nodejs.org/job/node-stress-single-test/nodes=macos15-x64/867/testReport/ shows 2 failures out of 7000 tests actually completed for that test on main, so unfortunately I think we do need more than 1000 to validate it. For the positive case we'll have to do the lot (happy to kill it if it starts failing early of course, but hopefully not!)

Test time shouldn't be too long, the 7000 runs took 35 minutes so the 10000 tests themselves should run under an hour (after the build).

@panva

panva commented Sep 1, 2026

Copy link
Copy Markdown
Member

FWIW the main one was at 7000ish after 4h30m, the PR is now 1650ish after 2h15m at NOT OK: 0.

@lpinca

lpinca commented Sep 1, 2026

Copy link
Copy Markdown
Member

On my machine

$ uname -a
Darwin imac.local 24.6.0 Darwin Kernel Version 24.6.0: Tue Jul 21 20:50:07 PDT 2026; root:xnu-11417.140.69.711.44~1/RELEASE_X86_64 x86_64
$ python3 tools/test.py --repeat=10000 test/parallel/test-http2-bidirectional-write-deadlock.js

Main:

=== release test-http2-bidirectional-write-deadlock ===                    
Path: parallel/test-http2-bidirectional-write-deadlock
node:events:505
    throw er; // Unhandled 'error' event
    ^

Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:216:20)
Emitted 'error' event on ServerHttp2Stream instance at:
    at emitErrorNT (node:internal/streams/destroy:170:8)
    at emitErrorCloseNT (node:internal/streams/destroy:129:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:90:21) {
  errno: -54,
  code: 'ECONNRESET',
  syscall: 'read'
}

...

[03:07|% 100|+ 9995|-   5]: Done                                           

Failed tests:
out/Release/node /Users/luigi/code/node/test/parallel/test-http2-bidirectional-write-deadlock.js
out/Release/node /Users/luigi/code/node/test/parallel/test-http2-bidirectional-write-deadlock.js
out/Release/node /Users/luigi/code/node/test/parallel/test-http2-bidirectional-write-deadlock.js
out/Release/node /Users/luigi/code/node/test/parallel/test-http2-bidirectional-write-deadlock.js
out/Release/node /Users/luigi/code/node/test/parallel/test-http2-bidirectional-write-deadlock.js

With this patch:

[03:11|% 100|+ 10000|-   0]: Done                                          

All tests passed.

@pimterry

pimterry commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Thanks @lpinca! That's super helpful.

On the timing, I think I'm just confused by the conflicting timing reports - https://ci.nodejs.org/job/node-stress-single-test/nodes=macos15-x64/867/testReport/ says 35 minutes for the 7000 test run, but looking at that build's total time and the current test speed that doesn't seem right at all... I'll keep an eye on it and the pending queue and cut it if it takes too long. So far looking good though.

@panva

panva commented Sep 1, 2026

Copy link
Copy Markdown
Member

I've already stopped it. Both @lpinca, myself, and the 3 hours worth of stress on top of it is evidence enough that this didn't make things worse, only better.

@panva

panva commented Sep 1, 2026

Copy link
Copy Markdown
Member

When CI unwinds its current massive backlog we'll see how the rest of https://ci.nodejs.org/job/node-test-pull-request/76843/ fares and possibly [commit-queue].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. flaky-test Issues and PRs involving tests that fail intermittently in CI. needs-ci PRs that need a full CI run. test Issues and PRs related to Node.js core tests and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants