test: fix flaky cleanup in http2 test - #65701
Conversation
Signed-off-by: Tim Perry <pimterry@gmail.com>
This comment has been minimized.
This comment has been minimized.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #65701 +/- ##
==========================================
+ Coverage 90.05% 90.06% +0.01%
==========================================
Files 754 754
Lines 255759 256395 +636
Branches 48329 48505 +176
==========================================
+ Hits 230328 230929 +601
- Misses 16542 16578 +36
+ Partials 8889 8888 -1 🚀 New features to boost your workflow:
|
|
Thanks @panva! I have just cancelled and re-run the PR test though, it's now here: https://ci.nodejs.org/job/node-stress-single-test/881/. The main test linked there is the 2nd I've tried, after the first passed, and it only fails because the test run count is bumped up to 10,000. New PR stress test now does 10,000 runs for the PR as well to make it a fair fight. |
The |
|
https://ci.nodejs.org/job/node-stress-single-test/nodes=macos15-x64/867/testReport/ shows 2 failures out of 7000 tests actually completed for that test on main, so unfortunately I think we do need more than 1000 to validate it. For the positive case we'll have to do the lot (happy to kill it if it starts failing early of course, but hopefully not!) Test time shouldn't be too long, the 7000 runs took 35 minutes so the 10000 tests themselves should run under an hour (after the build). |
|
FWIW the |
|
On my machine Main: With this patch: |
|
Thanks @lpinca! That's super helpful. On the timing, I think I'm just confused by the conflicting timing reports - https://ci.nodejs.org/job/node-stress-single-test/nodes=macos15-x64/867/testReport/ says 35 minutes for the 7000 test run, but looking at that build's total time and the current test speed that doesn't seem right at all... I'll keep an eye on it and the pending queue and cut it if it takes too long. So far looking good though. |
|
I've already stopped it. Both @lpinca, myself, and the 3 hours worth of stress on top of it is evidence enough that this didn't make things worse, only better. |
|
When CI unwinds its current massive backlog we'll see how the rest of https://ci.nodejs.org/job/node-test-pull-request/76843/ fares and possibly |
This new HTTP/2 test just flaked today & yesterday: a ECONNRESET on the server side.
Looking at this closely, I think this is very likely a race in the test teardown. The test simply destroyed the client and closed the server: it never ended the request that was in progress, and didn't cleanly close the client session. I think this triggers a race, where the server sends a GOAWAY to shut down the client, gets a ECONNRESET back if the client is always fully gone, and then throws that on the incomplete request.
This change now ends the request properly, and closes the client session and server cleanly, so everything should do the proper GOAWAY shutdown dance before anything is closed, guaranteeing no unexpected resets.
It's hard to validate (only failing on macos15-x64, and with low frequency, stress-test on main didn't reproduce) but I think doing proper cleanup here is good in any case, and doing clean shutdown throughout the test should make an unexpected reset structurally impossible.