Skip to content

test(runtime): bound the start-build race on its child process, not mocha's 5 s default - #541

Merged
Upd4ting merged 1 commit into
mainfrom
test/wec9b4589-13-start-build-race-stress
Nov 20, 2026
Merged

Upd4ting merged 1 commit into
mainfrom
test/wec9b4589-13-start-build-race-stress

Conversation

@Upd4ting

Copy link
Copy Markdown
Member

Why

Wec9b4589-13: the nightly ci / Stress job failed 3 of 3 runs on every schedule since 2026-11-17 (runs 44003000993, 44223258026, 44444276974, 44660719011). The 3 child-process tests of test/core/runtime/start-build-race time out at 5000 ms beside 4 busy loops per CPU. The test reached main with the promotion of 2026-11-16 (#524), and the first nightly after that failed.

What changed

  • Each step of the test already waits on an event: the prune child prints ready, a lease names the entry, and the child exits. The only wall-clock bound was mocha's 5000 ms default, which also has to cover a ts-node child booting on a loaded host. The suite now uses this.timeout(CHILD_TIMEOUT_MS + 5000), as module-cache.leases.test.ts does for one child per test. runInChild already kills a child after CHILD_TIMEOUT_MS, and that rejects printed and exited with the child's output. A hung child therefore still fails with a reason instead of mocha's bare timeout.
  • With the timeout raised, a load run showed a second failure: SyntaxError: Unexpected end of JSON input in readLeases. RunLeaseFile stages each write as <lease>.json.tmp, then renames it. The core reads only .json names, but the test helper parsed every file in .leases. The leased() poll could therefore read a half-written staging file. readLeases now skips names that do not end in .json, as the core does.
sequenceDiagram
  participant T as test
  participant C as prune child
  T->>C: spawn (killed after CHILD_TIMEOUT_MS)
  C-->>T: ready
  T->>T: prepare + verify, poll readLeases (.json only) until the lease names FIRST
  T->>T: rewrite build.json to REBUILT, open the gate
  C-->>T: exit 0
  Note over T: bounded by CHILD_TIMEOUT_MS + 5000, not 5000
Loading

Proof

Local reproduction, test/core/runtime/start-build-race.test.ts alone, beside 4 × nproc busy loops (16 CPUs).

Before:

  3 passing (16s)
  3 failing
  1) ... starts the new build when a prune removes the folders of the build.json it read before it claimed them:
     Error: Timeout of 5000ms exceeded.
  2) ... starts the new build with --refresh-config, and resolves antelope.config.ts once:
     Error: Timeout of 5000ms exceeded.
  3) ... starts the new build with --refresh-config when antelope.config.ts was edited for it:
     Error: Timeout of 5000ms exceeded.

With the timeout raised but the helper unchanged, under the same load:

  1) ... starts the new build when a prune removes the folders ...:
     SyntaxError: Unexpected end of JSON input
      at readLeases (test/helpers/cache-leases.ts:106:32)
      at leased (test/core/runtime/start-build-race.test.ts:90:31)

After, at the host's own load average of about 125 on 16 CPUs, which is above the Stress job's 4 loops per CPU:

    ✔ starts the new build when a prune removes the folders of the build.json it read before it claimed them (1984ms)
    ✔ starts the new build with --refresh-config, and resolves antelope.config.ts once (2003ms)
    ✔ starts the new build with --refresh-config when antelope.config.ts was edited for it (1996ms)
  6 passing (6s)

The host is shared with other jobs, so 64 more busy loops took it to about 11 runnable processes per CPU. At that load, each of the 2 runs lost one child to its own 30 s CHILD_TIMEOUT_MS kill. That load is about three times the Stress job's, so this PR keeps CHILD_TIMEOUT_MS as it is. The 20 suites that use the lease helpers pass (145 tests).

A manual run cannot reach the nightly tier, so the next scheduled nightly is the Stress proof.

Follow-ups

  • none

@Upd4ting
Upd4ting marked this pull request as ready for review November 20, 2026 22:19
@Upd4ting
Upd4ting enabled auto-merge November 20, 2026 22:19
@Upd4ting
Upd4ting added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 03996f1 Nov 20, 2026
20 of 25 checks passed
@Upd4ting
Upd4ting deleted the test/wec9b4589-13-start-build-race-stress branch November 20, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant