Skip to content

fix(ci): run the formal checks on Windows and size test bounds for a loaded host - #126

Merged
Upd4ting merged 2 commits into
mainfrom
fix/wec9b4589-19-nightly-windows-stress
Nov 20, 2026
Merged

Upd4ting merged 2 commits into
mainfrom
fix/wec9b4589-19-nightly-windows-stress

Conversation

@Upd4ting

@Upd4ting Upd4ting commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Summary

Wec9b4589-19. The nightly CI has failed every night since 2026-11-17, in two ways. This PR fixes both.

Windows legs (Node 22 on Windows, Node 24 on Windows). scripts/test.sh runs formal-pipeline-chains/check.mjs. That script spawned pnpm with no shell, and Node on Windows only finds .exe and .com files, so it failed with spawn pnpm ENOENT. formal-json-value-identity and formal-merge-repeated-keys did the same with pnpm and with the extensionless node_modules/.bin/ajs. All three now run the typescript/bin/tsc and @antelopejs/core CLI scripts with process.execPath, as the other three formal checks already did. The build step is now tsc itself, without rimraf dist, because each check runs named test files only. No shell: true is used, so Node 24 prints no DEP0190 warning.

Stress (3 of 3 runs failing). Every failure came from a wall-clock bound sized for an idle host, or from one wait that was too short:

Test Bound before Worst seen under load Change
Probe tests that share TIMEOUT_MS (probe-steps, probe-locks, probe.spec, probe-url.spec, probe-options) 1 or 2 × 1 s 5.7 s locally, 3.6 s in CI TIMEOUT_MS 1 s → 5 s
probe-many 3 × 400 ms 6.2 s in CI uses the shared TIMEOUT_MS (deadline 15 s)
probe-steps latency test 2 × 100 ms 208 ms DELAY_MS 100 → 500 ms
Large external JSON dedup 100 ms/row (group: 200 ms/row) 250 ms/row 400 ms/row for both, and GROUP_BUDGETS removed
pipeline-chains feed property waited 300 ms of quiet for the probe row's removal the removal arrived later, as an extra leading removed event now waits up to DELIVERY_MS for that removal

Each test still catches the bug it names. A quadratic regex over the 100k-character URL value takes 17 s here, above the 5 s bound. The quadratic respell the dedup budget guards took over 1.1 s per row on an idle host, above 0.4 s per row. probe-many still counts the prepared statements, which is what catches a probe that confirms every relation.

flowchart LR
  T["scripts/test.sh chains"] --> C["check.mjs"]
  C -->|"before: spawn('pnpm', …)"| E["ENOENT on Windows"]
  C -->|"after: spawn(process.execPath, [tsc or ajs CLI, …])"| N["the same Node, on every OS"]
Loading

Related issue

Wec9b4589-19, from ANT-3515. Nightly runs failing since 2026-11-17: 44002817791, 44223078548, 44444088606, 44660523649.

Verification

A manual run cannot reach the nightly tier, because the ci-tier action returns nightly only for schedule. The next scheduled nightly is the final proof. Until then, these are the local reproductions.

Windows ENOENT, reproduced with pnpm removed from PATH.

Before (check.mjs from origin/main):

$ PATH=<node only> node .agents/skills/formal-pipeline-chains/check.mjs --no-build
tsc failed on .agents/skills/formal-pipeline-chains
Error: spawn pnpm ENOENT
exit 2

After (the build step runs too):

$ PATH=<node only> node .agents/skills/formal-pipeline-chains/check.mjs
exit 0
$ PATH=<node only> node .agents/skills/formal-json-value-identity/check.mjs --runs 20
exit 0
$ PATH=<node only> node .agents/skills/formal-merge-repeated-keys/check.mjs --runs 20
exit 0

Stress, reproduced like the CI job. The script starts 4 busy loops per CPU (64 here), then runs the probe test files against two MySQL 8.4 servers in parallel.

Before:

== server a
  133 passing (3m)
  9 failing
  probe, on a relation another session holds a lock on   AssertionError: expected 5320.39 to be below 2000
  probe, with each option a URL's query may set           AssertionError: expected { outcome: 'unreachable', …(1) } to deeply equal { outcome: 'signed-in', …(4) }
  probe, at each step of a live server                    AssertionError: expected 5554.13 to be below 2000
  probe, at each step of a live server                    AssertionError: expected 208 to be below 200
  probe, with a URL value made to be slow                 AssertionError: expected 2315.52 to be below 1000
  …
== server b
  134 passing (3m)
  8 failing
  probe, against a database of many relations             AssertionError: expected 4945.66 to be below 1200
  …

After:

== server a
  142 passing (3m)
== server b
  142 passing (4m)

Also run beside the same load after the change:

  • distinct-large.test.js: passes twice on MySQL 8.4 and three times on MariaDB 11.8.
  • The pipeline-chains check: exit 0 on MySQL 8.4 and on MariaDB 11.8.

Run without load:

  • The probe files pass, 142 tests, in 68 s, up from 38 s, because stalled-server tests now wait out a 5 s timeout.
  • pnpm run test:unit: 647 passing.
  • pnpm run typecheck, plus oxlint and oxfmt on the changed files: clean.

Review

An independent review of the diff (Sonnet 5.5) found no defects in the spawn changes. It raised three points about how much the larger bounds weaken the tests:

  • probe-locks. With a 5 s timeout, the server's own lock wait (6 s) ends before the 10 s bound, so the time bound alone would not catch a probe with no read timer. The outcome assertion does catch it. With the read timer removed, the test fails with AssertionError: expected 'signed-in' to equal 'unreachable' (measured). No change.
  • probe-url.spec. A slow-value bug that costs 1 to 5 s against a closed port would now pass. The bug that test names, a quadratic over the 100k-character value, takes 17 s. Under load the closed-port probes took up to 2.6 s, so a tighter bound would flake. Kept as is.
  • Chains feed Opened. Late events from two or more open attempts could still leak. Every failure in the four nightly runs was the single late removal fixed here. A count-based handshake must also handle a removal whose insert came before the feed opened, so it waits for evidence of that case.

Follow-ups

  • none

Checklist

  • The pull request title follows Conventional Commits.
  • I added or updated tests when behavior changed.
  • I updated documentation when users or contributors are affected.
  • I documented breaking changes and migration steps.
  • I did not include credentials or unrelated changes.

@Upd4ting
Upd4ting marked this pull request as ready for review November 20, 2026 22:28
@Upd4ting
Upd4ting enabled auto-merge November 20, 2026 22:28
@Upd4ting
Upd4ting added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 699f2a2 Nov 20, 2026
8 of 11 checks passed
@Upd4ting
Upd4ting deleted the fix/wec9b4589-19-nightly-windows-stress branch November 20, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant