Skip to content

fix(ci-node): build what the tests read in Stress again with typecheck - #104

Merged
Upd4ting merged 2 commits into
mainfrom
ci/ant-3515-nightly-failures
Nov 20, 2026
Merged

Upd4ting merged 2 commits into
mainfrom
ci/ant-3515-nightly-failures

Conversation

@Upd4ting

@Upd4ting Upd4ting commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Summary

ANT-3515. Goal: every nightly ci.yml run is green, Windows legs included. This pull request sorts the failures of the 2026-11-20 nightlies by cause and fixes the one shared cause still live in this repository.

What was broken. Since 2026-11-20, Stress in dms-saas has failed on every run with Error: dist/ is missing: run the build before this check. #93 (merged 2026-11-19 05:59 UTC) cut the Stress build step from checks:prepare + typecheck down to checks:prepare only. dms-saas builds its dist/ in typecheck, and its test script reads that dist/. In the Node.js job, pnpm run checks runs typecheck before the tests, so that job still passes. Stress runs no checks, so its tests found no build. The 11-19 nightly ran the workflow from before #93 and passed. The 11-20 nightly ran it from after #93 and failed all 3 runs.

Fix. Stress runs pnpm run --if-present typecheck after checks:prepare again, as #71 first designed it. --if-present means a caller with no typecheck script still passes. The README paragraph on Stress names the step. No input, default, secret or permission changes, so callers need no edit.

The other shared cause, GitHub billing, needs no change here: see the table and the note under it.

Nightly failures of 2026-11-20, by cause

History covers each repository's scheduled runs from 11-16 to 11-20, all it has. Steady means the same cause failed two or more nights in a row up to 11-20. Flaky means it failed once, or in 1 of the 3 Stress runs, and passed on other nights. New means 11-20 was the first failure, too soon to say.

Repository Failing job (11-20) Cause Where it lives Flaky or steady
brevo-private every job, 0 steps billing: GitHub-hosted runners blocked (Windows legs, Release) org billing, settled by Antony steady since 11-19
cdn every job, 0 steps billing: GitHub-hosted runners blocked (Windows legs, Release) org billing, settled by Antony steady since 11-19
dms-antelope-cloud every job, 0 steps billing: GitHub-hosted runners blocked (Windows legs, Release) org billing, settled by Antony steady since 11-19
mailchimp every job, 0 steps billing: GitHub-hosted runners blocked (Windows legs, Release) org billing, settled by Antony steady since 11-19
mqtt every job, 0 steps billing: GitHub-hosted runners blocked (Windows legs, Release) org billing, settled by Antony steady since 11-19
test-module-cloud every job, 0 steps billing: GitHub-hosted runners blocked (Windows legs, Release). It has no Windows job because it passes windows: false on purpose (test-module-cloud#7: its only test is the build) org billing, settled by Antony steady since 11-19
dms-saas Stress dist/ missing: #93 dropped typecheck from the Stress build this repository, fixed here steady since 11-20 (started with #93)
antelopejs Stress ajs project start while another run rebuilds and prunes (start-build-race test) times out at 5000 ms under load, 3 of 3 runs repo-local steady since 11-17
interface-core Stress AntelopeTestConfig.setup tests time out at mocha's default 2000 ms under load, 3 of 3 repo-local steady since 11-17
template-todo-typescript Stress before all hooks of todo API and task ownership (project boot) time out at 2000 ms under load, 3 of 3 repo-local steady since 11-17
mongodb Stress group mapper orderBy on an array key times out at 10000 ms under load, 3 of 3 repo-local steady since 11-18
payment-subscriptions Stress [fallback] a subscription that keeps failing to process fails an equality assertion under load, 1 of 3 repo-local steady since 11-18 (Stress red each night)
dms-media Stress registration.test.ts > names the explorer block by the name the DMS serves it under times out at 5000 ms, 1 of 3 repo-local steady since 11-19 (Stress red each night)
mysql Stress; Node 22/24 on Windows Stress: the probe tests (many relations, slow URL value) time out under load. Windows: .agents/skills/formal-*/check.mjs calls spawnSync("pnpm") with no shell → spawn pnpm ENOENT repo-local steady since 11-17
dms-database Stress; Node .nvmrc on Windows Stress: access write-back property test, data journeys and diagram layout routes time out at 2000 ms. Windows: the root before all seed hook times out at 2000 ms repo-local Stress steady since 11-17; Windows new
dms-assistant Test .nvmrc on Windows; vitest (windows-latest, sidecar) .agents/skills/formal-*/check.mjs calls spawnSync("pnpm") with no shell → spawnSync pnpm ENOENT; the sidecar's tool-guard and session-tool-results-read tests fail on Windows repo-local steady since 11-16
dms Test .nvmrc on Windows; backend (windows-latest, builder:1-3/5) e2e-kit tests on Windows: EPERM renaming run.json.next over run.json, write ECONNRESET; builder: a declaration file edited between two builder operations is not re-parsed repo-local Windows steady since 11-17; Stress flaky (11-17, 11-19)
dms-lang Node .nvmrc on Windows tests/package.test.mjs reads the pnpm pack output as JSON and gets Unexpected end of JSON input on Windows repo-local new
dms-mailing backend-tests on Windows after all hook removeRunUnlessFailed times out at 2000 ms on Windows repo-local new
file-storage-s3 Test .nvmrc on Windows root before all hook connects to the S3 service at 127.0.0.1:5005, which Windows legs do not start (ECONNREFUSED) repo-local new
api Stress formal-route-tree test times out at 2000 ms, 1 of 3 repo-local flaky
mollie Stress [mollie] the hosted page language gets undefined for the payer's language after the stand-in webhook answered "temporarily unavailable", 1 of 3 repo-local flaky
dms-api e2e (2) the monitor page assertion /for 3 min · checked hh:mm:ss/ depends on wall-clock time repo-local flaky

On 11-19 only, the e2e frontend-build step failed in dms-mailing, dms-media and dms-saas the same night. It passed on 11-20 with no change here, so it is not counted above.

Billing. The six private repositories ran no job on 11-19 and 11-20 (GitHub's annotation, quoted above). #103 merged at 03:45 UTC on 11-20, after that night's run at 02:17 (the run's referenced_workflows names fb14e4f). It moved their Linux legs, Stress and checks onto antelope-ci. Manual runs at 03:47 UTC passed on antelope-ci (cdn, brevo-private, mqtt). Their nightly Windows legs still need GitHub-hosted windows-latest, so they stay red while billing refuses jobs. Antony settles the billing; this repository changes nothing for it.

flowchart LR
  I[install, setup-commands, cache] --> P[checks:prepare]
  P --> T["typecheck (restored)"]
  T --> S[services]
  S --> R["pnpm run test x3 beside 4 busy loops per CPU"]
Loading

Related issue

ANT-3515, items 1 and 3 of its Scope. The change reaches callers when v2 moves to its merge commit.

Verification

The Stress steps run locally on a fresh clone of dms-saas (b11f00a), Node 24.22.2, pnpm 10.34.9, after pnpm install --frozen-lockfile.

Before, main's step pnpm run --if-present checks:prepare, then CI=1 pnpm run test:

Error: dist/ is missing: run the build before this check.
    at builtFiles (file:///…/dms-saas/packages/interface-dms-saas/scripts/check-exports.mjs:92:11)
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @antelopejs/interface-dms-saas@0.10.2 check:exports: `node scripts/check-exports.mjs`
 ELIFECYCLE  Test failed. See above for more details.
exit 1

After, the step's run text parsed out of this branch's ci-node.yml, then CI=1 pnpm run test:

--- step:
pnpm run --if-present checks:prepare
pnpm run --if-present typecheck
---
> @antelopejs/interface-dms-saas@0.10.2 check:exports
> node scripts/check-exports.mjs
Interface package exports are valid: 51 built modules, all reachable through 10 exports targets.
exit 0
  • actionlint .github/workflows/*.yml (1.7.12): no output, exit 0.
  • PNPM=$(command -v pnpm) node --test scripts/ci-tier/*.test.ts, which reads ci-node.yml: 75 pass, 0 fail.

No manual run can reach Stress: only a schedule event gives the nightly tier. The first nightly after v2 moves is the end-to-end check.

Follow-ups

  • existing ANT-3475: billing blocks the GitHub-hosted runners of the six private repositories (Windows legs, Release). Antony settles the billing; no windows: false and no skip in ci-node.yml. Done means: their next nightly runs its Windows legs.
  • proposed AntelopeJS/antelopejs: start-build-race test times out at 5000 ms under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/interface-core: AntelopeTestConfig.setup tests time out at 2000 ms under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/template-todo-typescript: the before all project boot times out at 2000 ms under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/mongodb: group-orderby-array-sort times out at 10000 ms under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/payment-subscriptions: [fallback] a subscription that keeps failing to process fails under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/dms-media: registration.test.ts explorer block test times out at 5000 ms under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/mysql: formal-*/check.mjs spawns pnpm with no shell (ENOENT on Windows), and the probe tests time out under Stress. Done means: the Windows legs and Stress pass.
  • proposed AntelopeJS/dms-database: the access write-back, data journeys and diagram layout routes tests time out under Stress, and the Windows seed hook times out at 2000 ms. Done means: Stress and the Windows leg pass.
  • proposed AntelopeJS/dms-assistant: formal-*/check.mjs spawns pnpm with no shell (ENOENT on Windows), and the sidecar's tool-guard and session-tool-results-read tests fail on Windows. Done means: both Windows jobs pass.
  • proposed AntelopeJS/dms: the e2e-kit tests fail on Windows (EPERM on the run.json rename, ECONNRESET), and builder shards 1-3 fail a declaration file edited between two builder operations. Done means: the Windows jobs pass.
  • proposed AntelopeJS/dms-lang: tests/package.test.mjs gets empty pnpm pack JSON on Windows. Done means: the Windows leg passes.
  • proposed AntelopeJS/dms-mailing: removeRunUnlessFailed times out at 2000 ms on Windows. Done means: backend-tests on Windows passes.
  • proposed AntelopeJS/file-storage-s3: the tests that need the S3 service run on Windows, where no service starts. Done means: the Windows leg passes.
  • proposed AntelopeJS/api: the formal-route-tree test times out at 2000 ms under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/mollie: the hosted page language loses the payer's language after a stand-in webhook error under Stress. Done means: Stress passes 3 of 3 runs.
  • proposed AntelopeJS/dms-api: the monitor page e2e assertion depends on wall-clock time. Done means: e2e passes on reruns at any minute.
  • proposed AntelopeJS/cloud, AntelopeJS/antelope-web, AntelopeJS/agent-skills, AntelopeJS/cloud-commit-url-probe: windows: false has no comment saying why. Done means: each ci.yml states its reason beside the input.

Checklist

  • The pull request title follows Conventional Commits.
  • I added or updated tests when behavior changed. No test covers the Stress steps; the proof above runs them.
  • I updated documentation when users or contributors are affected.
  • I documented breaking changes and migration steps. None.
  • I did not include credentials or unrelated changes.

Skills

no skill change

#93 left only checks:prepare in the Stress build step. A caller whose tests read the output of typecheck, such as dms-saas and its dist/, failed every Stress run of the nightly tier since.
@Upd4ting
Upd4ting marked this pull request as ready for review November 20, 2026 21:59
@Upd4ting
Upd4ting added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 886217c Nov 20, 2026
3 checks passed
@Upd4ting
Upd4ting deleted the ci/ant-3515-nightly-failures branch November 20, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant