[bpamiri]

Closes #4450.

Summary

Docs-harness wheels commands failed intermittently, or stalled on CI, when several started at the same moment.

Cause: concurrent JVM starts in one CLI home race in Lucee's OSGi bundle cache:

• the error is BundleException: Bundle symbolic name and version are not unique (org.lucee.xml.apis, commons-text, …);
• then Cannot invoke "javax.script.ScriptEngine.put(…)" because "engine" is null.

Lucee's loader runs with felix.cache.locking=false and installs its required bundles on every JVM start, so two JVMs starting together collide. verify-docs runs blocks four at a time, and since #4425 every run gets a fresh home.

Fix: runExec() now starts wheels commands one at a time, each WHEELS_START_GAP_MS (default 1 s) after the previous start. Once started, they still run in parallel, and other programs aren't gated. If a spawn throws, the next start isn't held up.

Measured with 12 runs of verify-docs on the sample at four-way concurrency:

Variation	Failed	Time
No stagger (develop)	4 of 12
Gap 0 ms (ordering only)	6 of 12	68 s
Gap 250 ms	0 of 12	78 s
Gap 500 ms	0 of 12	87 s
Gap 1 s (the default; room for slower CI JVMs)	0 of 12	112 s
-Dfelix.cache.locking=true, no gap	3 of 12	548 s
Copying an already-warm lucee-server/ into the home	3 of 8

Neither of the last two helps, so a start gate is the fix on our side. A draft upstream report for the Lucee/LuCLI side exists (not posted). Users running several wheels commands at once against one home can hit the same race.

Test 51 ("verify-docs reports pass/fail counts") now runs the real four-way path again. It no longer sets VERIFY_DOCS_CONCURRENCY=1 or skips on CI (both added as a stopgap in #4451). It keeps the 60 s command bound and the child's output in its failure message.

Tests

• test/exec.test.mjs:
  • wheels starts are staggered: a fake wheels called three times at once starts ≥ 600 ms apart with a 1 s gap;
  • WHEELS_START_GAP_MS parsing.
  • 9 / 9, three runs in a row.
• Full pnpm test:docs-harness: 76 / 76 in 83 s.

🤖 Generated with Claude Code

[bpamiri]

rev1-r3 review @ 66c69a7: code APPROVE (final pending PR CI)

This is the real fix for #4450, and it removes #4451's stopgap as agreed.

Start gate. nextWheelsStart() chains a promise per wheels call:

• Each call waits for the previous call's release.
• The release is a setTimeout(release, startGap()) fired right after spawn(), so the gap is measured between actual JVM starts, not between command completions, and commands still overlap once started.
• The finally also releases, so a spawn that throws doesn't wedge the queue. A double release just resolves an already-resolved promise.
• Only program === 'wheels' is gated.

Evidence. The measurement table makes the case. No gap fails 4 of 12 runs and ordering alone 6 of 12, while any gap of 250 ms or more fails 0 of 12. felix.cache.locking=true and a pre-warmed lucee-server/ don't help, which rules out the cheaper alternatives.

Test 51. It is back on the real four-way path, with no VERIFY_DOCS_CONCURRENCY=1 and no CI skip, and it keeps the 60 s bound and the child's output in the failure message.

New tests. A fake wheels (through WHEELS_BIN) called three times at once must start at least 600 ms apart with a 1 s gap; the slack covers the fake's own Node boot. There is also a startGap() parsing table.

Two notes, not blocking:

• Full-run wall time. The gate caps wheels starts at about one per second, so a full verify:docs with a few hundred {test:cli} / {test:compile} invocations now has a floor of roughly one second per invocation. Please compare the verify job duration on this PR's CI with recent develop runs. If it grew a lot, a CI default around 250–500 ms, which your table shows is already 0 of 12, is the knob.
• Slow-runner margin. The gap assumes a JVM finishes installing its bundles within the gap. That's where to look if this race ever shows up again on a slow runner. WHEELS_START_GAP_MS is already the escape hatch.

Gate: PR CI (verify must pass with test 51 on the real path). I'll post the final when it's green.

[bpamiri]

rev1-r3 review @ 66c69a7: APPROVE (final)

PR CI is now 20 pass / 0 fail / 0 pending, with 3 skipping. In verify the harness ran 76 tests, 76 pass, 0 fail, 0 skipped:

• verify-docs reports pass/fail counts is ok 61, a real pass on the four-way path with no stopgap. This is the test-51 test, now numbered 61 in this run.
• wheels starts are staggered… is ok 42.

The wall-time concern from my review is answered. This PR's verify job took 16m18s, against about 17½ min for the #4451 run and 11–17 min for other recent docs-verify runs. The 1 s start gap didn't add any measurable time.

Ready to merge. This closes #4450.