Skip to content

fix(docs): run the verify-docs counts test serially and show its output - #4451

Merged
bpamiri merged 1 commit into
developfrom
peter/docs-harness-t51-diagnostics
Oct 6, 2026
Merged

bpamiri merged 1 commit into
developfrom
peter/docs-harness-t51-diagnostics

Conversation

@bpamiri

@bpamiri bpamiri commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Refs #4450.

Summary

The docs-harness test "verify-docs reports pass/fail counts" fails intermittently, and since the harness step became blocking (#4444) it fails "Verify docs" on PRs. On CI it ran into the 240 s command bound from #4447 and failed 2 !== 0; before #4447 it hung.

Cause (now visible locally, because the test prints the child's output):

  • Several wheels JVMs starting at once in the run's fresh isolated CLI home (fix(docs): run verify:docs and its harness tests in a throwaway CLI home #4425) race in Lucee's OSGi bundle cache.
  • The error is org.osgi.framework.BundleException: Bundle symbolic name and version are not unique (org.lucee.xml.apis, commons-text, …), followed by Cannot invoke "javax.script.ScriptEngine.put(…)" because "engine" is null.
  • verify-docs runs the per-block examples four at a time, so the sample's {test:cli} and {test:compile} blocks start JVMs together. The fixture's wheels new then failed all 4 retries, and verify-docs exited 2.
  • Locally that failed in 4 of 12 runs. With VERIFY_DOCS_CONCURRENCY=1 it passed 8 of 8. Cloning the real home's lucee-server/ into the isolated home did not help (3 of 8 failed), so the race isn't only a fresh context's first start.

This PR changes only this test:

  • It runs the sample's two per-block examples one at a time (VERIFY_DOCS_CONCURRENCY=1). With two examples, that costs nothing.
  • It bounds each command at 60 s (WHEELS_EXEC_TIMEOUT_MS), so a stall surfaces in a minute, not four.
  • On failure it prints the child's stdout and stderr as test diagnostics, and in the assertion message. Before, a failure showed only 2 !== 0.
  • On CI, if it still fails, it's skipped (citing docs harness: 'verify-docs reports pass/fail counts' stalls on CI #4450) rather than failing the blocking step. Locally a failure still fails.

The full verify:docs run still uses 4-way concurrency and can hit the same race. Fixing that, by serializing JVM starts or warming the bundle cache properly, is #4450, which stays open.

Tests

  • 4 of 4 local runs of the test pass.
  • Forcing the child to fail (WHEELS_BIN=/nonexistent) with CI=1 marks it skipped and prints the diagnostics. Without CI, it fails with the output in the message.

🤖 Generated with Claude Code

Several wheels JVMs starting at once in the fresh isolated home race in
Lucee's OSGi bundle cache and fail (locally) or stall (CI), so the test failed
intermittently and, with the harness step blocking, failed Verify docs. The
test now runs its two blocks one at a time (8/8 locally; 4-way failed 4/12),
bounds each command at 60 s, prints the child's output when it fails, and is
skipped on CI if it still fails while #4450 is open.

Refs #4450.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Peter Amiri <peter@alurium.com>
@bpamiri bpamiri added this to the v4.2.0 milestone Oct 6, 2026
@bpamiri

bpamiri commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

rev1-r3 review @ 6b81574: code APPROVE (final pending PR CI)

This meets the conditions agreed for a temporary skip:

  • Narrow: only the verify-docs reports pass/fail counts test changes, in one test file. The skip applies only when CI is set and the child exits non-zero. Locally a failure still fails, and a passing run still asserts code === 0 and /passed/.
  • Cited in-line: the comment and the t.skip() reason both name docs harness: 'verify-docs reports pass/fail counts' stalls on CI #4450, which is open on the v4.2.0 milestone and covers the real fix: the full verify:docs at 4-way concurrency can hit the same race.
  • Diagnosable: on failure the child's stdout and stderr go out as t.diagnostic lines and into the assertion message, so the next CI occurrence shows exactly which command failed. Before this, a failure showed only 2 !== 0.
  • Root cause written down: concurrent wheels JVM starts race in Lucee's OSGi bundle cache in the fresh isolated home (Bundle symbolic name and version are not unique). It reproduced in 4 of 12 runs at concurrency 4 and 0 of 8 serially, and copying lucee-server/ didn't help. That's good evidence the fix belongs in JVM-start serialisation or bundle-cache warm-up, not in this test.

Mechanics check: verify-docs.mjs:89 reads VERIFY_DOCS_CONCURRENCY, and runEntry merges process.env with the overrides, so the child's runExec picks up WHEELS_EXEC_TIMEOUT_MS=60000 through defaultExecTimeout().

Housekeeping: #4450 has the milestone but no assignee on GitHub. Please assign it, so it's clear who owns un-skipping this.

Gate: PR CI. I'll post the final when it's green.

@bpamiri

bpamiri commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

rev1-r3 review @ 6b81574: APPROVE (final)

PR CI is now 20 pass / 0 fail / 0 pending, with 3 skipping. In verify the harness ran 74 tests, 74 pass, 0 fail, and verify-docs reports pass/fail counts passed outright (ok 59, not # SKIP). Running it serially was enough on this run, and the CI skip didn't need to fire.

Ready to merge. The real fix, serialising JVM starts or warming the bundle cache for the full 4-way verify:docs, stays tracked in #4450.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant