Skip to content

BrowserTest: a skipped browser spec is reported as Passed #4590

Closed

Description

@bpamiri

What happens

wheels.wheelstest.BrowserTest.beforeAll() sets this.browserTestSkipped = true and returns quietly when:

  • the Playwright JARs are missing (Wheels.BrowserNotInstalled);
  • the JVM is unavailable;
  • the browser launch fails or times out;
  • CI is detected without opt-in.

browserDescribe()'s aroundEach then returns without calling the spec body. TestBox counts every such it as Passed, and the run reports totalSkipped: 0.

A spec whose only assertion can't pass (expect(...).toBe("probe")) reports 1 pass, 0 fail when Playwright is incomplete (for example, ~/.wheels/browser/lib holding playwright-*.jar and driver-*.jar but not driver-bundle, gson, Java-WebSocket or slf4j-*). Nothing in the output says the browser layer was unavailable. A local "green" browser run then proves nothing.

Expected

A skipped browser spec is reported as Skipped, with the reason.

Suggested fix (verified in an application subclass)

TestBox's skip(message) (BaseSpec.skip) marks the current spec Skipped. Registering a beforeEach inside browserDescribe() that calls it when the browser layer is unavailable gives 0 pass / 0 fail / N skipped, with no spec body run:

public void function browserDescribe(required string title, required any body) {
    var me = this;
    var innerBody = arguments.body;
    describe(arguments.title, function() {
        beforeEach(function() {
            if (me.browserTestSkipped) skip(me.$browserSkipReason());
        });
        // ... existing beforeEach($startBrowserContext) / aroundEach / afterEach ...
        innerBody();
    });
}

beforeAll() can record the reason in each catch branch: the missing JAR from Wheels.BrowserNotInstalled's message, the launch error's message, or "CI without WHEELS_BROWSER_CI_ENABLE".

Note: TestBox's skip argument to it() / describe() doesn't work for this. A closure there is evaluated when the suite is registered, before beforeAll() has decided whether to skip.

Environment: Wheels 4.0.x, Lucee 7.0.4, TestBox as bundled in vendor/wheels/wheelstest/system.

Activity

  1. bpamiri commented on Oct 15, 2026

    @bpamiri
    CollaboratorAuthor

    This is fixed on develop by #4263 (merged 2026-10-04), so it's in 4.2.0. The report was against 4.0.x.

    browserDescribe() now calls TestBox's skip() with a reason whenever the browser layer is unavailable, so those specs are reported as Skipped, not Passed. Verified on Lucee 7 + SQLite at develop 432ae4ec6, running directory=wheels.tests.specs.wheelstest:

    • Playwright not installed: pass 300 / fail 0 / error 0, totalSkipped 32. Every browserDescribe spec has status Skipped with Playwright is not installed. Run \wheels browser setup` to run browser specs.` Those bundles report pass 0.
    • WHEELS_CI=true without WHEELS_BROWSER_CI_ENABLE: the same 32 specs are Skipped, with Browser specs are skipped in CI (WHEELS_CI is set). Set WHEELS_BROWSER_CI_ENABLE=true to run them.

    Upgrading to 4.2.0 resolves it. Closing; please reopen if you still see a browser spec reported as Passed on 4.2.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions