Skip to content

CI: Run Windows pytest suite with xdist and randomized order - #695

Closed
echoix wants to merge 3 commits into
mainfrom
windows-pytest-xdist-random-order
Closed

echoix wants to merge 3 commits into
mainfrom
windows-pytest-xdist-random-order

Conversation

@echoix

@echoix echoix commented Sep 6, 2026 •

Copy link
Copy Markdown
Owner

Description

Windows CI runs the pytest suite with a single worker and always in the same
order. Install pytest-xdist and pytest-randomly and mirror the parallel /
needs_solo_run split that pytest.yml already uses on Linux.

Motivation and context

xdist workers are long-lived processes which handle several test files each,
so state left behind by one test can affect a later one running in the same
worker. With a single worker and a fixed order, whichever test runs first
never changes, so a failure which depends on the order has no way to show up.
Randomizing the order is what makes that kind of failure surface.

Linux CI does not use pytest-randomly yet; it can be added there in a
separate step if this turns out to be useful.

How has this been tested?

Windows CI on this branch, and about 30 runs of the randomized job while
looking for an order-dependent failure on Windows. The results were identical
in all of them (the same pre-existing failures, none of them related to the
order), so the randomization does not add noise to the job.

Windows CI timing: xdist vs. single worker

Compared Windows CI (osgeo4w.yml) job data with pytest-xdist enabled (55
job records from 13 runs across this investigation's branches) against
single-worker pytest (118 job records spanning 11 months of main history,
2025-07-25 to 2026-06-21).

pytest step timing. xdist splits the old single step into two — a
parallel step plus a new needs_solo_run serial step, each paying its own
test-collection overhead — compared here against the combined total:

n mean median min max
Single worker (main, before) 5* 15.1 min 13.3 min 10.9 min 19.5 min
xdist parallel step alone 53 10.7 min 10.9 min 6.7 min 13.7 min
xdist parallel + solo combined 8** 12.0 min 12.0 min 9.6 min 13.8 min

* GitHub's job-log/step retention (roughly 3-4 months) limits step-level
"before" data to the 5 most recent main runs; treat as indicative, not
definitive.
** Only the 8 full-scope, PR-triggered jobs ran both steps; 45
workflow_dispatch fishing-attempt jobs used earlier in this investigation
intentionally skipped the solo/gunittest/thorough steps to isolate the
parallel step, so they only contribute to the parallel-step-alone row.

Net: the pytest step is about 20% faster with xdist, even after paying for
a second collection pass.

Whole-job duration (compile + all test steps; not expected to move,
since pytest is a small slice of it):

n mean median
Single worker (main, before) 114 84.5 min 84.4 min
xdist, full-scope PR jobs 5 84.0 min 90.3 min

No meaningful difference — the job is dominated by MSYS2 setup, compiling
GRASS, and the (unchanged) single-worker gunittest thorough-test step.

Timeouts / hangs. 0 GitHub Actions timed_out conclusions in either
group (0/53 xdist, 0/118 single-worker). A handful of cancelled jobs in
both groups (3/55 xdist, 4/118 single-worker) were checked individually and
are routine cancel-in-progress supersession from newer pushes to the same
branch, not hangs or timeouts.


Written with the help of Claude Code, under my review.

Windows CI ran pytest single-worker with a fixed test order, unlike
Linux CI's pytest-xdist + needs_solo_run split. Under a fixed order,
whichever test happens to run first in a given worker never changes,
so an order-dependent bug that only breaks the first call to a
C-based GRASS tool/library in a fresh worker process can stay hidden
indefinitely. Install pytest-xdist and pytest-randomly and mirror
Linux CI's parallel / needs_solo_run split so this class of bug
surfaces instead of hiding.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@echoix echoix closed this Sep 12, 2026
@echoix
echoix deleted the windows-pytest-xdist-random-order branch September 12, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant