[echoix]

Description

Investigated a hypothesis: on Windows, the first call to a C-based GRASS
tool/library within a given process can fail while later calls in the same
process succeed, and a fixed test order can hide this indefinitely because
whichever test happens to run first never changes.

Two real fixes, plus infrastructure to guard against this bug class going
forward:

• Fixed r.kappa's test_file testsuite test: it held a NamedTemporaryFile
  handle open while passing its path to r.kappa as the output file, which
  Windows rejects (a second process can't open a file the creator still holds
  open; POSIX allows it). Fixed the same way as r.mfilter's testsuite fix in
  r.mfilter: Avoid reopening NamedTemporaryFile on Windows in testsuite OSGeo/grass#7878 (grass.script.core.tempfile() instead), and removed the
  now-unneeded @xfail_windows marker.
• Windows CI (osgeo4w.yml) ran the pytest suite single-worker with a fixed
  order, unlike Linux CI's pytest-xdist + needs_solo_run split. Added
  pytest-xdist and pytest-randomly, mirroring Linux CI's structure.
• Added an opt-in GRASS_GUNITTEST_RANDOM_ORDER environment variable to
  grass.gunittest.loader.discover_modules() (it always sorted directories
  and files alphabetically before), and enabled it by default for the
  Windows thorough-test run. test_thorough.sh/.bat run the entire
  gunittest testsuite tree as a single long-lived process in a fixed
  traversal order — a more precise structural match for "fixed order hides
  an order-dependent bug" than xdist workers, since no parallelism is even
  needed for the mechanism to apply.

Investigation result: across roughly 56 Windows CI job runs (30 with
randomized pytest-xdist order, 15 with randomized gunittest discovery order,
plus verification runs), results were byte-for-byte identical regardless of
order — the same 8 pre-existing gunittest failures every time, all of which
match a set already catalogued by #682 as pre-existing, non-order-related
issues (network dependency, missing OpenGL on a headless runner, a numpy
doctest version difference, an md5sum mismatch, etc.), not something this
investigation should chase further here. No order-dependent failure was
found. That's a real (negative) result, not a dead end: the pytest-randomly
and GRASS_GUNITTEST_RANDOM_ORDER additions stay as permanent, validated
(15 real CI runs, no new noise introduced) protection against this bug class
recurring, even though it didn't reproduce today.

One genuine bug surfaced along the way —
python/grass/pygrass/raster/testsuite/test_raster_region.py fails with a
self-contained FERROR: Input window changed while maps are open for read
inside raster2numpy() handling, unrelated to test order — will be filed and
fixed as a separate, independent PR to keep it out of this one.

Motivation and context

doc/development/testing.md documents that xdist workers are long-lived
processes handling many test files sequentially ("several [tests] per
process"), so state and initialization order can leak between unrelated
tests sharing a worker. Windows CI wasn't exercising randomized order at all
in either test runner, so this class of bug had no way to surface, whether
or not one is currently present.

How has this been tested?

• The r.kappa fix mirrors an already-merged, already-verified fix
  (r.mfilter, r.mfilter: Avoid reopening NamedTemporaryFile on Windows in testsuite OSGeo/grass#7878) for the identical anti-pattern.
• The CI workflow changes and the grass.gunittest.loader randomization were
  validated across dozens of real Windows CI runs during the investigation
  (see above) before being kept as permanent defaults; the final state on
  this PR has also been run through normal Linux (pytest.yml) and Windows
  (osgeo4w.yml) CI.

Windows CI timing: xdist vs. single worker

Compared Windows CI (osgeo4w.yml) job data with pytest-xdist enabled (55
job records from 13 runs across this investigation's branches) against
single-worker pytest (118 job records spanning 11 months of main history,
2025-07-25 to 2026-06-21).

pytest step timing. xdist splits the old single step into two — a
parallel step plus a new needs_solo_run serial step, each paying its own
test-collection overhead — compared here against the combined total:

	n	mean	median	min	max
Single worker (main, before)	5*	15.1 min	13.3 min	10.9 min	19.5 min
xdist parallel step alone	53	10.7 min	10.9 min	6.7 min	13.7 min
xdist parallel + solo combined	8**	12.0 min	12.0 min	9.6 min	13.8 min

* GitHub's job-log/step retention (roughly 3-4 months) limits step-level
"before" data to the 5 most recent main runs; treat as indicative, not
definitive.
** Only the 8 full-scope, PR-triggered jobs ran both steps; 45
workflow_dispatch fishing-attempt jobs used earlier in this investigation
intentionally skipped the solo/gunittest/thorough steps to isolate the
parallel step, so they only contribute to the parallel-step-alone row.

Net: the pytest step is about 20% faster with xdist, even after paying for
a second collection pass.

Whole-job duration (compile + all test steps; not expected to move,
since pytest is a small slice of it):

	n	mean	median
Single worker (main, before)	114	84.5 min	84.4 min
xdist, full-scope PR jobs	5	84.0 min	90.3 min

No meaningful difference — the job is dominated by MSYS2 setup, compiling
GRASS, and the (unchanged) single-worker gunittest thorough-test step.

Timeouts / hangs. 0 GitHub Actions timed_out conclusions in either
group (0/53 xdist, 0/118 single-worker). A handful of cancelled jobs in
both groups (3/55 xdist, 4/118 single-worker) were checked individually and
are routine cancel-in-progress supersession from newer pushes to the same
branch, not hangs or timeouts.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist

• PR title provides summary of the changes and starts with one of the
  pre-defined prefixes
• My code follows the code style
  of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.

---

Developed with the help of an AI coding agent (Claude Code), under human
review and direction throughout, including a live back-and-forth on how to
interpret the CI results and where to take the investigation next.

🤖 Generated with Claude Code

[echoix]

Split the r.kappa testsuite fix into its own PR: #694. Reverted it here (commit 5756390) so it doesn't land twice; this PR now only carries the Windows CI randomization work.

---

Generated by Claude Code

[echoix]

Is there a cleaner way to have the random order in gunittest? Like a real option, with the possibility of giving the same seed again? Also, if tests are run not in order, can the generated test report be kept in order though, so it can be compared more easily between runs?