[echoix]

Description

This adds options to grass.gunittest.main to run test files in random order, and turns them on for the thorough gunittest runs in CI on all platforms.

The seed option and its message follow pytest-randomly, the plugin already used for the pytest suite in CI:

# Random order with a newly generated seed:
python -m grass.gunittest.main --location nc_spm_full_v2beta1 --location-type nc --random-order
# Prints, e.g., "Using --randomly-seed=2349923583"

# Repeat the same order:
python -m grass.gunittest.main --location nc_spm_full_v2beta1 --location-type nc --randomly-seed 2349923583

# Reuse the seed of the previous run (read from the report in the output directory):
python -m grass.gunittest.main --location nc_spm_full_v2beta1 --location-type nc --randomly-seed last

• --random-order runs the test files in a random order from a new seed.
• --randomly-seed N runs them in the order generated from N. --randomly-seed last reuses the seed of the previous run. Both imply --random-order.
• Unlike pytest-randomly, which shuffles as soon as it is installed, random order is opt-in, so local runs and other users of grass.gunittest.main keep the fixed order.
• The seed is printed at the start of the run. It is also stored in the reports: random_seed in test_keyvalue_result.txt and a sentence in index.html. So the order of a CI run can be reproduced locally from the log or from the downloaded report.
• The reports list test files in discovery order whatever order they ran in, so reports from different runs can be compared directly:
  • testfiles.html is still written as each test file finishes, so a partial report is available during the run. At the end it is rewritten in discovery order.
  • The per-file lists in test_keyvalue_result.txt are sorted together.
  • The testsuite pages are built from the discovered list.
• Discovery itself does not change. Only the order the test files run in is shuffled, using random.Random(seed).
• CI: --random-order is added to test_thorough.sh (Ubuntu, macOS, CMake, Docker image test) and test_thorough.bat (Windows).
• Docs: new "Running test files in random order" section in the gunittest docs on running tests.

Without the new options, behavior and report contents are unchanged.

Motivation and context

The gunittest test files always run in the same alphabetical order. A test that passes only because of what an earlier test did, or that fails only after a particular other test, stays hidden as long as the order never changes. The pytest suite in CI already runs in random order with pytest-randomly. This brings the same protection to the gunittest suite, with the seed needed to reproduce an order-dependent failure.

Each test file runs in its own process and mapset. So this can expose dependencies through state that outlives a process, such as files in shared directories or the PERMANENT mapset, but not in-process state. Tests inside one test file still run in the usual unittest order.

How has this been tested?

• New pytest tests in python/grass/gunittest/tests/grass_gunittest_random_order_test.py run the invoker against a minimal location and a tree of testsuite directories. The tree includes names like a, a/b, a-b and a.b, which sort differently as strings than in a directory walk. The tests check that:
  • the default order is the discovery order
  • the same seed gives the same order and a different seed a different one
  • the HTML and key-value reports keep discovery order for a shuffled run
  • the seed appears in the reports only when used and can be read back for last
  • invalid seed values are rejected
• Ran grass.gunittest.main by hand with --random-order, --randomly-seed N, --randomly-seed last (same order as the previous run; an error when the previous run was not in random order) and without the options.
• Ran the thorough tests in random order repeatedly in CI on all platforms (see below).

Results of repeated runs in random order

To check whether random order turns up order-dependent failures in the current test suite, I ran the thorough tests of this branch repeatedly in CI, each run with its own seed:

CI job	Runs with tests	Test files per run	Failures not seen in fixed order
Ubuntu 24.04, all except temporal	18	294	0
Ubuntu minimum config, all except temporal	18	294	0
Ubuntu 24.04, temporal	18	48	0
Ubuntu minimum config, temporal	18	48	0
macOS	19	347	1, environmental
CMake (Ubuntu)	18	342	0
Windows (OSGeo4W)	20	356	1, environmental

That is 129 jobs, 128 distinct seeds and about 32,000 test file runs. Each job was compared with the fixed-order results of the same job on main.

No order-dependent failure was found. The two failures not seen in fixed order were both environmental:

• macOS: scripts/g.extension/testsuite/test_addons_download.py failed because the runner lost DNS resolution in the middle of the test (Could not resolve host: github.com). The other g.extension test files passed earlier in the same run, and the file passed in the other 18 runs.
• Windows: scripts/v.import/testsuite/test_parallel.sh got 49 instead of 50 maps because one of the 50 imports it runs at once hit database is locked in SQLite. This is a race inside the test itself, independent of which tests ran before, and the file passed in the other 19 runs.

The test files which fail on Windows in fixed order failed the same way in random order. python/grass/pygrass/raster/testsuite/test_raster_region.py, which is already flaky in fixed order (G_malloc: unable to allocate ...), failed in 11 of 20 runs: 8 times with the same memory error and 3 times by timing out after 300 s.

So random order does not add failures to CI with the current test suite, while an order dependency introduced later can surface and be reproduced from the printed seed.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing
  functionality to not work as before)

Checklist

• PR title provides summary of the changes and starts with one of the
  pre-defined prefixes
• My code follows the code style
  of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.

Suggested squash commit message

grass.gunittest: Add option to run test files in random order

A fixed order of test files can hide a test which passes only because
of what an earlier test did. Add a --random-order option to
grass.gunittest.main to run the test files in a random order, and a
--randomly-seed option to set the seed or reuse the last one, following
pytest-randomly used for the pytest suite. The seed is printed and
stored in the reports, so that the same order can be repeated.

Reports list test files in discovery order regardless of the order
they ran in, so that reports of different runs can be compared. The
main HTML page is still written while the tests run and rewritten in
discovery order at the end.

Run the thorough gunittest tests in CI in random order on all
platforms.

Developed with the help of an AI coding agent (Claude Code).

---

Developed with the help of an AI coding agent (Claude Code), under my review and direction.