[echoix]

I wanted to have something similar to the random order of tests I had added for pytest, but for gunittest. In the past, I had seen and fixed tests that changed the dataset that was used by later tests. The tests run later could depend on the side effects of that. Having a stable order hides these effects and interdependencies. When using pytest-xdist, shuffling the order of the tests/test files helps with tests using similar resources that cause contention; with gunittest here, only the test file order changes, not the order inside a test, which is still controlled by unittest. And since we still run one test at a time, resource contention isn't something changed here.

The final test report that is uploaded is sorted back so it is exactly how it looks right now, and the reports of two different runs with different seeds can be compared normally with a stable order.

I updated this work yesterday and launched 20 reruns against all the workflows, to try to catch eventual problems once merged, and the results are shown below.

I wanted to keep an API similar to the pytest plugin we already use. The first iteration of this, last month, was for Windows only, and activated by an env var, but I thought it would be useful to have for real.

See below for the rest of AI PR body and suggested commit message

Description

This adds options to grass.gunittest.main to run test files in random order, and turns them on for the thorough gunittest runs in CI on all platforms.

The seed option and its message follow pytest-randomly, the plugin already used for the pytest suite in CI:

# Random order with a newly generated seed:
python -m grass.gunittest.main --location nc_spm_full_v2beta1 --location-type nc --random-order
# Prints, e.g., "Using --randomly-seed=2349923583"

# Repeat the same order:
python -m grass.gunittest.main --location nc_spm_full_v2beta1 --location-type nc --randomly-seed 2349923583

# Reuse the seed of the previous run (read from the report in the output directory):
python -m grass.gunittest.main --location nc_spm_full_v2beta1 --location-type nc --randomly-seed last

• --random-order runs the test files in a random order from a new seed.
• --randomly-seed N runs them in the order generated from N. --randomly-seed last reuses the seed of the previous run. Both imply --random-order.
• Unlike pytest-randomly, which shuffles as soon as it is installed, random order is opt-in, so local runs and other users of grass.gunittest.main keep the fixed order.
• The seed is printed at the start of the run. It is also stored in the reports: random_seed in test_keyvalue_result.txt and a sentence in index.html. So the order of a CI run can be reproduced locally from the log or from the downloaded report.
• The reports list test files in discovery order whatever order they ran in, so reports from different runs can be compared directly:
  • testfiles.html is still written as each test file finishes, so a partial report is available during the run. At the end it is rewritten in discovery order.
  • The per-file lists in test_keyvalue_result.txt are sorted together.
  • The testsuite pages are built from the discovered list.
• Discovery itself does not change. Only the order the test files run in is shuffled, using random.Random(seed).
• CI: --random-order is added to test_thorough.sh (Ubuntu, macOS, CMake, Docker image test) and test_thorough.bat (Windows).
• Docs: new "Running test files in random order" section in the gunittest docs on running tests.

Without the new options, behavior and report contents are unchanged.

Motivation and context

The gunittest test files always run in the same alphabetical order. A test that passes only because of what an earlier test did, or that fails only after a particular other test, stays hidden as long as the order never changes. The pytest suite in CI already runs in random order with pytest-randomly. This brings the same protection to the gunittest suite, with the seed needed to reproduce an order-dependent failure.

Each test file runs in its own process and mapset. So this can expose dependencies through state that outlives a process, such as files in shared directories or the PERMANENT mapset, but not in-process state. Tests inside one test file still run in the usual unittest order.

How has this been tested?

• New pytest tests in python/grass/gunittest/tests/grass_gunittest_random_order_test.py run the invoker against a minimal location and a tree of testsuite directories. The tree includes names like a, a/b, a-b and a.b, which sort differently as strings than in a directory walk. The tests check that:
  • the default order is the discovery order
  • the same seed gives the same order and a different seed a different one
  • the HTML and key-value reports keep discovery order for a shuffled run
  • the seed appears in the reports only when used and can be read back for last
  • invalid seed values are rejected
• Ran grass.gunittest.main by hand with --random-order, --randomly-seed N, --randomly-seed last (same order as the previous run; an error when the previous run was not in random order) and without the options.
• Ran the thorough tests in random order repeatedly in CI on all platforms (see below).

Results of repeated runs in random order

To check whether random order turns up order-dependent failures in the current test suite, I ran the thorough tests of this branch repeatedly in CI, each run with its own seed:

CI job	Runs with tests	Test files per run	Failures not seen in fixed order
Ubuntu 24.04, all except temporal	18	294	0
Ubuntu minimum config, all except temporal	18	294	0
Ubuntu 24.04, temporal	18	48	0
Ubuntu minimum config, temporal	18	48	0
macOS	19	347	1, environmental
CMake (Ubuntu)	18	342	0
Windows (OSGeo4W)	20	356	1, environmental

That is 129 jobs, 128 distinct seeds and about 32,000 test file runs. Each job was compared with the fixed-order results of the same job on main.

No order-dependent failure was found. The two failures not seen in fixed order were both environmental:

• macOS: scripts/g.extension/testsuite/test_addons_download.py failed because the runner lost DNS resolution in the middle of the test (Could not resolve host: github.com). The other g.extension test files passed earlier in the same run, and the file passed in the other 18 runs.
• Windows: scripts/v.import/testsuite/test_parallel.sh got 49 instead of 50 maps because one of the 50 imports it runs at once hit database is locked in SQLite. This is a race inside the test itself, independent of which tests ran before, and the file passed in the other 19 runs.

The test files which fail on Windows in fixed order failed the same way in random order. python/grass/pygrass/raster/testsuite/test_raster_region.py, which is already flaky in fixed order (G_malloc: unable to allocate ...), failed in 11 of 20 runs: 8 times with the same memory error and 3 times by timing out after 300 s.

So random order does not add failures to CI with the current test suite, while an order dependency introduced later can surface and be reproduced from the printed seed.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• PR title provides summary of the changes and starts with one of the
  pre-defined prefixes
• My code follows the code style
  of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.

Suggested squash commit message and title

grass.gunittest: Add option to run test files in random order

A fixed order of test files can hide a test which passes only because
of what an earlier test did. Add a --random-order option to
grass.gunittest.main to run the test files in a random order, and a
--randomly-seed option to set the seed or reuse the last one with the
special value last, following pytest-randomly used for the pytest
suite. Random order stays opt-in. The seed is printed and stored in the
reports, so that the same order can be repeated.

Reports list test files in discovery order regardless of the order
they ran in, so that reports of different runs can be compared. The
main HTML page is still written while the tests run and rewritten in
discovery order at the end.

Run the thorough gunittest tests in CI in random order on all
platforms.

Developed with the help of an AI coding agent (Claude Code).

---

Developed with the help of an AI coding agent (Claude Code), under my review and direction.

[wenzeslaus]

Makes sense...but it is also another argument for migrating from gunittest. Please, see the comments.

"Types of changes" has checkboxes which GitHub interprets as TODOs so I actually almost skipped this because it looked like WIP.

The commits don't comply with AGENTS.md.

[wenzeslaus]

While random_seed is clear and I like it, in the tools, it turned out we are rarely using it and seed is just shorter and usually clear in the context. In fact, random_seed was only used by r.sim.water and I replaced that by seed in recent r.sim series of PRs. This is mostly internal, so no big deal either way. (For completeness: There is one or two places where we use seed with meaning of initial state.)

[echoix]

So what would you like?

[echoix]

Makes sense...but it is also another argument for migrating from gunittest. Please, see the comments.

"Types of changes" has checkboxes which GitHub interprets as TODOs so I actually almost skipped this because it looked like WIP.

The commits don't comply with AGENTS.md.

Thanks, and sorry about the checkboxes. I removed the unchecked "Types of changes" entries so the description no longer looks like a WIP.

I reworded both commits: the co-author and session trailers are gone, and AI use is acknowledged in the message text as AGENTS.md asks. I also rebased on current main and force-pushed.