Skip to content

grass.script, grass.app, grass.temporal: Strengthen tests found weak by mutation testing - #753

Open
echoix wants to merge 5 commits into
mainfrom
strengthen-weak-tests
Open

echoix wants to merge 5 commits into
mainfrom
strengthen-weak-tests

Conversation

@echoix

@echoix echoix commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Description

Strengthen existing tests that executed code but would not fail if it were wrong, and fix a leak in RegionManager that one of the new tests found. Each topic is its own commit:

  • grass.app: the mapset lock and unlock tests were smoke tests checking only the exit status. Now check the lock file, a failed lock on a locked mapset, forced lock removal, and the error messages of failed project and mapset creation.
  • grass.temporal: datetime_math tests for singular unit names, counts other than one, decrementing by a string, singular units in adjust_datetime_to_granularity, and intervals spanning days in compute_datetime_delta.
  • grass.script: MaskManager, RegionManager and RegionManagerEnv restoring the previous state, RegionManager removing the region it saved, and raster_info value conversions (previously only tested indirectly).
  • grass.script fix: a nested RegionManager did not remove its saved region, because g.remove was in the else branch of deactivate() that only runs when WIND_OVERRIDE was not set before. It is now removed in both cases, like MaskManager does.

Motivation and context

Mutation testing (one small change to the code at a time, such as deleting a call or swapping an operator, then running the tests that cover that line) showed lines which are executed by tests but where no test fails when the line is changed. The new tests were written to fail on those changes. The nested RegionManager leak was found this way.

Open question, not changed here: after mapset lock --force-remove-lock on an already locked mapset, the old lock is removed but no new lock exists, although lock_mapset() documents returning the new lock file. The new test does not assert either behavior.

How has this been tested?

Mutants killed by the tests before and after, with the same mutants of the same source:

Module Before After
grass/temporal/datetime_math.py 395 475
grass/app/cli.py 121 128
grass/script/raster.py 128 148

The new tests pass, and the nested RegionManager test fails without the fix. Ruff check and format pass for the changed files.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • PR title provides summary of the changes and starts with one of the
    pre-defined prefixes
  • My code follows the code style
    of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.

The tests were written with AI assistance, and the expected values were checked against the behavior of the code.


Generated by Claude Code

The mapset lock and unlock tests only checked the exit status, so a lock
command which did nothing would pass. Check that the lock file is created
and removed, that locking a locked mapset fails and keeps the lock, that
a lock can be forced, and that unlocking a mapset without a lock is not
an error. Check the error messages printed when creating a project or a
mapset fails.

Found with mutation testing, run on the existing tests to look for
changes of the code which none of them detect. Written with AI
assistance, and the expected values were checked against the behavior
of the code.
Test singular unit names and counts other than one for incrementing, test
decrementing by a string, including months and years, and test intervals
spanning days for compute_datetime_delta, where hours and minutes are
computed over days.

Found with mutation testing, run on the existing tests to look for
changes of the code which none of them detect. Written with AI
assistance, and the expected values were checked against the behavior
of the code.
Check that MaskManager restores a mask name set before it started, that
RegionManager removes the region it saved and that a nested RegionManager
makes the outer region active again. Add tests of raster_info, which was
only exercised indirectly: the extent and resolution are numbers, and the
minimum and maximum are numbers or None for a raster with only nulls.

Found with mutation testing, run on the existing tests to look for
changes of the code which none of them detect. Written with AI
assistance, and the expected values were checked against the behavior
of the code.
RegionManager.deactivate() removed the region it saved only when no
WIND_OVERRIDE was set before activation. In a nested RegionManager the
inner one found the region of the outer one, restored it, and left its
own region in the mapset. Remove the saved region in both cases, like
MaskManager does with its mask.

Found by checking whether the tests detect the removal of the g.remove
call, with AI assistance.
RegionManagerEnv saves the value of GRASS_REGION before activation, but
no test checked that it is restored when managers are nested.

Found with mutation testing, with AI assistance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants