Skip to content

r.sim.water: Add pytest tests with synthetic data - #7968

Merged
wenzeslaus merged 36 commits into
OSGeo:mainfrom
wenzeslaus:add-r_sim_water-tests
Sep 29, 2026
Merged

wenzeslaus merged 36 commits into
OSGeo:mainfrom
wenzeslaus:add-r_sim_water-tests

Conversation

@wenzeslaus

@wenzeslaus wenzeslaus commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Add pytest tests for r.sim.water (SIMWE), 31 tests of the parameters and outputs on one-row and one-column slopes and 11 tests of flow on small synthetic landforms (pit, peak, ridge, V-shaped valley, saddle, hollow versus spur, terraces). They run on tiny synthetic domains and need no sample dataset, taking about 15 seconds on my local machine and about 19 seconds of worker time on the Ubuntu CI job. Each landform test states what the model must do on that landform for any random seed and also records (pins) the full depth grid for a fixed seed, so any change in the simulated result, intended or not, shows up in the tests.

These tests complement the existing gunittest, which pins full depth and discharge rasters at 1e-6 on a 100x100 cell area of the North Carolina sample dataset with realistic, spatially varying inputs, while this new suite needs no dataset and checks that each parameter moves the result in the physically expected direction, including several parameters the gunittest does not exercise. Writing the suite led to the investigation and identification of issues in the model, for example that multi-threaded runs lose water mass and are not reproducible with a fixed seed, two causes of which are fixed in #7964 and #7965, and that the flow_control documentation describes the opposite of what the code does, fixed in #7971. On the other hand, the new suite does only limited pinning of the outputs to specific numbers.

The eight landforms used by the landform tests (each test asserts a physical pattern of where water collects on its landform), with elevation on the left and simulated depth on the right, for seed 42 and the walker count each test uses:

The eight synthetic landforms used by the landform tests, elevation on the left and simulated depth on the right

Details (AI generated)

r_sim_water_test.py covers the parameters and outputs on one-row and one-column slopes, most of them created as rasters in module-scoped projects shared through temporary mapsets:

  • Rain, Manning's n, infiltration, slope steepness and the diffusion parameters (diffusion_coeff, hmax, halpha, hbeta) change depth in the direction physics requires. Where the kinematic wave gives an exponent (depth proportional to R^(3/5) and S^(-3/10)), the ratio of two runs is also checked against it.
  • Depth increases cell by cell downstream on both east-west and north-south slopes, checked at 2000 walkers.
  • Uniform rain, Manning's n and infiltration rasters give the same result as the equivalent scalar values.
  • Discharge falls within an order-of-magnitude mass-balance bracket, which also catches unit errors.
  • mintimestep is a floor on the time step: a value below the computed step changes nothing, a floor that moves walkers at most one cell per step leaves depth unchanged within noise, and a floor that makes walkers skip cells inflates depth by a factor predicted from the integration scheme's first-step accounting.
  • flow_control increases depth, and more trapping increases it further.
  • Duration, time-series output (-t with output_step), observation points with a logfile, walker output (also as a time series), the error output, and computing slopes when dx and dy are omitted.
  • Results are stable across seeds, and nprocs=4 agrees with nprocs=1 within Monte Carlo noise.
  • The -s flag gives different results on two runs. GRASS_RANDOM_SEED and SOURCE_DATE_EPOCH are removed from the tool's environment for this test, because when either is set, for example in reproducible builds, the automatic seed is fixed and the two runs would be identical.

The one-row domain of these tests, shown as the 1x6 east slope with unit dx and elevation 6 to 1 m on which test_depth_increases_downstream runs at 2000 walkers, with simulated depth per cell after two minutes rising toward the outlet, beside its 6x1 north-flowing counterpart; the parameter tests use a 1x5 version at the default ten walkers:

The 1x6 east slope and the 6x1 north slope, elevation and simulated depth per cell

Depth per cell for the pairs of runs the parameter tests compare on the 1x5 east slope at the default ten walkers, the tests' own conditions. The dips between neighboring cells are Monte Carlo noise, since with ten walkers and diffusion on, one walker leaving the one-row strip sideways removes a tenth of the mass from every cell below it. The tests therefore assert on total depth, shown at the end of each line, as their docstrings state:

Depth per cell for the runs each parameter test compares

r_sim_water_landforms_test.py simulates small 2D landforms: pit, peak, ridge, V-shaped valley, saddle, hollow versus spur, and a terraced slope. Each test states what the model must do on that landform for any seed: the pit collects water in its center, the peak and ridge shed water down their flanks, the valley concentrates flow on its floor, the saddle sends more water to its low quadrants, the hollow concentrates more than a spur of equal and opposite curvature, and terraces hold water on the flat steps. Symmetry and rotation tests (a landform against its 90-degree rotation) check that flow does not depend on direction. Tolerances and thresholds come from multi-seed sweeps, with a margin of at least twice the largest spread seen:

Hollow versus spur on shared color scales

On the same southward slope and shared color scales, the concave hollow must hold more than twice as much water on its axis as the convex spur, and its maximum depth must also be more than twice the spur's:

Ridge, valley and terrace next to their 90-degree rotations

A ridge, a valley and a terrace rotated 90 degrees and simulated on their own grids must, once the result is rotated back, match the original in total depth within 5% and in per-column sums within 10%.

Assertions that compare single cells or columns between simulations (peak, ridge, valley and the rotation tests) run with 10000 walkers. At the default of about two walkers per cell, per-column depth varies too much between seeds to be compared, and the seed-to-seed spread leveled off at 10000 walkers. The hollow and spur run with 1000 walkers, which keeps their depth pattern readable at no cost in runtime, and the other landform tests compare aggregates that are robust at the default walker count.

In addition, each landform's full depth grid for seed 42 is recorded in the test (pinned) to two decimal places and compared at 0.01 mm. This is the coarsest precision at which a change in the random number generator still fails the test on every landform (the smallest shift measured was 0.04 mm), so any change in the simulated result, intended or not, shows up in the tests.

The elevation surfaces are NumPy arrays passed directly to the tool, so the shape of each landform is visible in the test code and cannot drift from a separate description of it. Rotated variants are derived with np.rot90.

All tests use a fixed seed with nprocs=1. Threads currently share the random number generator state, so multi-threaded runs are not reproducible even with a fixed seed. Only the nprocs test runs with more than one thread, and it compares totals within Monte Carlo noise.

The per-landform figures below print the value in every cell.

A square bowl must collect most water in its center cell, which is the domain maximum and holds over 30% of the total depth, with depth decreasing symmetrically outward:

Pit

A conical peak must shed water outward symmetrically: the second ring around the summit holds more per cell than the first, and the flat summit itself holds more than the first ring because its water leaves only by diffusion, which is an artifact of the computed slope being zero there:

Peak

A ridge must shed water down both flanks symmetrically: the lower flank columns hold more than the shoulders next to the crest, and the flat crest holds more than its shoulders, an artifact of the computed slope being zero on the crest:

Ridge

A V-shaped valley draining south must concentrate water on its floor, more than three times the hillslope mean with the given settings and domain size, with floor depth increasing downstream and the two sides mirroring each other:

Valley

A saddle must send more water into its low east and west quadrants than into its high north and south quadrants, and the saddle point must be wetter than the high-quadrant mean:

Saddle

On the same southward slope, a concave hollow must hold more than twice as much water on its axis as a convex spur, with the hollow axis its wettest column:

Hollow

A convex spur spreads water away from its axis, so its axis holds less than half of what the hollow axis holds. Because the computed slope on the crest is zero, the model keeps water there and the crest is not its driest column, an artifact of the slope computation rather than a feature of real spurs:

Spur

The same spur and the hollow at 10000 walkers, which removes the remaining walker noise of the 1000-walker run above, with the tool's own downhill directions drawn on the elevation. On the spur the directions diverge from the crest and about four fifths of the water leaves through the east and west edges, so depth grows outward from the shoulders as expected. Two artifacts add to that pattern. The wet crest column, about 40 percent too deep, comes from its computed cross-slope being zero, and the wet edge columns and southern corners come from the 3x3 kernel halving the slope at the domain edge. The hollow funnels all of its water down its axis:

Spur and hollow at 10000 walkers with the tool's downhill directions

Terraced slopes must hold water on the flat steps rather than on the steep drops: each step holds more than the drop above it and more than the step above it:

Terrace

Suggested squash-merge commit message:

r.sim.water: Add pytest tests with synthetic data

Add 42 pytest tests for r.sim.water (SIMWE) on tiny synthetic domains
that need no sample dataset and run in 15 to 20 seconds, 31 tests of the
parameters and outputs on one-row and one-column slopes and 11 tests of
flow on small landforms (pit, peak, ridge, valley, saddle, hollow versus
spur, terraces).

The slope tests check that rain, Manning's n, infiltration, slope
steepness and the diffusion parameters move depth in the physically
expected direction, with the kinematic-wave exponents where they apply,
that depth increases cell by cell downstream, that raster and scalar
parameters agree, that discharge respects a mass-balance bracket, that
mintimestep acts as a floor with a predictable accuracy cost, and that
flow_control, duration, time-series output, observation points, walker
output, the error output, internally computed slopes, the -s flag and
nprocs behave as documented.

The landform tests assert what the model must do on each landform for
any random seed, with tolerances from multi-seed sweeps, and record the
full depth grid for seed 42 to two decimal places, compared at 0.01 mm,
so that any change in the simulated result shows up. Assertions on
single cells or columns run with 1000 or 10000 walkers, where the
seed-to-seed spread levels off. Elevation surfaces are NumPy arrays
passed directly to the tool, and rotated variants are derived with
np.rot90.

The suite complements the gunittest on the North Carolina sample
dataset, which pins realistic rasters at 1e-6 but needs the dataset and
cannot tell a wrong parameter direction from a right one. All tests use
a fixed seed with nprocs=1 because multi-threaded runs are not
reproducible.

The tests, the seed sweeps and the tolerance analysis were done with
substantial AI assistance (Claude Code) and reviewed by the author.

The tests, the seed sweeps and the tolerance analysis were done with Claude Code (Fable and Opus) and reviewed by me.

landforms_make_figures.py
spur_vs_hollow_figure.py
spur_vs_hollow_runs.py
tiny_domains_make_figures.py

…ow for that to cut the runtime slightly (by 0.5s on a good day)
Add 11 tests using 2D synthetic landscapes (pit, peak, ridge, valley,
saddle, hollow vs spur, terraced slope) with theoretical assertions
about spatial depth patterns and full-grid regression checks. Rotation
tests verify directional isotropy. All tests run in ~3 s.

Tests were generated with the help of AI (Claude).
…ance for diff between single and multi-thread to difference between seeds (rather than assuming it should fit within a tighter tolerance with the given seed and thread combo).
Introduce module-scoped session fixtures for the shared long-slope and
diffusion domains and read-only landform fixtures, so each base landform
and the diffusion reference run is computed once and reused by its shape
and rotation tests. Fold run_sim_error into a direct call, replace the
hand-rolled approx loops with numpy.testing.assert_allclose, and drop the
redundant np.asarray wrapping.

Developed with substantial assistance from an AI agent (Claude).
…tions

Replace the discharge smoke check (sum > 0) with an order-of-magnitude
mass-balance bracket, give the mintimestep test a two-sided ratio bound
instead of the unfalsifiable rel=2.0 (which accepted any positive value),
and relax the time-series map count to a robust lower bound that does not
depend on the stochastic last-walker exit time.

Developed with assistance from an AI agent (Claude).
Run the rotation-invariance tests with 10000 walkers (instead of the
~70-240 tool default on these small grids) so the per-column profile
comparison is meaningful, and set the rotation tolerances from a
multi-seed orientation-spread measurement. Reduce the diffusion tests
from 400000 to 40000 walkers: their assertions are structural
inequalities whose relative margins are walker-count independent (held
across 20 seeds from 2000 to 400000).

Developed with assistance from an AI agent (Claude).
test_random_seed_flag asserts that two runs with -s differ, but
G_srand48_auto() takes a fixed seed from GRASS_RANDOM_SEED or
SOURCE_DATE_EPOCH when either is set, for example in reproducible
builds, which makes the runs identical. Remove the two variables from
the environment passed to the tool so the test exercises the time and
PID based seeding it is meant to cover.
…meters

The peak and ridge assertions held for seed 42 by luck at the default
walker count: across 13 seeds, ring symmetry on the peak failed for
about half of them, and the edge-versus-crest comparison on the ridge
rested on a false assumption, the crest being the second wettest column.
Run both tests at 10000 walkers with tolerances derived from seed
sweeps, assert interior properties only (accumulation down the flanks
and outward from the summit) to stay clear of edge effects, and grow the
peak to 7x7 so a second interior ring exists. The hollow-versus-spur
test compared two bowls; replace the surfaces with a parabolic
cross-section of equal curvature and opposite sign, on which the hollow
holds at least 3.6 times the spur's axis depth in every seed.

Pin every landform's full grid in millimeters with two decimals against
a 0.01 mm tolerance, the coarsest at which a change in the random number
generator still moves at least one cell (0.04 mm) on every landform,
instead of loose tolerances that left most cells unconstrained.

Give the elevation surfaces as arrays passed straight to the tool, which
reproduces the r.mapcalc result bitwise, so the shape is visible in the
code and no docstring table can drift from an expression. This removes
the per-test project setup and four one-consumer fixtures, and collapses
three rotation tests into one parametrized test that derives the rotated
surface with np.rot90.

The seed sweeps, the hollow and spur redesign, and the tolerance analysis
were done with substantial AI assistance (Claude Code).
Rename the discharge and nprocs tests after their assertions, say
duration where iterations no longer apply, and drop stale comments on the
time-series map count and an old walker count. State the measured reason
the saddle point is wet: zero computed gradient plus rain converging from
the center column. Note that the rotation tests pin nothing.
@github-actions github-actions Bot added the raster Related to raster data processing label Sep 28, 2026
@wenzeslaus
wenzeslaus force-pushed the add-r_sim_water-tests branch from 8f57f23 to af59787 Compare September 29, 2026 14:19
@wenzeslaus
wenzeslaus enabled auto-merge (squash) September 29, 2026 18:14
@wenzeslaus
wenzeslaus disabled auto-merge September 29, 2026 18:18
@wenzeslaus
wenzeslaus merged commit 6902ccf into OSGeo:main Sep 29, 2026
27 checks passed
@wenzeslaus
wenzeslaus deleted the add-r_sim_water-tests branch September 29, 2026 18:19
@github-actions github-actions Bot added this to the 8.6.0 milestone Sep 29, 2026
wenzeslaus added a commit to wenzeslaus/grass that referenced this pull request Sep 29, 2026
Add 42 pytest tests for r.sim.water (SIMWE) on tiny synthetic domains
that need no sample dataset and run in 15 to 20 seconds, 31 tests of the
parameters and outputs on one-row and one-column slopes and 11 tests of
flow on small landforms (pit, peak, ridge, valley, saddle, hollow versus
spur, terraces).

The slope tests check that rain, Manning's n, infiltration, slope
steepness and the diffusion parameters move depth in the physically
expected direction, with the kinematic-wave exponents where they apply,
that depth increases cell by cell downstream, that raster and scalar
parameters agree, that discharge respects a mass-balance bracket, that
mintimestep acts as a floor with a predictable accuracy cost, and that
flow_control, duration, time-series output, observation points, walker
output, the error output, internally computed slopes, the -s flag and
nprocs behave as documented.

The landform tests assert what the model must do on each landform for
any random seed, with tolerances from multi-seed sweeps, and record the
full depth grid for seed 42 to two decimal places, compared at 0.01 mm,
so that any change in the simulated result shows up. Assertions on
single cells or columns run with 1000 or 10000 walkers, where the
seed-to-seed spread levels off. Elevation surfaces are NumPy arrays
passed directly to the tool, and rotated variants are derived with
np.rot90.

The suite complements the gunittest on the North Carolina sample
dataset, which pins realistic rasters at 1e-6 but needs the dataset and
cannot tell a wrong parameter direction from a right one. All tests use
a fixed seed with nprocs=1 because multi-threaded runs are not
reproducible at this point.

The tests, the seed sweeps and the tolerance analysis were done with
substantial AI assistance (Claude Code) and reviewed by the author.
wenzeslaus added a commit that referenced this pull request Sep 29, 2026
The manual and the option description called flow_control a permeability
ratio, the probability of particles to pass through a structure, but the
code traps a particle (reverses its velocity and scales it to 10%) when a
uniform random number is at or below the cell value, so higher values
increase depth. The comparison, the name of the trap array, the skip for
cells with value 0, and the documentation text are all unchanged since
the first commit in the repository (db49180, 2008, then under
raster/simwe/), where the option was still named traps, so the code is
not a later regression and is self-consistent only under the trapping
reading. Correct the documentation rather than the code to keep results
unchanged for existing users. r.sim.sediment does not expose this
parameter. The test test_flow_control_increases_depth added in #7968
asserts the behavior the new text describes.

The history analysis and the wording were done with AI assistance
(Claude Code).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module Python Related code is in Python raster Related to raster data processing tests Related to Test Suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants