Skip to content

r.sim.water: Add a benchmark for the number of threads - #8002

Merged
wenzeslaus merged 9 commits into
OSGeo:mainfrom
wenzeslaus:rsim-water-benchmark
Oct 8, 2026
Merged

wenzeslaus merged 9 commits into
OSGeo:mainfrom
wenzeslaus:rsim-water-benchmark

Conversation

@wenzeslaus

@wenzeslaus wenzeslaus commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Adds a benchmark script for r.sim.water with different numbers of threads and its figures in a Performance section of the documentation. The figures show the scaling with the per-walker random number states (#7980, now merged).

It also extends the nprocs plot of grass.benchmark to show the range of the runs for all metrics and to make higher resolution figures.

speedup for r.sim.water with different numbers of cells, shaded range of the 3 runs

Details (AI generated)

The script, figures and this description were made with Claude Code and reviewed by the author of the PR.

Benchmark script

raster/r.sim/r.sim.water/benchmark/benchmark_r_sim_water_nprocs.py runs r.sim.water with nprocs 1 to 16, 3 repeats each in shuffled order (grass.benchmark.benchmark_nprocs), on the elevation raster of the SECREF North Carolina dataset (secref_northcarolina_usa_epsg6542) at 4 m, 2 m and 1 m resolution of the same extent. Coarser than 1 m, the elevation is the average of the 1 m cells (r.resamp.stats); dx and dy come from r.slope.aspect -e; r.sim.water otherwise runs with its defaults and random_seed=1. It plots time, speedup and efficiency with grass.benchmark.nprocs_plot() at 1200 x 900 pixels. It saves the results as JSON and can redo the plots from that file without rerunning (python benchmark_r_sim_water_nprocs.py r_sim_water_benchmark.json). Resolutions, maximum threads and repeats are constants at the top.

Run in a temporary mapset of the dataset:

grass --tmp-mapset ~/grassdata/secref_northcarolina_usa_epsg6542 \
    --exec python benchmark_r_sim_water_nprocs.py

grass.benchmark

The subpackage is experimental; its documentation now says that it is changed together with the benchmarks which need more from it.

  • nprocs_plot() shaded the range of the runs in gray and only for time. It now shades it in each line's color for time, speedup and efficiency: for time between the fastest and the slowest run, for speedup over the ratios of every serial run to every run with the given nprocs, and for efficiency that divided by nprocs. With only a few runs per nprocs, the range of the runs is shown rather than a confidence interval.
  • New figsize and dpi parameters, used here for 1200 x 900 pixel figures.
  • Results without all_times are plotted again instead of failing (the docstring called it optional), the figure is closed after saving, and the default title for time, which was compared with "times", is set.
  • The range computation is a function of its own, nprocs_ranges(), tested in the new python/grass/benchmark/tests/grass_benchmark_plots_test.py (9 tests, under 1 s): the ranges for each metric, the serial runs at the smallest nprocs, results without all_times, and the size of the saved image. The existing gunittest suites of the subpackage pass.

Figures

The three figures in the documentation come from a run with main including #7980 and the 48-bit multiplication helper of #8013 (c26f5dd), 26 minutes on an Intel Xeon W-2295 (18 cores, 36 logical CPUs) shared with other work (1-minute load average 4 to 6). An earlier run with #7980 at 7c0d776, before it was merged, gave times within about 2 %.

The figures are 1200 x 900 pixels, twice the 600 pixel width of the documentation, so that they stay sharp on high density screens; the HTML page shows them at 600 x 450 and links to the full image. Quantized and compressed as the style guide asks (pngquant 128 colors instead of pngnq, then optipng -o5), 23 to 35 kB each.

resolution cells 1 thread best
4 m 33k 0.96 s 0.17 s (15 threads)
2 m 131k 12.4 s 1.50 s (15 threads)
1 m 525k 113.1 s 12.2 s (16 threads)

More threads than 16 were tried separately (3 repeats of 16, 28 and 32 threads in shuffled order): 28 threads took about as long as 16 (1 m: 13.6 s against 14.8 s; 4 m: 0.28 s against 0.19 s) and 32 threads clearly longer (1 m: 19.6 s; 4 m: 2.4 s). The CPU has 18 physical cores and the machine was shared (load average 8 before the run, 30 during it), so beyond 16 the figures would mostly show the load, not r.sim.water; the script stops at 16.

Before #7980, more threads made r.sim.water slower (4 m: 1.3 s with 1 thread, 2.7 s with 4), because all threads shared one random number generator.

Suggested squash commit message

r.sim.water: Add a benchmark for the number of threads
Measure r.sim.water with nprocs from 1 to 16 on the elevation of the
SECREF North Carolina dataset at 4 m, 2 m and 1 m resolution, using
grass.benchmark as the benchmarks of other tools do. The script plots
time, speedup and efficiency and saves the results as JSON, from which
it can redo the plots.

In grass.benchmark, nprocs_plot() shades the range of the runs in each
line's color for time, speedup and efficiency instead of a gray band for
time only (for speedup over the ratios of every serial run to every
parallel run), takes figsize and dpi, plots results without all_times,
and sets the default title for time. The range is computed by
nprocs_ranges(), which has pytest tests. The documentation of the
experimental subpackage says that it is changed together with the
benchmarks which need more from it.

Add the figures to the documentation in a Performance section, as the
other parallel tools have, at twice the 600 pixel width, shown at 600
pixels in the HTML page. They were made with the per-walker random
number states of OSGeo/grass#7980.

Written with the help of Claude Code.

Measure r.sim.water with nprocs from 1 to 16 on the elevation of the
SECREF North Carolina dataset at 4 m, 2 m and 1 m resolution, using
grass.benchmark as the benchmarks of other tools do. The script plots
time, speedup and efficiency and saves the results as JSON.

Written with the help of Claude Code.
Show the execution time, speedup and efficiency from the benchmark
script for 1 to 16 threads on the SECREF elevation at 4 m, 2 m and 1 m,
as the documentation of other parallel tools does. The figures were
made with the per-walker random number states of OSGeo#7980; with
the shared generator on main, more threads make r.sim.water slower, so
this belongs after that change.

Name the script's output files after the tool, as the figures are.

Written with the help of Claude Code.
Plot the mean of the repeated runs with a shaded 95% confidence interval
of the mean (Student's t) for time, speedup and efficiency, each band in
the color of its line. grass.benchmark.nprocs_plot() draws only a gray
min-max band and only for time, so the script plots on its own. The
interval of a speedup combines the relative intervals of the two mean
times; the speedup at one thread is 1 by definition.

Render at 1200 x 900 pixels, twice the 600 pixel width of the
documentation, so the figures stay sharp on high density screens; the
HTML page shows them at 600 x 450 and links to the full image. The PNGs
are quantized to 128 colors and compressed with optipng -o5.

The script can redo the plots from its JSON results without rerunning.

Written with the help of Claude Code.
With 3 runs per point, a confidence interval of the mean says more about
the t quantile than about the runs, so shade the range of the runs
instead: from the fastest to the slowest run for time, and over the
ratios of every serial run to every parallel run for speedup and
efficiency.

Drop the sentence that nprocs sets the number of threads, which the
parameter description already says.

Written with the help of Claude Code.
Follow the documentation of the other parallel tools: a Performance
subsection at the end of the notes, opening with the nprocs sentence
they share, and the processor named with its clock. Add that the time
step, and so the number of iterations, follows from the cell size and
the mean flow velocity, so the run time depends on the terrain, as the
r.horizon documentation notes for its algorithm.

Written with the help of Claude Code.
@github-actions github-actions Bot added raster Related to raster data processing Python Related code is in Python HTML Related code is in HTML module docs markdown Related to markdown, markdown files labels Oct 2, 2026
@wenzeslaus
wenzeslaus marked this pull request as ready for review October 7, 2026 04:06
wenzeslaus and others added 3 commits October 7, 2026 13:06
The figures were made with the head of OSGeo#7980 before it was
merged. Rerun the benchmark script on this branch merged with main, which
now contains OSGeo#7980 (per-walker random numbers) and OSGeo#8013 (the 48-bit
multiplication in a helper), with the same sizes, threads and repeats.
The times agree with the earlier run within about 2 % (1 m: 113.1 s
with 1 thread, 12.2 s with 16, speedup 9.3).

Written with the help of Claude Code.
nprocs_plot() shaded the range of the runs in gray and only for time.
Shade it in each line's color for time, speedup and efficiency: for
speedup, over the ratios of every serial run to every run with the given
nprocs, and for efficiency, that divided by nprocs. The range is computed
by nprocs_ranges(), which returns None for results without all_times, so
such results are plotted again instead of failing. Add figsize and dpi so
that figures for the documentation can be made at a higher resolution,
and close the figure after saving it. The default title for time was
never set because the metric was compared with "times".

The subpackage is experimental; say in its documentation that it is
changed together with the benchmarks which need more from it.

The r.sim.water benchmark now plots with nprocs_plot() instead of its own
matplotlib code; its figures are redrawn from the same results.

Written with the help of Claude Code.
@github-actions github-actions Bot added libraries tests Related to Test Suite labels Oct 7, 2026
@wenzeslaus
wenzeslaus merged commit becef3d into OSGeo:main Oct 8, 2026
25 of 26 checks passed
@wenzeslaus
wenzeslaus deleted the rsim-water-benchmark branch October 8, 2026 03:10
@github-actions github-actions Bot added this to the 8.6.0 milestone Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs HTML Related code is in HTML libraries markdown Related to markdown, markdown files module Python Related code is in Python raster Related to raster data processing tests Related to Test Suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants