Skip to content

r.sim: Write time series maps at their output step times - #8003

Merged
wenzeslaus merged 7 commits into
OSGeo:mainfrom
wenzeslaus:simwe-time-series-schedule
Oct 3, 2026
Merged

wenzeslaus merged 7 commits into
OSGeo:mainfrom
wenzeslaus:simwe-time-series-schedule

Conversation

@wenzeslaus

@wenzeslaus wenzeslaus commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

With the -t flag, r.sim.water writes the series maps every so many iterations and names them by the simulated time rounded to minutes. The write times drift away from the output steps, so two writes can get the same name and the first map is overwritten, the map for the last step is often missing, and when the time step is longer than output_step no map is written at all while the tool succeeds.

This writes each output step at the iteration closest to its time and names and timestamps the maps by that time. The series always ends with the maps named by the duration. The depth and discharge values do not change and neither do the outputs without -t. It keeps the run summary (#7972) consistent.

Details (AI generated)
  • What was wrong: the tool wrote a series map when i % iterout == 0 with iterout = (int)(output_step / time step) and named it by (int)(i * time step / 60 + 0.5) minutes. The truncation makes the write times drift from j * output_step, which leads to duplicate names (the earlier map is overwritten), skipped or odd minutes with an even output_step, a missing last step, and no maps at all when iterout truncates to 0 (for example mintimestep=17.5, time step 70 s, output_step=1), all with exit code 0.
  • Change: output step j is written at the iteration whose simulated time is closest to j * output_step and is named and timestamped by j * output_step minutes with the existing name format. A map holds the state at most half a time step before or after the time in its name. output_data() gets the name time as a separate argument next to the simulated time. Setup.iterout is removed. The OpenMP region is untouched.
  • Why the closest iteration: the time step comes from the cell size and the mean velocity (or mintimestep), so iterations usually do not fall on the output times. Writing a step at the first iteration at or after its time would put every map up to one time step late. With a time step of 59 s and 1-minute steps, the map named 01 would hold the state at 118 s; with the closest iteration it holds the state at 59 s. Shortening the time step to make the times fit would change the results of -t runs compared to runs without -t.
  • Semantics decisions:
    • (a) Time step longer than output_step: one iteration is the closest one to several step times. It writes once, named by the step closest to it, and the tool warns once at setup that the time step is longer than output_step and that output steps may be skipped. The warning is not printed for output_step of zero or less, where only the final maps are written. With 1-minute steps, the series is 01, 02, 04 to 10 for a 70 s time step, 02, 04, 06, 08, 10 for 120 s and 03, 06, 09, 10 for 180 s. A step past the duration is never written, which matters when the duration is not a multiple of output_step.
    • (b) Final map: the series always ends with maps named by duration, written at the end of the run, when the duration is not a multiple of output_step (duration 10, step 3 gives 03, 06, 09, 10) and when all walkers left the domain early (the series then ends with the duration map right after the last written step). When the last iteration is the closest one to the duration, the duration maps come from the same rule and nothing extra is written. The duration map can have the same state as the step before it when both come from the last iteration.
    • (c) Outputs without -t: unchanged, including their timestamp, which still uses the simulated time rounded to minutes. The mismatch between the write time (miter * dt) and the timestamp ((miter + 1) * dt) was already fixed on main by the run summary (r.sim: Add -p flag with format=json run summary and normalize history #7972), which computes both from the completed iterations.
    • (d) Walkers and observation points: walker vector maps (walkers_output) used the same iterout test and now follow the new schedule (walkers_01, ...). Observation points are logged every iteration and did not use iterout, so they are unchanged.
    • (e) Run summary: there is no output interval field in the summary. outputs[].simulated_time stays the time the maps were written and outputs[].timestamp is now the time the maps are named by, so in a series the timestamp is a multiple of output_step and the simulated time is within half a time step of it (the final duration map can be up to one time step earlier). The docs of both fields and of outputs are updated. The example summary in the manual needs no change: with its time step of 1.063 s, the iteration closest to 10 minutes is iteration 564 at 599.6 s, which is the one the example already shows.
  • r.sim.sediment: it has no -t flag (settings.ts is always false) and its output_step only affects a message, so its outputs go through the unchanged non-series path. Its testsuite passes and its manual needs no change.
  • Verification:
    • Outputs without -t on the nc_spm test region (10 m, man_value=0.1, nprocs=1): depth and discharge fcell, cell, colr and cell_misc files are byte-identical to main, the timestamps are the same and the walker vector points are identical.
    • With the fix, the final series map is byte-identical to the non-series map of the same run, so series values are unchanged and only when and under which name maps are written changes.
    • On the nc_spm test region with default settings and man_value=0.1, main writes .09 twice and no .10 with output_step=1. With the fix it writes .01 to .10, each once.
    • test_complex in the gunittest suite writes a 5-minute series for 15 minutes. Its .15 reference did not need to change: on main, the 15-minute map was written at the last iteration (1731, at 899.6 s), and with the fix it is the final duration map written at the same iteration (the maps are byte-identical). The 5 and 10 minute maps can come from a neighboring iteration, but the suite does not compare them.
  • Tests run: the r.sim.water pytest suite (61 passed, 22 s) and the r.sim.sediment pytest suite (6 passed), after merging main, the r.sim.water gunittest suite with nc_spm (5 tests, 1 skipped as before, OK) and the r.sim.sediment gunittest suite (1 test, OK). All five new tests fail on main and pass with the fix.

Reproducer: 500 m x 500 m region at 10 m, elevation row() * 0.1, man_value=0.1, duration=10, output_step=1, random_seed=1, nprocs=1, time step 10.17 s.

write main: map main: simulated time this PR: map this PR: simulated time
1 dep.01 50.9 s dep.01 61.0 s
2 dep.02 101.7 s dep.02 122.1 s
3 dep.03 152.6 s dep.03 183.1 s
4 dep.03 (overwrites) 203.5 s dep.04 244.1 s
5 dep.04 254.3 s dep.05 295.0 s
6 dep.05 305.2 s dep.06 356.0 s
7 dep.06 356.0 s dep.07 417.1 s
8 dep.07 406.9 s dep.08 478.1 s
9 dep.08 457.8 s dep.09 539.2 s
10 dep.08 (overwrites) 508.6 s dep.10 (end of run) 590.0 s
11 dep.09 559.5 s

Tests

New file raster/r.sim/r.sim.water/tests/r_sim_water_time_series_test.py with five tests, one of them with three cases. They run on the reproducer slope narrowed to 5 x 50 cells (a module fixture), where the time step is 10.69 s and main writes dep.04 twice.

Time: the file takes 1.1 s on its own. The r.sim.water pytest suite goes from 21.0 s (54 tests) to 22.1 s (61 tests), an increase of about 5%. All tests in the file fail with the old schedule.

  • test_every_output_step_written_once (0.2 s):
    • Why: it is the reproducer case, a time step which does not divide 60 s.
    • Checks: the depth maps in the summary and in the mapset are exactly dep.01 to dep.10, the summary and raster timestamps are the step minutes, each intermediate map is written within half a time step of the time in its name, the last one at the end of the run, and the walker maps are exactly walkers_01 to walkers_10.
    • Catches: duplicate names and overwritten maps, drifting names, the missing last step and walker maps on a different schedule.
  • test_final_step_named_by_duration (0.1 s): output_step=3 must give dep.03, dep.06, dep.09 and dep.10. Catches a series without the duration map when the duration is not a multiple of the step.
  • test_time_step_longer_than_output_step (three cases, 0.35 s): time steps of 70 s, 120 s and 180 s with 1-minute steps. With 70 s, the iteration ending at 210 s is equally close to 3 and 4 minutes and writes the later one (series 01, 02, 04 to 10). With 120 s and 180 s, the iterations fall exactly on every second and third step (series 02, 04, 06, 08, 10 and 03, 06, 09, 10). Each case also checks the warning. Catches the old no-maps-at-all case, several maps from one iteration, and maps named by a later step than the one the iteration falls on.
  • test_no_output_step_past_duration (0.1 s): with a time step of 160 s, a duration of 11 minutes and 3-minute steps, the last iteration ends at 640 s and the step closest to it is 12 minutes. Checks the series 03, 06, 09, 11. Catches maps named past the duration; it fails without the check against the duration.
  • test_final_step_after_early_stop (under 0.1 s): with man_value=0.002, all walkers leave the slope before the first minute, so the series must still end with dep.02 timestamped 2 minutes. On main, no map is written.

Changed existing tests:

  • test_json_summary_time_series now checks the timestamps against the step minutes instead of the rounded simulated time, since that is the documented meaning now. It passes on main and with the fix.
  • test_duration_affects_time_series_progression keeps its assertions. Its docstring and comment said the final 20-minute map depends on walkers surviving, which is no longer true, and are updated.

Squash-merge commit message

Title:

r.sim: Write time series maps at their output step times

Body:

With -t, r.sim.water wrote the series maps every (int)(output_step /
time step) iterations and named them by the simulated time rounded to
whole minutes. The truncated interval made the write times drift away
from the output steps, so two writes could round to the same minute and
the first map was overwritten, names could skip a minute or land on odd
minutes with an even output_step, and the last step was often missing.
When the time step was longer than output_step, the interval was zero
and no map was written at all while the tool still succeeded.

Write output step j at the iteration whose simulated time is closest to
j times output_step, and name and timestamp the maps by that time. The
time step comes from the cell size and the mean velocity, so iterations
usually do not fall on the output times. When one iteration is the
closest one to several steps, write only the step closest to it and
warn that the time step is longer than output_step. Do not write steps
past the duration. Always end the series with maps named by the
duration which hold the state at the end of the run, also when the
duration is not a multiple of output_step or when the run stopped
early. Walker vector maps follow the same schedule.

For users, each output step now appears once, named and timestamped by
a multiple of output_step, and the maps named by the duration are always
present. A series map holds the state at most half a time step before
or after the time in its name instead of up to several minutes before
it. The depth and discharge values are unchanged and so are the outputs
without -t. In the run summary, simulated_time of an output is the time
the maps were written and timestamp the time they are named by. New
tests check the names, timestamps and write times of the series, a
duration which is not a multiple of output_step, time steps longer
than output_step including its multiples, a step past the duration, and
an early stop.

Written with the help of Claude Code.

The analysis, code and tests were done with Claude Code (Opus and Fable) and reviewed by me.

With -t, r.sim.water wrote the series maps every (int)(output_step /
time step) iterations and named them by the simulated time rounded to
whole minutes. The truncated interval made the write times drift away
from the output steps, so two writes could round to the same minute and
the first map was overwritten, names could skip a minute or land on odd
minutes with an even output_step, and the last step was often missing.
When the time step was longer than output_step, the interval was zero
and no map was written at all while the tool still succeeded.

Write output step j at the first iteration whose simulated time reaches
j times output_step, and name and timestamp the maps by that time. When
one iteration reaches several steps, write only the latest one and warn
that the time step is longer than output_step. Always end the series
with maps named by the duration which hold the state at the end of the
run, also when the duration is not a multiple of output_step or when
the run stopped early. Walker vector maps follow the same schedule.

For users, each output step now appears once, named and timestamped by
a multiple of output_step, and the maps named by the duration are always
present. A series map holds the state up to one time step after its
name instead of up to several minutes before it. The depth and
discharge values are unchanged and so are the outputs without -t. In
the run summary, simulated_time of an output is the time the maps were
written and timestamp the time they are named by.

Written with the help of Claude Code.
Check on a slope where the output step does not fall on an iteration
that each output step is written once under its name and timestamp,
that walker maps follow the same schedule, and that the series ends with
the maps named by the duration when the duration is not a multiple of
output_step, when the time step is longer than output_step, and when
all walkers left the domain early.

Check the timestamps in the summary of a time series run against the
output steps instead of the rounded simulated time, and update the
description of the series written in the duration test.

Written with the help of Claude Code.
The series maps of an output step were written at the first iteration
which reached the time of the step, so a map held the state up to one
time step after the time in its name. With a time step close to
output_step, that was almost a whole output step: with a time step of
59 s and output_step of 1 minute, the map named 01 held the state at
118 s.

Write the maps of an output step at the iteration closest to its time
instead, so that a map holds the state at most half a time step before
or after the time in its name; in the example above, at 59 s. An
iteration is the closest one to the steps up to half a time step ahead
of it. That range is limited to the duration, because with a time step
at least twice as long as output_step it would otherwise include steps
past the duration. As before, when one iteration is the closest one to
several steps, only the latest step is written, and the series ends
with the maps named by the duration.

Describe the rule in the manual with the time in the map name instead
of the name, and update the example summary, where the first output is
now written one iteration earlier. Check the new rule in the tests and
add a test with a time step much longer than output_step, which fails
without the limit.

Written with the help of Claude Code.
Resolve the conflict in the run summary example of the r.sim.water
manual by taking the example from main in both the Markdown and the
HTML file. The example needs no change for the time series schedule:
with its time step of 1.063 s, the iteration closest to 10 minutes is
iteration 564 at 599.6 s, which is the one the example already shows.
The warning about a time step longer than output_step was also printed
for output_step of zero or less, as "longer than output_step of 0 s",
although no series step is written at all in that case, only the final
maps. Print it only for a positive output_step, as the series writes
themselves are limited.

Say that output steps may be skipped, because in a short run every step
can still be written, for example with a time step of 61 s, output_step
of 1 minute and duration of 2 minutes.

Written with the help of Claude Code.
Run the time series tests on a slope 5 cells wide instead of 50. The
run which writes ten walker vector maps took 0.37 s on the wide slope
and takes 0.10 s on the narrow one, because there are ten times fewer
walkers to write. The time step is now about 10.7 s, which still does
not divide the output step; the old schedule wrote dep.04 twice there.

Use a project without a coordinate reference system, which is created
without running a tool, and make the walkers leave early on the same
slope with a very low Manning's n instead of setting up the shared
project for one test.

The file now takes 0.9 s instead of 1.4 s. All five tests still fail
with the old schedule.

Written with the help of Claude Code.
Comment thread raster/r.sim/simlib/hydro.c Outdated
With a time step longer than output_step, an iteration is the closest
one to several output steps, and it wrote the latest of them. When the
time step was a multiple of output_step, the iterations fell exactly on
output steps, but the maps were named by later ones: with a time step
of 120 s and output_step of 1 minute, the series was 03, 05, 07, 09, 10
with the state at 120 s in the map named 03.

Write the output step closest to the iteration instead. The series in
the example is now 02, 04, 06, 08, 10, and with a time step of 180 s it
is 03, 06, 09, 10. A step more than half a time step ahead of the
iteration is left to the next iteration, so with a time step shorter
than output_step the maps are the same as before. A map now holds the
state at most half of the shorter of the time step and output_step
before or after the time in its name. A step past the duration is not
written, which can only happen when the duration is not a multiple of
output_step.

Test time steps of two and three times output_step, and replace the
test of a step past the duration, which the 290 s time step no longer
reaches, with a duration of 11 minutes and output_step of 3 minutes.

The problem was reported in the review of the pull request. Written
with the help of Claude Code.
@github-actions github-actions Bot added raster Related to raster data processing Python Related code is in Python C Related code is in C HTML Related code is in HTML module docs markdown Related to markdown, markdown files tests Related to Test Suite labels Oct 2, 2026
@wenzeslaus
wenzeslaus disabled auto-merge October 3, 2026 03:08
@wenzeslaus
wenzeslaus enabled auto-merge (squash) October 3, 2026 03:08
@wenzeslaus
wenzeslaus merged commit 8b60e9f into OSGeo:main Oct 3, 2026
27 checks passed
@wenzeslaus
wenzeslaus deleted the simwe-time-series-schedule branch October 3, 2026 03:08
@github-actions github-actions Bot added this to the 8.6.0 milestone Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C Related code is in C docs HTML Related code is in HTML markdown Related to markdown, markdown files module Python Related code is in Python raster Related to raster data processing tests Related to Test Suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants