Repository navigation
r.param.scale: loosen curvature test tolerance across platforms - #7584
Merged
Merged
Conversation
The curvature methods (profc, planc, minic) produce values very close to zero. Each value comes out of a least-squares fit and a small matrix solve, and the last few digits depend on the compiler and CPU. So the same DEM matches to about seven significant figures across platforms, but not bit for bit. The test's absolute tolerance was 1e-9, which was tighter than that platform-to-platform variation. On the Linux CI runner the planc max at window size 9 came out about 2.6e-9 from the reference and failed, even though it still agreed to seven figures. Raising the floor to 5e-8 covers that variation with room to spare, and it is still far smaller than the shift a real bug would cause, so the test keeps catching real regressions.
github-actions
Bot
added
raster
krcoder123
changed the title
krcoder123
changed the title
echoix
approved these changes
Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
My previous PR #7534 was merged into main but ended up having a "(ubuntu-24.04, 3.13) (push)" error once it was pushed, so this PR is to just fix that one little issue.
The curvature numbers were tiny, very close to zero. The last digits came out a little different depending on the compiler and CPUs, so the values match to about 7 figures but not exactly. My old tolerance of (1e-9) was too tight so planc at size 9 failed on the Linux runner by a very small amount. I'm raising the floor to be 5e-8 so it handles the platform rounding properly and it's still far below the size change a real bug would have caused if there was one.