perf: compute forecast quantiles block-wise to bound memory

[EnvDroneSense]

Changes proposed in this PR:

• Add climada.util.forecast.sparse_quantile_axis0, which takes a quantile along axis 0 of a sparse matrix by densifying a block of columns at a time rather than the whole matrix. The block size is derived from a max_memory_mb budget.
• Call it from HazardForecast._quantile (intensity and fraction) and ImpactForecast._quantile (imp_mat), replacing the .toarray() on the full matrix.

This PR fixes #1203

Peak memory no longer scales with the number of centroids. Measured on a full DWD ICON-EU-EPS run, 2560 events x 164,984 cells, float32, with [icon_full.py](https://github.com/user-attachments/files/31507435/icon_full.py) :

old    8110.9 ms (best of 3)   peak   3382.3 MB
new    2375.7 ms (best of 3)   peak    280.5 MB

The script asserts the two paths return equal arrays on every run.

One constraint worth flagging: implicit zeros have to take part in the quantile. A column of [0, 0, 0, -1, 9] has median 0.0, not the 4.0 you get from its two stored values. That is why the helper densifies blocks instead of reducing over matrix.data, which would be faster but silently wrong.

[peanutfun]

@EnvDroneSense Thank you for raising a PR! I am happy about your contribution and your code generally looks good, but your PR description is lengthy, has many unsupported claims, and overall looks AI-generated.

In your "Results" section you claim a speedup and reduction in memory usage with the new code. How did you conduct these measurements? What code did you execute? Later, you claim that setting overwrite_input=True is faster. I see that it is reasonable to save memory, but the entire addition of the new function already serves that purpose. Did you measure the additional speedup as well? Also, the entire "Speedup" section never shows speedup but discusses saving memory. Finally, you claim that results are identical, but the unit tests use assert_allclose instead of assert_array_equal.

The dim is not None branch of both _quantile methods delegates to reduce_unique_selection and is not changed here. It was not measured, and if it densifies by another route then #1203 is only partly addressed by this PR. Happy to extend the scope if you would prefer that handled in one go.

This is not necessary. The "dim is not None branch" calls reduce_unique_selection, which performs a subselection and then delegates back to the calling attribute, in this case _quantile, so there is no extension needed to apply your changes to that particular branch.

I would be happy if you could answer my questions regarding your benchmarks. Please also review my code comments, which I would like to see resolved before approving a merge.

Again, thank you for your contribution! ☺️

[EnvDroneSense]

@peanutfun sorry for the messy PR. (I have rewritten it)

On the speedup: part of that was me misinterpreting my own benchmark results, so I have redone them from scratch. This time I measured against real data rather than synthetic icon_full.py, and the results look quite good:

run 2026-08-27 00:00:00
2560 events x 164,984 cells, density 0.055, dense would be 1,689.4 MB

old    8110.9 ms (best of 3)   peak   3382.3 MB
new    2375.7 ms (best of 3)   peak    280.5 MB

On overwrite_input=True: I measured it separately this time. At the default 8 MB budget it makes no difference to peak memory (280.5 MB either way) but is about 2x faster (2078 vs 4985 ms), because without it every block gets copied before sorting. So it is a speed optimization rather than a memory one. icon_overwrite.py

2560 events x 164,984 cells, budget 8 MB = 781 columns over 212 blocks, best of 5

overwrite_input=True      2078 ms   peak  280.5 MB
overwrite_input=False     4985 ms   peak  280.5 MB
identical

I have also implemented the changes you recommended. One thing I am unsure about: the default for max_memory_mb is currently an arbitrary 8 MB. Is that acceptable, and if not, how would you suggest picking a better one? Should it go into climada.conf instead?

Thank you for your review, I am trying to learn more about code optimization.

[peanutfun]

@EnvDroneSense Excellent work! Using your benchmark code, I could replicate these numbers on my computer. The tests also run through on my end. A final request: Please update the test test_quantile_uses_block_wise_helper with a mock for sparse_quantile_axis0, ensuring that it is called with the correct matrix object, and add a similar test for the HazardForecast class. Once this is done, we are ready to merge!

[EnvDroneSense]

@peanutfun Great. Both tests mock sparse_quantile_axis0 and assert it receives the matrix object itself; the HazardForecast one covers both calls.

[peanutfun]

Excellent work! Thanks again ☺️