Repository navigation
Description
Found by reading native/src/general.cpp. The effect has not been measured on a GPU.
-
device_trapezoid_plasmawritesplasma_exposure_and never reads it, and a trapezoid device run allocates the band column's guard and weight rows. Drop the write and keepplasma_exposure_'s allocation, since device statistics use the buffer. -
accept_updateis an update pass and three reductions on devices, against oneNormSumspass on the host. -
collect_trial(the Runge-Kutta and batch path),weighted_errorandlow_storage_steptake a fresh host mirror per trial, where the Adams trial now reads pinned memory.
Allocate the band buffers only for band plasma, use NormSums on devices, and keep persistent pinned mirrors as adams_norms_ does, each listed in memory_ledger(), minding that a pinned copy is asynchronous. Measure on an NVIDIA or AMD data-centre GPU; the device rows in benchmarks/history/ are the baseline. The roadmap's general entry gives the wider picture.
Done when
Each change's python -m benchmarks ab --device ratios on a data-centre GPU are in its pull request and show no loss, or it is dropped, and pytest -m gpu passes on a CUDA and a HIP build, which CI does not build.
Checked against commit 9d5d92f; the code links are pinned to it.
Also
Two more device costs, from reading and unmeasured, so both are suspicions:
- Reduction occupancy on short grids.
reduction_slotsgives a one-column problem at mostn / 1024teams. At 2^14 samples that is 16 teams of 1024 bins, at the default reduction block of 512 threads about 2 bins per thread, which suggests low GPU occupancy.benchmarks.mdlists other block sizes as measured and not adopted, but says nothing ofreduction_min_bins. Vary it against the block size at 2^12 to 2^16 samples on a data-centre GPU. - Default tiles.
Range2keeps Kokkos's default tiles on CUDA and HIP, where the host's tiling gave Exponential Adams performance: host Range2 tiling, host cost, fused trial #55 21 % on an AMD EPYC 7763 and 16 to 18 % on an Apple M5 Max (rdw, 16 threads). SEP 28 left device tile sizes as a separate item. The only device record found is the band column's tiled kernels on an NVIDIA A40, where 512-sample tiles gave 1.010 and 1.019; whether those areRange2kernels was not traced. Try otherRange2tiles on CUDA and HIP, or record that the default stands.
Added on 2026-11-01, checked against commit 8a9f764.
Activity
Take a device's accept pass in one reduction, through a pinned mirror
jtravs commentedon Nov 5, 2026
Where this stands after #520, which isn't merged:
low_storage_step's host mirror: one pinned scalar.accept_update: the update and its three norms in one pass, its partial sums in pinned memory.collect_trialandweighted_error's mirrors, on the Runge-Kutta and batch path: not changed.device_trapezoid_plasmastill writesplasma_exposure_without reading it, and a trapezoid device run still allocates the band column's rows.No figure isolates the mirror, whose saving should show mainly on small problems. Nothing here has been measured on the Runge-Kutta path, so that's the next thing to time before changing it.