Skip to content

The on_axis_* statistics copy the whole state to the host on device builds #359

Closed

Description

@jtravs

What is missing

The on_axis_* statistics (SEP 19) read one row of the field at the axis. On a device build the native backend copies the whole state to the host and scans the row there, once per saved plane that asks for them. SEP 19 left the device form out on purpose, as "needed only to avoid the state copy on a GPU", and deferred it to GPU qualification.

Why it matters

Possibly not much. On a large Cartesian run the copy is the whole state at each such plane. Its cost has not been measured on a GPU, so that is the first step. Read, not run.

To do

  • Time a saved plane with and without on_axis_* on a data-centre GPU, on a Cartesian run of at least 384 x 384.
  • If the copy matters, add a one-row device scan through the existing plasma scan, keeping tests/backends/test_on_axis_statistics.py's parity against numpy_ref. If not, close this with the number.

Done when

The cost is recorded with device, grid and date, and either the statistics no longer copy the state on a device, with parity holding on a CUDA and a HIP build, or the issue closes because the copy is under 1 % of the run's time at one plane per 100 steps.


Checked against commit 8a9f764. Takes over the roadmap's entry "The on-axis statistics on device builds".

Activity

  1. added
    performanceSlower or larger than it should be, with no wrong result
    area: gpuCUDA and HIP builds: qualification and performance
    on Nov 1, 2026
  2. jtravs commented on Nov 5, 2026

    @jtravs
    ContributorAuthor

    Measured and fixed in #520. on_axis_intensity and the on_axis_* row now read one row on the device, so no state crosses.

    768 x 768 x 600, 5.27 GiB Whole state to the host on_axis_intensity now
    NVIDIA H200 162 ms at best, with #520's crossing; 2,929 ms before it 11.8 ms
    AMD Instinct MI325X 106 ms; 506 ms before 14.1 ms

    The MI325X's first read of a run took 184 ms. I'd guess that's building its plan, but I didn't check. #520 will close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: gpuCUDA and HIP builds: qualification and performanceperformanceSlower or larger than it should be, with no wrong resultpriority: lowWorth doing, no hurry

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions