More efficient mpi sorting

[spossann]

Addresses also #402

The basic new things are two new keywords pushes_eta and local_eval_onlyfor pyccel kernels; they reduce the calls to mpi_sort to when actually needed.

I now see a good strong scaling for usual VlasovAmpere and ColdPlasmaVlasov test cases.

Opus 5.5:

I've implemented the plan and it cuts mpi_sort_markers calls by 2× to 10× depending on the model, with no change in results.

What changed

pusher.py

• pushes_eta: bool is a new required keyword, so a new Pusher(...) call can't forget it.
  • With False, the pusher never sorts, and passing a mpi_sort value raises an assert.
  • With True, the markers are always sorted at the end: mpi_sort=None becomes "last".
• local_eval_only: bool = False: when True, sorting happens only after the last stage. It asserts there are no eval_kernels.
• Skipping redundant sorts: all sorts before a kernel now go through _sort_for_alpha(). It tracks the alpha the markers were last sorted for in _sorted_for.
  • On entry the markers count as sorted for any alpha, and that stays true until the kernel moves them.
  • A sort is skipped if the markers are already sorted for the requested alpha.
• put_particles_in_boxes still runs per stage, as you asked. Markers outside the local domain get the overflow box index, so this is safe with local_eval_only.

Call sites (all 17)

• local_eval_only=True (plus pushes_eta=True): PushEta and PushRandomDiffusion. I dropped their explicit mpi_sort="each".
• pushes_eta=True: PushEtaPC, PushDeterministicDiffusion, and both guiding-center propagators (implicit and explicit).
• pushes_eta=False: the 9 velocity/weight pushers. I checked that none of their kernels write markers[ip, 0:3].
• I added the flags to the calls in test_pushers.py and _bench_push_vxb_analytic.py.

Sort calls over 3 time steps (default params, same count on 2 and 4 ranks)

Model	Before	After
Vlasov	27	3
VlasovAmpereOneSpecies	30	3
LinearMHDVlasovCC	30	3
PressureLessSPH	24	3
GuidingCenter / DriftKineticElectrostaticAdiabatic (implicit, default)	24	12
same models, explicit RK4	48	24

Checks

• test_pushers.py: 102 passed on 1, 2 and 4 ranks. ruff format and ruff check are clean.
• For the models above I ran old code (via git stash) against new code on 1, 2 and 4 ranks with a fixed seed and compared the final markers by ID:
  • Bit-identical for Vlasov, PressureLessSPH and the guiding-center models, implicit and explicit.
  • VlasovAmpereOneSpecies and LinearMHDVlasovCC differ only by round-off, at most 6e-13. With fewer sorts the row order within each rank changes, so the order of the accumulation sums changes.

A separate bug I found while checking (not fixed)

With pseudo_random loading under MPI, marker IDs are duplicated at rank boundaries: 14 to 54 duplicates for Np=5000. _first_marker_id (base.py:1167) is computed from the estimated n_mks_load, but loading then resets it to the actual counts, so neighbouring ranks' ID ranges overlap. The default LoadingParameters() also has seed=None, so each rank draws a different random stream. This made the first old-vs-new comparison look badly broken until I excluded the duplicate IDs. The fix should be about one line; I can do it separately if you want.

Not done yet

• The five manual sorts in current_coupling_5d_gradb.py.
• I didn't add the debug assert from the plan. A strict "every marker is on its own rank" check would fire wrongly for markers lying exactly on a boundary.
• The comparison only covers the default configs. Nothing tested PushEtaPC or the diffusion propagators directly.
• I haven't measured wall-clock scaling. These test runs are too small for timings to mean anything.

I also saved two memory notes: one on the new pushes_eta / local_eval_only rules, one on the duplicate-ID issue.

[max-models]

Looks good! You have to fix the merge conflict from my changes before merging.

[spossann]

Fixed the merge conflicts in 71ab6c4

A question is whether we put self._sort_for_alpha into self._evaluate. Then we would have to move the line self._sorted_for = "any" before the init kernel evaluations, in order for sorting to be skipped. If we put it into the evaluate logic, one can be sure that the markers are sorted correctly for evaluation.

[max-models]

Fixed the merge conflicts in 71ab6c4

A question is whether we put self._sort_for_alpha into self._evaluate. Then we would have to move the line self._sorted_for = "any" before the init kernel evaluations, in order for sorting to be skipped. If we put it into the evaluate logic, one can be sure that the markers are sorted correctly for evaluation.

Yes, I think this is a good idea. If I understand it right, the benifit woud be that in both the init_kernels and eval_kernels loops will now go through that _sort_for_alpha check when they call _evaluate. Previously this was only done in the eval_kernels loop.