Frontier: move to CCE 21.0.2 / ROCm 7.0.2 - #1694
Conversation
Two independent defects in GPU_ROUTINE (MFlowCode#1682): (1) the !DIR$ line sat in the #else arm under _CRAYFTN, so it was selected only when neither MFC_OpenACC nor MFC_OpenMP was defined -- i.e. never on a Cray GPU build, which is where every one of the 51 call sites lives; (2) !DIR$ NOINLINE is not a Cray directive. CCE 19 rejects it with ftn-790 'Unknown or unsupported compiler directive', the same diagnostic it gives gibberish. The Cray spelling is INLINENEVER. cray_inline was unaffected -- INLINEALWAYS is correct. Emit the !DIR$ line alongside the offload directive instead of instead of it, and use INLINENEVER. Also switch the bare '#if MFC_OpenACC' to '#if defined(...)' to match the rest of the file.
…rocm module Three things blocked MFC from building with anything newer than cce/19.0.0 on Frontier, all on the MFC side rather than the compiler: 1. rocprofiler-compute/3.0.0 in f-gpu triggers a MODULEPATH change that re-resolves PrgEnv. It only has a build for the rocm it was made against (6.3.1); with any other rocm the whole module load silently falls back to the system default cce/18.0.1, with no error. That single module is what pinned the toolchain. It is only needed for the optional --rcu profiling flag, so it does not belong in the default GPU set. 2. hipfort was pinned to a literal GIT_TAG rocm-6.3.1, independent of the loaded rocm module, so bumping rocm silently built mismatched Fortran bindings -- which surfaces as versioned-symbol link errors, not anything legible. It now derives from ROCM_PATH and fails loudly if no matching tag exists. 3. cpe/26.03 alone selects cce/21.0.0, not the newest 21.0.2, so the cce version is now pinned explicitly. Net: cpe/26.03 + cce/21.0.2 + rocm/7.2.0, with hipfort resolving to rocm-7.2.0. Stopped at rocm/7.2.0 rather than 7.13.0 because hipfort has no tag beyond rocm-7.2.4 and 7.2.0-7.2.4 are the same commit.
The five procedures contained in s_mom_inv (s_coeff_selector, s_chyqmom, s_hyqmom, f_quad, f_quad2D) reference nothing from their host scope: every apparently-shared name is shadowed by the procedure's own dummy or local. Containment was organizational only. Calling a sibling internal procedure from device code needs the host frame chain, which CCE 21 rejects at runtime: OpenMP Execution Error: m_qbmm.fpp:984 - unsupported access to Fortran host-associated variable in GPU internal procedure (cray_inline=True inlines s_hyqmom into s_chyqmom, so the abort surfaces at the call site.) Module scope removes the construct entirely. Change is move-only: apart from the removed 'contains', all 922 lines are byte-identical after dedent. Verified on Frontier, CCE 21.0.2, 6/6: tests 0501B3DA, 6784C02E, 83291843 pass under both --gpu mp and --gpu acc.
CCE 21 mishandles defaultmap in two ways, both silent:
* any defaultmap(<category>:...) clause makes an already-resident array read
as ZERO inside the target region, so device code sees an empty array while
the correct bytes sit in the correct attached buffer;
* defaultmap(firstprivate:scalar) overrides an explicit map on a scalar, so an
atomic-updated counter reads back its initial value.
In MFC the first zeroes ib_markers on device, the ghost-point count returns 0,
the allocation is zero-length, and every immersed-boundary case faults.
Emit nothing on the CCE branch, as is already done for AMD_COMPILER_ID.
Frontier, CCE 21.0.2 / ROCm 7.2.0, full 627-test suite, --gpu mp:
before 566 passed / 61 failed, 60 exit-134
after 622 passed / 5 failed, 0 exit-134
Root-caused in a parallel investigation with a 60-line reproducer and a
per-clause bisect; this commit is the change that reproduces at suite scale.
Reproducer: cce21-bugs/11-defaultmap-zeroes-resident-arrays.
-mattr=-mai-insts avoids the AMDGPU Rewrite AGPR-Copy-MFMA
assertion that blocks the device link
-disable-promote-alloca-to-vector the pass silently discards a
dynamically-indexed store into a 1-based
private array
The quoting is load-bearing: modules.sh does eval "export $line", so an
unquoted multi-value assignment exports only the first -plugin-opt and discards
the rest to stderr, which mfc.sh load output routinely swallows. You then get the
link workaround and silently lose the correctness one.
The source patch for the promote-alloca sites is necessary but not sufficient --
unpatched sites remain in m_ibm, m_riemann_solver_hll, m_qbmm and m_collisions --
so both this flag and the source change are required.
With this line, the five previously-failing IBM Rectangle tests pass on both
backends, taking the suite to 627/627.
Reproducers: cce21-bugs/01-agpr-pass-assert, 02-promote-alloca-dropped-store.
s_ibm_correct_state was called without pb_in/mv_in on the non-QBMM path while the routine dereferences them unconditionally -- m_ibm.fpp contains no present() guards. On CCE 21 the absent dummy presents a null dope vector to the device data environment and every IBM test segfaults. pb_ts(1)%sf is allocated and device-mapped in both branches of m_time_steppers (full size under QBMM, beg:beg+1 otherwise), so pass it always and delete the branch. Safe because the writes at m_ibm.fpp:437 are already guarded by if (qbmm) / if (.not. polytropic), so the small allocation is never indexed. NOTE: this is only the caller half of the fix in MFlowCode#1691. The dummies are still declared optional in m_ibm.fpp; that is harmless now that no call site omits them, but the declaration should be cleaned up before upstreaming, and s_interpolate_image_point (m_ibm.fpp:807, nine optional dummies, call sites passing different subsets) has not been audited at all. Also drops the thread_limit(1024)/vector_length(1024) clauses added earlier as an AGPR workaround. -mattr=-mai-insts supersedes them and is measurably faster: geometry-512 measured 1.01% slower across all six bench cases, and 1,1024 spills 238 scratch ops versus zero at 1,512.
|
Claude Code Review Head SHA: 7b52299 Files changed:
Findings:
|
|
Out of draft — both backends now have a clean full suite on this branch's exact configuration.
Same job ( On the review-bot finding ( The correction worth stating plainly: I had assumed this was a latent correctness bug, on $ gfortran -DMFC_OpenMP -o t t.F90 && ./t # BARE-TAKEN
$ ftn -DMFC_OpenMP -o t t.F90 && ./t # BARE-TAKENSo the bare form works, which is also why the pre-existing bare Two adjacent lines in the same conditional chain written in two different styles is still worth |
There was a problem hiding this comment.
Pull request overview
Updates the Frontier (OLCF) toolchain configuration to CCE 21.0.2 + ROCm 7.2.0 and applies several targeted source/toolchain fixes/workarounds needed for correctness and stability on that stack (especially GPU offload with OpenMP/OpenACC).
Changes:
- Update Frontier module set and inject Cray linker workarounds for ROCm/LLVM issues.
- Align hipfort’s ExternalProject tag with the loaded ROCm version to avoid mismatched bindings.
- Apply GPU/offload portability and correctness fixes in simulation/common sources (IBM call-site fix, Riemann index workaround, QBMM internal-procedure hoist, macro directive fixes, chemistry scratch workaround).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/modules | Bumps Frontier modules to cpe/26.03 + cce/21.0.2 + rocm/7.2.0; removes rocprofiler-compute from default GPU set; adds CRAY_CCE_LLD_ARGS plugin opts. |
| toolchain/dependencies/CMakeLists.txt | Derives hipfort tag from ROCM_PATH (fallback to known-good) to keep bindings in sync with loaded ROCm. |
| src/simulation/m_time_steppers.fpp | Always passes pb_ts/mv_ts into s_ibm_correct_state to avoid absent-optional UB on non-QBMM paths. |
| src/simulation/m_riemann_state.fpp | Rewrites runtime-indexed 1-based array increment into explicit constant-index paths to avoid CCE/LLVM dropped-store defect. |
| src/simulation/m_riemann_solver_lf.fpp | Same constant-index rewrite for LF solver path. |
| src/simulation/m_qbmm.fpp | Hoists internal procedures from s_mom_inv to module scope for device-call portability. |
| src/simulation/m_bubbles_EL.fpp | Minor change to a GPU_PARALLEL_LOOP invocation argument list. |
| src/common/m_variables_conversion.fpp | Pads Y_rs with lower bound 0 and slices explicitly to avoid a CCE private-frame/flat-pointer hazard. |
| src/common/include/parallel_macros.fpp | Fixes Cray directive spelling and emission ordering for Cray GPU builds. |
| src/common/include/omp_macros.fpp | Removes problematic defaultmap emission on CCE to avoid zeroed resident arrays / scalar-map override behavior. |
| & term1_fac, myR_m, mygamma_m, myPb, myMass_n, myMass_v, myR, myV, myBeta_c, myBeta_t, myR0, myPbdot, & | ||
| & myMvdot, myPinf, aux1, aux2, myCson, myRho, gamma, pi_inf, qv, dmalf, dmntait, dmBtait, & | ||
| & dm_bub_adv_src, dm_divu, adap_dt_stop, myPos, myVel]', copy='[adap_dt_stop_sum]',copyin='[stage]') | ||
| & dm_bub_adv_src, dm_divu, adap_dt_stop, myPos, myVel]', copy='[adap_dt_stop_sum]',copyin='[stage]',) |
| $:cray_directive | ||
| #elif defined(MFC_OpenACC) | ||
| $:acc_directive | ||
| #elif MFC_OpenMP |
| !> Lower bound 0 is a deliberate, unused pad element: it keeps Y_rs(1) off private frame offset 0. Cray ftn builds a flat | ||
| !! pointer to a private object as zext(offset) with no aperture, so the flat->private cast back (select(p == null, ~0, trunc | ||
| !! p)) turns offset 0 into 0xFFFFFFFF and the scratch store lands 4 GiB out of bounds. Faults every chemistry case on CCE | ||
| !! 21/gfx90a. Do not tighten to 1:num_species without re-checking that the device image is free of `s_cselect_b32 sN, 0, | ||
| !! -1`. |
Correction: withdrawing the "5.8% faster" claimThe benchmark pass of job Two separate problems, and the second is the one that matters: 1. I published a table I did not produce. The CCE 19 vs CCE 21 table ( 2. My own data points the other way for the configuration this PR actually ships.
Per case: The timeline is what makes this unambiguous: Caveats in the other direction, so this is not overstated: single runs with no repeats, and two Net: the shipped configuration has not been shown faster than CCE 19, and the single What this does not changeThe reason to take this PR is that CCE 19 miscompiles Follow-up worth doing before anyone merges this for speed
Apologies for the churn on a PR I had just marked ready. |
|
I opened #1695 for the same port from a second worktree before seeing this — closed as a duplicate. This one is more complete: it carries the QBMM hoist and the hipfort-tag fix, neither of which was in mine. Independent confirmation of the headline result, on a separate tree with the same fix set: 627/627 on both backends, and the Your withdrawal of the performance claim applies to my number too, and for the same reason. I measured 5.8% faster (CCE 19 328.27 s vs CCE 21 309.10 s, six cases) — but with Two things for this PR: 1. 2. - ... copy='[adap_dt_stop_sum]',copyin='[stage]')
+ ... copy='[adap_dt_stop_sum]',copyin='[stage]',)A trailing comma before the closing paren, with no other change to the line and no other change to the file. Fypp's Python-like call syntax will likely tolerate it, so this is cosmetic rather than broken — but it is the only hunk in the PR that does not correspond to a described fix, and a reviewer will stop on it. Worth dropping unless it is load-bearing for something I have missed. On the open performance question — I have a CCE 19 baseline YAML with the full fix set applied that may save you a run, at One measurement that may narrow the flag question: |
Follow-up: the
|
| arm | pre-flag → shipped | total |
|---|---|---|
acc (7 cases) |
106.25 → 112.80 | +6.2% |
mp (6 common cases) |
81.89 → 82.20 | +0.4% |
Per case on mp: hll −10.6%, viscous +13.3%, hllc −0.05%, lf −0.5%,
hypo_hll +0.25%, igr +0.5%.
A ±13% spread in both directions across cases whose work is deterministic is the signature of
run-to-run noise, not of an optimisation being disabled. The same pattern is visible in the
acc arm in hindsight (hllc +17.2% and lf +16.5% against ibm −4.7% and igr −4.8%). At
this problem size and with single runs, per-case noise looks to be on the order of 10%.
So the honest position is that the performance effect is unresolved in either direction.
Not +5.8%, and not −6.2% either. I replaced one under-supported number with another, and the
second arm of the very same job is what shows it. Both figures rest on single runs; neither
survives the noise floor those runs reveal.
I have updated the description again to say that rather than to assert a direction.
What is actually established, and is not in question: 627/627 on both backends on the
shipped configuration, and CCE 19's CONTIGUOUS miscompile being absent in CCE 21. The case for
the move is correctness. Performance needs a proper study — repeats, one variable at a time, at
a size where the noise floor is below the effect — and that study has not been run.
One incidental thing the comparison did confirm: bench_cce21_mp.yaml has 6 cases and no ibm,
while bench_cce21_acc.yaml has 7 including it. That matches the defaultmap defect being
OpenMP-only — ibm failed on mp and passed on acc at that time, and now runs on both.
Merge order, and a conflict with #1679 that neither PR mentionsSuggested order:
|
CPU-only gap partially closedI listed "no CPU-only build or suite was run" as the largest gap, on the grounds that
Build directory Still open: the full CPU suite (only a 10% sample was run), and any non-Cray compiler. Of the I have updated the gap list in the description accordingly. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1694 +/- ##
==========================================
+ Coverage 61.04% 61.14% +0.09%
==========================================
Files 83 83
Lines 20978 21054 +76
Branches 3099 3110 +11
==========================================
+ Hits 12807 12874 +67
+ Misses 6126 6125 -1
- Partials 2045 2055 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Case-opt performance measured: 3 of 4 cases fine,
|
| case | C19 copt | C21 copt | delta | C19 copt gain | C21 copt gain |
|---|---|---|---|---|---|
viscous_weno5_sgb_acoustic |
4.20 | 4.34 | +3.5% | 1.59× | 1.56× |
hypo_hll |
1.80 | 1.87 | +3.6% | 1.57× | 1.60× |
ibm |
5.16 | 5.32 | +3.0% | 2.00× | 1.95× |
igr |
2.02 | 3.25 | +60.8% | 1.56× | 1.11× |
grind run-to-run variance is 0.25–1.0%, so the 61% is far outside noise.
The good news
Case optimization is not broken by the workaround. On three of four cases CCE 21 lands
within 3–4% of CCE 19 and the case-opt speedup is preserved (1.56–1.95× vs 1.57–2.00×). Both
candidate workarounds — -plugin-opt=-instcombine-max-num-phis=0 and
-plugin-opt=-enable-load-in-loop-pre=false — give identical results within variance, so the
choice between them is not a performance question.
The blocker
igr costs 61% more per gridpoint on CCE 21, and its case-opt benefit collapses from 1.56× to
1.11×. Two facts on attribution:
igrnon-case-optimized is also slower on CCE 21 (3.62 vs 3.15, ~15%), so part of this is
independent of case optimization — but case-opt amplifies it 4×.- Both workarounds, which act on completely different passes (an InstCombine fold limit vs GVN
load-PRE), produce the same 1.11×. That points away from the workaround and toward CCE 21's
codegen for this kernel.
CCE 21 case-opt without any workaround cannot be measured — that is exactly the configuration
that fails to link, which is what this PR is working around.
Recommendation
This blocks the port. igr is the newest and most performance-sensitive solver path, and a
61% kernel regression is not acceptable regardless of the CCE 19 end-of-life timeline. The
remaining diagnosis is kernel-level: register pressure / occupancy for the IGR kernels under
CCE 21. First look shows CCE 21 emitting kernels at up to 256 VGPRs — the gfx90a threshold where
the unified register file starts spilling into AGPRs — but CCE 19's device image does not carry
comparable resource symbols, so this is not yet a like-for-like comparison.
Method note, because it changes conclusions
I first measured wall exec and reported that case optimization delivered only 1.03× and had
"collapsed". That was wrong. These runs are 5–22 s and exec is dominated by startup and
I/O, with ~10% run-to-run noise. grind isolates per-gridpoint RHS cost — what case optimization
actually targets — and has sub-1% variance. Every number above is grind. Anyone repeating this
should use it; exec at this problem size cannot resolve a 3% effect and barely resolves a 60%
one.
Case optimization: fixed and measured
Why
|
| candidate | what it disables | verdict |
|---|---|---|
-enable-load-in-loop-pre=false |
GVN load-PRE inside loops only | chosen — stops the malformed PHI from forming; LICM still hoists loop-invariant loads |
-enable-load-pre=false |
all load-PRE | broader, no benefit over the above |
-enable-pre=false |
all PRE | broader still |
-instcombine-max-num-phis=1 |
the multi-PHI walk | works; disables part of a fold used by every CCE device kernel |
-instcombine-max-num-phis=0 |
the whole fold | works; broadest of the instcombine options |
-plugin-opt=O1 |
the entire device LTO pipeline | rejected — would make case-optimized builds slower than the thing case-opt exists to speed up |
The GVN knob attacks the transform that creates the bad PHI rather than crippling
foldIntegerTypedPHI, which stays fully enabled everywhere else. It is one line to delete when
CCE ships a fix.
Case-opt performance vs today's production
Case-optimized, CCE 19 (pristine master) vs this branch on CCE 21, same node, 3 repeats,
medians of grind (ns/gridpoint/eq/rhs, variance < 1%):
| case | C19 copt | C21 copt | delta | C19 copt gain | C21 copt gain |
|---|---|---|---|---|---|
viscous_weno5_sgb_acoustic |
4.20 | 4.34 | +3.5% | 1.59× | 1.56× |
hypo_hll |
1.80 | 1.87 | +3.6% | 1.57× | 1.60× |
ibm |
5.16 | 5.32 | +3.0% | 2.00× | 1.95× |
igr |
2.02 | 3.25 | +60.8% | 1.56× | 1.11× |
Case optimization is intact. On three of four cases CCE 21 lands within 3–4% of CCE 19 and
the case-opt speedup is preserved.
igr: known regression, deliberately out of scope
igr costs 61% more per gridpoint on CCE 21 and its case-opt gain drops to 1.11×. This is
accepted for now — it does not block this PR — but it should not be forgotten, and the cause
is very likely our own workaround rather than a CCE 21 defect:
-mattr=-mai-insts (the AGPR link-crash workaround) disables MFMA/AGPR instructions. gfx90a has
a unified VGPR/AGPR register file, so this removes roughly half the allocator's budget.
Standalone reproducer, one file, no MFC:
| arm | max vgpr | max agpr | max scratch |
|---|---|---|---|
| baseline | 512 | 256 | 36 B |
-mattr=-mai-insts |
256 | none | 1060 B |
29× more spilling to off-chip scratch. MFC's IGR kernels show the same signature — three of them
go from 0 B to 448 B of scratch, and the CCE 19 device image carries 516 agpr_count records
against zero for CCE 21. IGR is the most register-hungry kernel in the code, which is why it
alone is hit.
Recovering it means narrowing or removing -mai-insts, which needs the underlying
AMDGPU Rewrite AGPR-Copy-MFMA assert fixed. Filed with the measured cost attached, since that
materially raises its priority.
Caveat kept explicit: the controlled test — rebuilding igr without the flag and re-timing —
was attempted and produced no valid result, because CRAY_CCE_LLD_ARGS reaches lld through
the environment and is not part of the build system's config hash, so the relink was a no-op
(md5-verified). The standalone table above is the controlled evidence for the mechanism; the
61% is the observed end-to-end effect. Those are different claims.
Method note
An earlier comment reported case-opt delivering only 1.03× and having "collapsed". That was
wrong — it measured wall exec, which on 5–22 s runs is dominated by startup and I/O with ~10%
noise. grind isolates per-gridpoint RHS cost and has sub-1% variance. Every number here is
grind.
OLCF documents cpe/26.03 + cce/21.0.2 with rocm/7.0.2, and advises keeping CCE and ROCm on consistent major clang versions. We had rocm/7.2.0 (clang 22) against CCE's clang 21; 7.0.2 is clang 20 and is the validated pairing for this cray-mpich. Measured equivalent: 627/0 on both mp and acc, 5/5 case-optimized builds, and grind within 1% on every comparable case (viscous -0.6%, hypo_hll -0.1%, ibm -0.7%, igr +0.5%). No reason to deviate from the supported configuration.
1b95701 moved the f slug to rocm/7.0.2 as the OLCF-recommended pairing for cpe/26.03, and applied the same version to famd and amdfund. That broke the AMD lanes: amd/7.2.0 with rocm/7.0.2 makes Lmod fail the load outright. module load python cmake cpe/25.09 PrgEnv-amd amd/7.2.0 rocm/7.0.2 -> rc=1, 'These module(s) ... cannot be loaded as requested: darshan-runtime' module load python cmake cpe/25.09 PrgEnv-amd amd/7.2.0 rocm/7.2.0 -> rc=0 modules.sh issues all modules in a single 'module load', so a failure loses every module including python -- which is why CI reported 'Python 3.6.15 (python3) is out of date. Required >= 3.9.' rather than anything about ROCm. Six AMD lanes failed (gpu-omp x3, cpu x2, Case Opt) while the same lanes are green on master. The 7.0.2 pairing rationale is specific to cpe/26.03 on the CCE slug, so keep it there and leave famd/amdfund on the version their compilers pair with. Verified both slugs now load rc=0. Committed with --no-verify: this branch does not yet carry the fix from MFlowCode#1697, so the pre-commit hook's pytest run would commit into the repository. Precheck was run manually and passed 7/7.
sbryngelson
changed the title
|
Pushed
The 7.0.2 pairing rationale is specific to Merge orderTake these in order; each is independent but the earlier ones remove hazards that affect the later ones:
No file overlaps between the three, so they can also merge in any order if preferred; the sequence above just front-loads the hazards. On the InstCombine workaround
|
Brings in MFlowCode#1697 (toolchain unit tests no longer commit into the contributor's own repository when run from the pre-commit hook) and MFlowCode#1696 (build dependency/configure skips keyed on real state rather than CMakeCache.txt). MFlowCode#1697 matters for this branch specifically: until now a commit here with the hook installed could have pytest rewrite the branch, which is why the previous two commits used --no-verify.
|
Measured the runtime cost of
The flag on this branch is the right pick. It is the cheapest of the options that actually remove the producer, and I verified that property independently: GVN goes from 4 mixed-address-space PHIs to 0 in the affected kernel with this flag set.
So this workaround costs 0.21% to protect a configuration worth 57%. Correctness of the workaround binaries (nothing before this tested them — only that the link succeeded): all five case-opt benchmarks run 10 timesteps clean, and output is bit-identical between candidate flags on four of five. Reproducer and full write-up for the underlying defect: https://github.com/sbryngelson/compiler-bugs/tree/main/cce/instcombine-phi-addrspace-cast |
Moves Frontier from
cce/19.0.0+rocm/6.3.1tocce/21.0.2+rocm/7.0.2.Result
Verified on the shipped configuration (
cpe/26.03+cce/21.0.2+rocm/7.0.2):--gpu mpfull suite--gpu accfull suiteROCm version selection
CCE 21's module states the constraint directly: "AMD GPU offloading support requires at least
ROCm 7.0 (ROCm 6.x is no longer supported)." Four versions qualify on Frontier.
rocm/7.0.2is what OLCF recommends forcpe/26.03, and they advise keeping CCE and ROCm onconsistent major Clang versions. An earlier revision of this PR used
rocm/7.2.0, which pairsCCE's Clang 21 with ROCm's Clang 22 — a mismatch, and outside the pairing validated for this
cray-mpich.cpe/26.03defaults to 21.0.0; we pin the patch releaseSwitching cost nothing, measured rather than assumed —
grind(ns/gridpoint/eq/rhs, medians of3, variance < 1%) on case-optimized builds:
viscous_weno5_sgb_acoustichypo_hllibmigrEvery delta is inside the noise band. Suite wall time is shorter on 7.0.2, but that is build
overhead — kernel cost is unchanged.
Performance vs today's production
Case-optimized, CCE 19 (pristine
master) vs this branch, same node, 3 repeats,grindmedians:viscous_weno5_sgb_acoustichypo_hllibmigrCase optimization is intact — three of four cases land within 3–4% of CCE 19 with the
case-opt speedup preserved.
igris a known regression, accepted for now, and the cause is very likely our ownworkaround rather than a CCE 21 defect.
-mattr=-mai-insts(needed for the AGPR link crash)disables MFMA/AGPR; gfx90a has a unified VGPR/AGPR register file, so this removes about half
the allocator's budget. Standalone one-file reproducer:
-mattr=-mai-insts29× more spilling. MFC's IGR kernels show the same signature — three go from 0 B to 448 B of
scratch, and the CCE 19 device image carries 516
agpr_countrecords against zero for CCE 21.IGR is the most register-hungry kernel in the code, which is why it alone is hit. The regression
is identical on ROCm 7.0.2 and 7.2.0 (3.27 vs 3.25), confirming it is CCE's linker decision
and not influenced by ROCm. Recovering it requires narrowing or removing
-mai-insts, whichneeds the underlying assert fixed upstream.
Caveat: the controlled test (rebuild
igrwithout the flag, re-time) produced no validresult —
CRAY_CCE_LLD_ARGSreacheslldvia the environment and is not in the build system'sconfig hash, so the relink was a no-op (md5-verified). The reproducer table is controlled evidence
for the mechanism; the 61% is the observed end-to-end effect.
What is in here
MFC bugs, correct on any compiler — worth taking independently of the toolchain move
m_qbmm.fpp— five procedures contained ins_mom_invreference nothing from it, so hoist themto module scope. Calling a sibling internal procedure from device code is not portable.
Move-only: apart from the removed
contains, all 922 lines are byte-identical. Also openstandalone as qbmm: hoist s_mom_inv's internal procedures to module scope #1688.
m_time_steppers.fpp—s_ibm_correct_statewas called withoutpb_in/mv_inon the non-QBMMpath while the routine dereferences them unconditionally (
m_ibm.fpphas nopresent()guards).
pb_ts(1)%sfis allocated and device-mapped in both branches, so pass it always anddelete the branch. See IBM: s_ibm_correct_state dereferences absent optional dummies (pb_in/mv_in) with no present() guard #1691.
parallel_macros.fpp—!DIR$ NOINLINE <name>is not a valid named directive (ftn-790), andit sat in a preprocessor arm never selected on GPU builds. Both fixed: the spelling is
INLINENEVER, emitted alongside the offload directive rather than instead of it.Scope correction. An earlier revision said this makes
cray_noinline"take effect on CrayGPU builds". That is overstated. The directive now reaches the compiler, but CCE 21 emits the
routine with
alwaysinlineregardless, so it remains inert in device code — verified fromthe device image, the pre-LTO bitcode, and against the documented behaviour in
man 7 inlinealways. All 51cray_noinlinesites are affected. Filed as a compiler defect.toolchain/dependencies/CMakeLists.txt— derive the hipfort tag fromROCM_PATHinstead ofpinning it. This is what let the ROCm version change above be a one-line edit. Note ROCm's
system hipfort is not a usable fallback on 7.0.2, 7.1.1, or 7.2.0: all three ship
hipfort_hipfft.modand advertise ahipfftcomponent but omithipfort-hipfft-targets.cmake, which their own config includes unconditionally.CCE 21 workarounds — three device-link flags in
toolchain/modules:-mattr=-mai-insts— avoids theAMDGPU Rewrite AGPR-Copy-MFMAassert that otherwise blocksthe device link. Carries the
igrcost above.-disable-promote-alloca-to-vector—AMDGPUPromoteAllocaToVectorsilently discards a store toarr(v)whenarris 1-based andvis a runtime value. Three sites are also patched insource (
m_riemann_state.fpp,m_riemann_solver_lf.fpp); unpatched ones remain inm_ibm,m_riemann_solver_hll,m_qbmm,m_collisions.-enable-load-in-loop-pre=false— fixes the case-optimization link abort (InstCombinefoldIntegerTypedPHIbuilding a cast across address spaces). Chosen over four alternativesbecause it stops the malformed PHI from forming while leaving the fold enabled everywhere.
-plugin-opt=O1also works but was rejected — it would make case-optimized builds slower thanthe thing case-opt exists to speed up.
Plus source fixes:
omp_macros.fppstops emittingdefaultmapon the CCE branch (anydefaultmap(<category>:...)clause makes an already-resident array read as zero inside thetarget region; this zeroed
ib_markersand faulted every immersed-boundary case — removing ittook
--gpu mpfrom 566/627 to 622/627), andm_variables_conversion.fppkeepsY_rsoffprivate frame offset 0.
The quoting is load-bearing.
modules.shdoeseval "export $line", so an unquotedmulti-value assignment exports only the first
-plugin-optand silently discards the rest. See#1689.
Correctness measurements
--gpu mp--gpu accdefaultmapremoved-disable-promote-alloca-to-vectorAll five residuals were the same promote-alloca defect, at magnitudes from 1.3e-10 to 1.9e-3. The
small ones were not floating-point noise — worth stating, because on a converged field they
look exactly like it, and I initially misread them that way.
Known gaps
Full list on #1684. What a reviewer should weigh:
igrregresses 61% (above) — accepted, not fixed.m_ibm.fppstill declares the dummiesoptional— harmless now that no call site omits them,but IBM: s_ibm_correct_state dereferences absent optional dummies (pb_in/mv_in) with no present() guard #1691 is only half-applied.
s_interpolate_image_point(nine optional dummies) is unaudited.find_in_present_table failed for 'length(:)'atm_ib_patches.fpp:145— a residency bug thatdefaultmap(tofrom:aggregate)was masking; now unmasked..github/workflows/test.yml:333marks the Frontier lanecontinue-on-error. That is why thecase-optimization failure sat unnoticed until CI was read by hand. Worth removing if this lands.
Compiler defects
Twelve, each with a standalone reproducer, at https://github.com/sbryngelson/compiler-bugs,
filed with OLCF/HPE. Three produce silent wrong answers: the promote-alloca dropped store, the
defaultmapzeroing, and the CCE 19CONTIGUOUSdefect.Context: #1684. Related: #1685, #1687, #1688, #1689, #1691, #1693.