Frontier module set silently pins CCE 19, and hipfort drifts from the loaded ROCm

[sbryngelson]

Frontier's module set silently pins CCE 19, and hipfort drifts from the loaded ROCm

Two independent defects in the Frontier toolchain configuration. Both are live on master
today, and together they are why MFC still builds with cce/19.0.0 — the one CCE version
carrying the IPA store-dropping bug that broke every multi-rank GPU test in #1679.

1. rocprofiler-compute silently reverts the compiler

toolchain/modules:54

f-gpu python craype-accel-amd-gfx90a rocprofiler-compute/3.0.0

rocprofiler-compute triggers a MODULEPATH change that re-resolves PrgEnv. It only ships a
build for the ROCm it was made against, and with any other ROCm the entire module load
falls back to the system default cce/18.0.1 — with no error, and no indication that the
cpe/... you asked for was dropped:

module line	resulting compiler
cpe/26.03 rocm/7.2.0 … craype-accel-amd-gfx90a	cce/21.0.0
the same + rocprofiler-compute/3.0.0	cce/18.0.1
the same + rocprofiler-compute/3.1.0	cce/18.0.1
cpe/25.09 rocm/6.4.2 … + either rocprofiler	cce/18.0.1
current cpe/25.03 rocm/6.3.1 … + rocprofiler-compute/3.0.0	cce/19.0.0 (works — only because 3.0.0 was built against 6.3.1)

So the current pin is the single combination that happens to work, and any ROCm bump
silently changes the compiler out from under the build rather than failing.

MFC only needs this module for the optional --rcu profiling flag
(toolchain/mfc/run/run.py:39). It should not be in the default GPU module set; load it
on demand when profiling is requested.

2. hipfort is pinned to a literal ROCm tag

toolchain/dependencies/CMakeLists.txt:148

ExternalProject_Add(hipfort
    GIT_REPOSITORY "https://github.com/ROCmSoftwarePlatform/hipfort"
    GIT_TAG        rocm-6.3.1

The tag is independent of the loaded rocm/X module, so bumping ROCm builds hipfort's
Fortran bindings for a different ROCm than the runtime. MFC consumes those bindings
directly (m_fftw.fpp, m_time_steppers.fpp: hipMalloc, hipMallocManaged,
hipDeviceSynchronize, hipfort_hipfft), and the failure mode is versioned-symbol link
errors rather than anything legible, e.g.

libcrayacc_amdgpu.so: undefined reference to `hipStreamSynchronize@hip_4.2'
libmpi_gtl_hsa.so:    undefined reference to `hsa_init@ROCR_1'

Deriving it from ROCM_PATH keeps the two together and fails loudly when no matching tag
exists, instead of silently building mismatched bindings:

set(MFC_HIPFORT_TAG "rocm-6.3.1")
if (DEFINED ENV{ROCM_PATH})
    string(REGEX MATCH "rocm-[0-9]+\\.[0-9]+\\.[0-9]+" MFC_ROCM_TAG "$ENV{ROCM_PATH}")
    if (MFC_ROCM_TAG)
        set(MFC_HIPFORT_TAG "${MFC_ROCM_TAG}")
    endif()
endif()

3. Minor: cpe/26.03 alone does not give the newest CCE

cpe/26.03 selects cce/21.0.0; the newest on Frontier is cce/21.0.2. If the intent is
"newest", the CCE version needs pinning explicitly alongside the CPE.

Why this matters beyond tidiness

.github/workflows/test.yml:333 already records that "cpe/25.03 introduced an IPA SIGSEGV
in CCE 19.0.0", and makes every Frontier lane continue-on-error. So MFC knows this
toolchain is bad, has made its CI non-blocking because of it, and the module set is what
keeps it there. The consequence is real: in #1679 all 11 multi-rank tests NaN'd on Frontier
CCE GPU and that would not have blocked the merge.

Working set on Frontier (module resolution and dependency build verified; a full MFC
build + regression run on both offload backends is in progress and I will follow up with
the result):

f-all cpe/26.03 cce/21.0.2 rocm/7.2.0
f-all cray-fftw cray-hdf5 python cmake
f-gpu python craype-accel-amd-gfx90a

with hipfort resolving to rocm-7.2.0. (ROCm 7.13.0 exists but hipfort has no tag past
rocm-7.2.4, and rocm-7.2.0…7.2.4 are all the same hipfort commit, so 7.2.0 is the
newest usefully-paired choice.)

[sbryngelson]

Follow-up with the build results promised above. The three toolchain fixes work — MFC's loader reaches cce/21.0.2 and the dependencies build clean, hipfort correctly deriving rocm-7.2.0. But moving off cce/19.0.0 is blocked by compiler defects, not by anything in MFC.

Every CCE newer than 19 fails to link, on both offload backends

Every Fortran source compiles — zero ftn errors in all cases. Only the device LTO link dies.

CCE	result
19.0.0 (current pin)	links; carries the store-dropping IPA bug worked around by #1679
20.0.2	lld heap corruption, malloc_consolidate(): unaligned fastbin chunk, in pass Infer address spaces on s_cbc$m_cbc_$ck_L596_13 (m_cbc.fpp:596)
21.0.0, 21.0.2	lld assertion SlotIndexes.h:96 "Attempt to compare reserved index" in pass AMDGPU Rewrite AGPR-Copy-MFMA

Both fail identically under --gpu mp and --gpu acc.

These are not caused by anything in MFC. I ran a controlled pair on identical compiler, ROCm and dependencies, differing only by reverting a local macro commit that activates 51 !DIR$ INLINEALWAYS/INLINENEVER directives: stock upstream master crashes identically, with those directives inactive.

Root cause of the 21.x failure, and a targeted workaround

CCE ships its own matching LLVM 21.1.8 with assertions in cce-clang/x86_64/bin, so the crash reduces from a 19 MB whole-program link to a 495 KB single-function module driven by llc alone — no MFC, no build system, no lld.

Ruled out by experiment, not inspection:

• the "amdgpu-agpr-alloc"="0" annotation (crashes with it removed, and at 4 and 32)
• register-spill pressure (amdgpu-waves-per-eu=1,1, amdgpu-num-vgpr=256, amdgpu-flat-work-group-size=1,64 all still crash)
• a single unlucky kernel (deleting it exposes a second)

What holds up: optnone on the function suppresses the crash, so the pass honours
skipFunction(). And exactly two functions in the entire program trigger it:

m_bubbles_EE.fpp:185   s_compute_bubble_EE_source     (25 privates + copy reduction)
m_bubbles_EL.fpp:633   s_compute_bubble_EL_dynamics   (32 privates + copy reduction)

Both are adaptive-dt bubble sub-stepping kernels — the highest register pressure in the
code. Note MFC contains no MFMA/matrix instructions at all, so this pass should have nothing
to transform.

That gives a targeted workaround rather than a global one: compile just those two TUs at
-O0 so their device functions carry optnone, leaving everything else fully optimised.

if (CMAKE_Fortran_COMPILER_ID STREQUAL "Cray" AND (MFC_OpenACC OR MFC_OpenMP) AND
    CMAKE_Fortran_COMPILER_VERSION VERSION_GREATER_EQUAL 20)
    set_source_files_properties(
        "${CMAKE_BINARY_DIR}/fypp/simulation/m_bubbles_EE.fpp.f90"
        "${CMAKE_BINARY_DIR}/fypp/simulation/m_bubbles_EL.fpp.f90"
        PROPERTIES COMPILE_OPTIONS "-O0")
endif()

I am testing this now and will report whether it links and passes.

For completeness: forcing the whole device link to -plugin-opt=O0 also avoids both
crashes, but there is no supported way to get that flag to lld — -Wl, reaches the host
GNU ld (bad -plugin-opt option), -Wc, reaches llvm-link (Unknown command line argument) despite being documented as device-linker arguments, and the driver invokes
lld by absolute path so a PATH shim is never consulted.

Why this is worth acting on regardless

.github/workflows/test.yml:333 already records that "cpe/25.03 introduced an IPA SIGSEGV
in CCE 19.0.0" and makes every Frontier lane continue-on-error. So MFC has known this
toolchain is bad and has made its CI non-blocking because of it. In #1679 all 11 multi-rank
tests NaN'd on Frontier CCE GPU and that would not have blocked the merge.

The three toolchain fixes in this issue are worth taking now — they are independently
correct and they unblock the upgrade the moment HPE fixes lld. Reproducers for both lld
crashes (including the reduced 495 KB one) are at
https://github.com/sbryngelson/compiler-bugs under cce/.

[sbryngelson]

Cross-referencing the compiler-side reproducers, which live in a separate repo so they can be handed to HPE independently of MFC:

entry	what it is
cce/lld-agpr-mfma-assert	CCE 21.0.0/21.0.2 — lld asserts in AMDGPU Rewrite AGPR-Copy-MFMA. Includes a self-contained 495 KB repro/ that crashes llc directly, plus the four hypotheses ruled out by experiment
cce/lld-infer-address-spaces-cce20	CCE 20.0.2 — lld heap corruption in Infer address spaces, reproduced on stock upstream master via a reverted-patch control
cce/contiguous-mix-dropped-stores	CCE 19.0.0 — the store-dropping IPA bug behind the #1679 multi-rank NaNs; why staying on 19.0.0 is not free either
cce/mir-roundtrip-bb-name	minor: llc prints MIR block labels its own parser rejects for Fortran names containing a comma; blocks MIR-level reduction

Each carries a Tracking table pointing back here.