Skip to content

fix(mir-importer): normalize pointer address spaces in tuple construction - #1454

Merged
nihalpasham merged 4 commits into
NVIDIA:mainfrom
uurl:fix/as7-tuple-normalization
Oct 15, 2026
Merged

nihalpasham merged 4 commits into
NVIDIA:mainfrom
uurl:fix/as7-tuple-normalization

Conversation

@uurl

@uurl uurl commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

A tuple holding a cluster-shared pointer from cluster::map_shared_rank now compiles. Tuple construction casts each element to the tuple's declared element type before building the aggregate, the same normalization that struct, enum, array, union and slice construction already apply, so the addrspace(7) pointer enters the generic slot through an addrspacecast instead of failing MIR verification.

Before: Verification failed for 'test_dsmem_ring_exchange':
        MirConstructTupleOp operand 0 type mismatch.
        Expected: mir.ptr <ui32, addrspace:0, kind:RawConst>
        Actual:   mir.ptr <ui32, addrspace:7, kind:RawConst>
After:  %p = addrspacecast ptr addrspace(7) %mapped to ptr
        insertvalue { ptr, i64 } undef, ptr %p, 0
        call { ptr, i64 } @bounce_cluster_pair(...)
        PTX -O2:    mapa.shared::cluster.u64 ; ld.shared::cluster.b32
        PTX NO_OPT: mapa.shared::cluster.u64 ; cvta.shared::cluster.u64 ; call.uni bounce_cluster_pair ; ld.b32
#[inline(never)]
#[device]
fn bounce_cluster_pair(pair: (*const u32, u64)) -> (*const u32, u64) {
    pair
}

let neighbor_ptr = unsafe { cluster::map_shared_rank(addr_of!(SHMEM) as *const u32, neighbor_rank) };
let pair = bounce_cluster_pair((neighbor_ptr, neighbor_rank as u64));
let bounced = unsafe { *pair.0 }; // reads the neighbor block's shared memory

What we fixed

  • AggregateKind::Tuple was the only aggregate arm that built MirConstructTupleOp straight from the translated operands. The verifier requires operand types equal to the element types, so any tuple holding a pointer outside the generic address space (cluster-shared addrspace(7), CTA-shared addrspace(3)) failed.
  • The arm now reads the translated MirTupleType element types and runs cast_to_expected_pointer_type_if_needed per element, keeping the pointer kind (RawConst stays RawConst) and declining kind changes as before.
  • The cluster example round-trips the mapped pointer through a (ptr, rank) tuple and a device fn, and its IR check requires the tuple-ABI call and the addrspacecast in the pre-opt IR; the compile-only example lane runs that check.

Address-space contract

  • A mapped pointer stored in an aggregate (struct, enum, array, and now tuple) is held in the generic form, which is also what CUDA C++ map_shared_rank returns. A later deref through the aggregate is a generic access unless LLVM can prove the space; at -O2 it restored ld/st.shared::cluster in every form tested, and a true mixed cluster/CTA join produced a correct generic load.
  • A plain deref of the mapped pointer itself still selects ld/st.shared::cluster.
Implementation
  • mir-importer translator/rvalue/aggregate.rs, tuple arm: downcast the translated tuple type to MirTupleType, zip get_types() with the element values, cast each through the shared helper and thread the previous op. The downcast cannot fail: translate_type always yields MirTupleType for a Rust tuple.
  • translator/rvalue/coerce.rs: a helper-contract test that an addrspace(7) RawConst pointer normalizes to the generic RawConst type through mir.cast. It documents the helper; the regression for the arm is the example plus its IR check.
  • The cast is mir.cast <PtrToPtr>; mir-lower emit_pointer_cast already emits llvm.addrspacecast when the spaces differ, so no lowering change.
  • cluster example: bounce_cluster_pair, the tuple round trip and comparison against the direct read; verify-code-shape.sh asserts the call and the addrspacecast in cluster.ll, read before opt -O2 because the private helper is inlined there.

Verification

Checked at 95133c9c merged with main (bc46b1ff, nightly-2026-10-11):

  • ✓ Fails before: the PR's cluster source built against main's backend reports the [BUG]: MIR tuple construction fails for cluster-shared AS7 pointer operands #1453 MirConstructTupleOp mismatch; the same source builds at this head.
  • ✓ cargo test -p mir-importer: 1164 passed at the PR head, 1169 on the merged tree, 0 failed. cargo clippy -p mir-importer --all-targets -- -D warnings and rustdoc with warnings denied pass; the example's own clippy passes.
  • ✓ cargo oxide run cluster on an RTX 5060 Laptop (author) and an RTX 5090 (sm_120): ring exchange 1001, 1002, 1003, 1000 and mapped store 3003, 3000, 3001, 3002; memcheck and synccheck 0 errors.
  • ✓ CUDA_OXIDE_NO_OPT=1 cargo oxide run cluster: same values with the PTX keeping cvta.shared::cluster.u64, call.uni bounce_cluster_pair and a generic ld.b32 through the tuple; memcheck 0 errors.
  • ✓ cargo oxide run cluster --emit-nvvm-ir --arch sm_120 (libNVVM): passes.
  • ✓ cargo oxide build cluster --arch sm_90 plus verify-code-shape.sh: PASS, the compile-only lane's form.
  • ✓ Further tuple forms (nested, by reference, array of tuples, Option<*const u32>, destructured return, *mut store through a tuple, layout-reordered (u8, *const u32, u16), control-flow join, pointer equality across the cast, mixed addrspace(7)/addrspace(3) join) compile and run correctly at -O2 and NO_OPT; memcheck and synccheck 0 errors. A by-value closure capture of the mapped pointer already compiled on main.
  • ✓ PTX and IR of the unchanged tuple examples are identical to main.
  • ✓ Example lockfiles, all repository guards, git diff --check and the per-example rustfmt check pass; DCO sign-off present.

Caveats

  • At -O2 the example's bounced == direct comparison folds to true because LLVM inlines the helper and merges the two loads; the enforced gate is the IR check, and the generic load through the tuple executes only in the NO_OPT build.
  • Pre-existing and outside this PR: assigning such a pointer into a tuple or struct field after construction (pair.0 = p), or [p, p] / [p; 2] into an addressable local, fails MirStoreOp verification on main as well; the store path lacks the same normalization. Tracked separately.
  • No sm_90 hardware run; the PTX ISA defines cvta.shared::cluster for sm_90+ and the runs were on sm_120. The legacy CUDA 12 NVVM route at sm_90 fails on this host for main too (PTX JIT error) and was not validated.

Closes #1453.

@uurl
uurl requested a review from nihalpasham as a code owner October 8, 2026 08:52
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Raul Estrada <raulestradaa@gmail.com>
@uurl
uurl force-pushed the fix/as7-tuple-normalization branch from ff2b278 to 95133c9 Compare October 8, 2026 08:55
@nihalpasham nihalpasham added the cuda-oxide SIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book label Oct 11, 2026
…README

Run rustfmt over the cluster example so the per-example formatting
check passes: the bounce_cluster_pair signature, the `let pair`
binding and the `let neighbor_value` if-expression each move back
onto one line. No semantic change.

Extend the test_dsmem_ring_exchange README bullet: the mapped pointer
is also round-tripped through a (ptr, rank) tuple and a device fn and
compared against the direct read.

Signed-off-by: nihalpasham <nihalp@nvidia.com>
The tuple round-trip comment attributed the missing call in the PTX to
the PTX optimizer. It is LLVM opt -O2 that inlines the private
#[inline(never)] helper, because noinline is not propagated to the IR,
so the call survives only in the pre-opt cluster.ll. Say so.

Add a require_ll_shape assertion that the mapped addrspace(7) pointer
is cast to a generic pointer before the tuple is constructed. The call
check only proves the helper was reached; this one pins the
normalization itself, which is what the round-trip exists to cover.

Signed-off-by: nihalpasham <nihalp@nvidia.com>
@nihalpasham

Copy link
Copy Markdown
Collaborator

/ok to test 8abc330

@nihalpasham nihalpasham added bug Something isn't working codegen Device code-generation pipeline (Rust MIR to IR to PTX) examples Example kernels and host harnesses under crates/rustc-codegen-cuda/examples labels Oct 15, 2026
@nihalpasham

Copy link
Copy Markdown
Collaborator

Checked at 8abc3308 on the 5090:

  • Building the new cluster source against main's backend reproduces the [BUG]: MIR tuple construction fails for cluster-shared AS7 pointer operands #1453 verifier error; with this change the importer casts the cluster-shared pointer before building the tuple, the same way the struct, enum and array arms already do.
  • cargo oxide run cluster passes all cluster and DSMEM tests on sm_120a, and the code-shape check passes at sm_120a and sm_90.
  • I pushed three things on top: main merged in for the nightly-2026-10-11 pin, rustfmt plus a README bullet, and the IR check now also asserts the address-space cast happens before the tuple is built.
  • 1,164 mir-importer tests, clippy and the guards pass. CI is green on this head.

Closes #1453.

@nihalpasham
nihalpasham merged commit 9f49388 into NVIDIA:main Oct 15, 2026
60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codegen Device code-generation pipeline (Rust MIR to IR to PTX) cuda-oxide SIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book examples Example kernels and host harnesses under crates/rustc-codegen-cuda/examples

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: MIR tuple construction fails for cluster-shared AS7 pointer operands

3 participants