Repository navigation
Conversation
`OptionalOp`, which combines the partial results of reductions and scans whose operator has no neutral element, returned one of its operands or a new `Optional`. The compiler merged those results through stack slots (a phi of pointers to allocas, which SROA cannot promote), so on CUDA the reduction loops kept local-memory loads and stores. Building one result from the operands' fields avoids the stack slots. Reductions without a neutral element get up to 3.5x faster on CUDA, 2.4x on PoCL, 1.5x on oneAPI and 1.3x on Metal.
This was referenced Oct 28, 2026
For a value that isn't of a primitive type, such as the tuple of a `mapreduce` with `(max(p[1], q[1]), max(p[2], q[2]))`, Julia lowers the select between the operands' values through stack slots, so the running partial went through local memory on every combine. On an RTX 6000 Ada that made a `Float32` pair reduction of 10^7 elements 3.3x slower in its kernels (14.1 us on main, 46.6 us with the select). Such values keep the old combine, which returns either operand.
This was referenced Oct 28, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reductions and scans whose operator has no known neutral element (no
neutralkeyword and noGPUArraysCore.neutral_elementmethod, so also Basereducewith a custom operator via GPUArrays) keep their partial results asOptionals, combined byOptionalOp. That returned one of its operands or a newOptional, so the compiler merged the possible results through stack slots: two allocas and aphi ptrbetween them, which SROA can't promote. For example, in the loop of the one-thread-per-output reduction kernel:On CUDA that stays local memory: the SASS of that kernel keeps 4
STLand 1LDL(local-memory stores and loads) for the running partial. This PR builds the result from the operands' fields instead (ifelseon the values,opcalled only when both are set), and the PTX has no local memory left. This only applies to primitive types: Julia selects between other values, such as the tuples of amapreducewith(max(p[1], q[1]), max(p[2], q[2])), through stack slots too, which made such a pair reduction of 10^7 elements 3.3x slower in its kernels on an RTX 6000 Ada (14.1 µs on main, 46.6 µs). Those, and non-bits types (whose value can be undefined), keep the old code.New/main time ratios for
Float32maxwithout a neutral element, 10^4 to 10^8 elements (median of 3 runs):dims=2reduceaccumulate!accumulate,dims=1Some cases get slower. PoCL scans along
dimsthat take a block per slice get 1.03–1.3x slower: PoCL runs each work-item's serial scan of its run (thread_scan!) as scalar code, and LLVM compiles the new combine into a chain of two masked blends afterop(select(sa, select(sb, op, a), b)), where main had branches that were always predicted. Fewer instructions run, but each step of the dependency chain takes longer. GPUs hide that latency; using the old combine in that kernel only would recover PoCL but give back the 4–12% gain on CUDA, so I left it.On CUDA,
accumulatealongdims=2of an (8, 10^7) array gets 1.12x slower on an RTX 5080 (not on an RTX 6000 Ada). That shape takes the split one-thread-per-slice scan, whose totals pass is bound by memory latency: on main, ptxas happened to unroll its loop and keep several loads in flight, which it doesn't do for the select-based combine. #199 makes that kernel load eight elements at a time, which makes this scan about 2x faster than main.On oneAPI, small sizes are dominated by a 55–155 µs fixed per-call latency that comes in two modes depending on how soon the next call follows, so medians of back-to-back runs at 10^4 elements flip between them (one run gave the
dims=2reduce 1.35). The kernel itself is faster there (8.8–9.7 µs against 12.9–14.8 µs), as are the minimum wall times (71–72 µs against 74–78 µs).Known issue: with this PR alone (a
Boolflag), Intel's CPU OpenCL runtime (2026.21, not in CI) segfaults in its X86 instruction selection while compiling theaccumulate!tile kernel; main and the wide flag of #192 compile fine. Merging this together with #192 (or #195, which keeps the wide flag there) avoids it. Tracked in JuliaGPU/OpenCL.jl#600 (not reduced yet).Tested reduce/accumulate on CUDA, PoCL, Metal, oneAPI, and with OpenCL on an Arc A770 and NVIDIA GPUs.