Repository navigation
POCL: vectorizable kernels without bounds checks, and FMA for muladd - #922
Merged
Merged
Conversation
Contributor
Benchmark ResultsShow table
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
maleadt
force-pushed
the
tb/pocl-codegen
branch
5 times, most recently
from
October 25, 2026 18:13
2ee7481 to
c155692
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## tb/pocl-launch-cache #922 +/- ##
========================================================
+ Coverage 82.50% 83.05% +0.54%
========================================================
Files 31 31
Lines 2510 2556 +46
========================================================
+ Hits 2071 2123 +52
+ Misses 439 433 -6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Every `@kernel` was compiled with `DynamicCheck`, so PoCL's loop over the work-items had a conditional body. On x86 it is vectorized with masked loads and stores, and on AArch64 it isn't vectorized at all, which made e.g. an axpy 2x (x86) to 5x (M1) slower than a serial Julia loop. On the POCL back-end, launches whose workgroups cover the ndrange exactly now run a separate kernel compiled with `NoDynamicCheck`. Custom iteration spaces keep the check, as they may map work-items outside of the ndrange. Other back-ends are unchanged: there the check is cheap, and a second kernel costs compile time. A launch only compiles the kernel it runs. With a tuned workgroup size, which kernel that is depends on the tuned size, so it is predicted from the device's limits; PoCL's kernels report those too, unless their estimated stack use limits their workgroup size. If the workgroups tuned for such a kernel don't cover the ndrange, the launch compiles the checked kernel after all. The unchecked kernel is launched with the context of the checked one, which only differs in a type parameter, and looked up by the same code: the kernel lookup takes the types to compile for as an argument it doesn't specialize on, and caches the unchecked kernel separately, so that launches alternating between both kernels find either on the fast path. The host code launching a kernel is thus compiled once per kernel, and choosing the kernel adds 12-21M instructions (3-6 ms) to the first launch of a kernel on a Xeon E3-1245 v6. Kernels that PoCL compiles faster without the check, like a reduction, launch for the first time 12-15 ms sooner.
Julia marks the multiplication and addition of `muladd` as contractable, but the SPIR-V back-end can only carry that with `SPV_KHR_float_controls2`, which PoCL doesn't accept. PoCL thus computed every `muladd` as a separate multiplication and addition, which made a chain of dependent `muladd`s 2.6x (x86) to 4x (M1) slower than in Julia. As a stopgap, compile `muladd` of `Float32` and `Float64` as `fma` when the host CPU, which PoCL's CPU device runs on, has hardware FMA: on x86 with the `fma` feature, and always on AArch64. The overlays live in a separate method table that only jobs with the new `fuse_muladd` compiler parameter use, so the choice is part of the cache key.
maleadt
force-pushed
the
tb/pocl-codegen
branch
from
October 26, 2026 07:40
c155692 to
4f49ec0
Compare
maleadt
added a commit
that referenced
this pull request
Oct 27, 2026
Since #922, `launch_kernel` looks up every kernel through `kernel_function_as`, which hides the kernel's type from inference so that PoCL compiles the lookup once for both checked and unchecked kernels. On other backends, this made every launch dispatch dynamically. Call `KI.kernel_function` directly there; the branch on `skip_bounds_check` folds at compile time. Per-launch host cost, ns (Julia 1.12, best of 200 batches of 64 launches, median of 6 interleaved runs): CUDA (RTX 5080) 1749 -> 1598 for a 1-d kernel and 1838 -> 1635 for a 3-d kernel (raw `@cuda`: 1468); AMDGPU (gfx1036) 1364 -> 1121 and 1351 -> 1124 (raw `@roc`: 727). A new kernel's first launch takes 6-17 ms less.
maleadt
added a commit
that referenced
this pull request
Oct 27, 2026
Since #922, `launch_kernel` looks up every kernel through `kernel_function_as`, which hides the kernel's type from inference so that PoCL compiles the lookup once for both checked and unchecked kernels. On other backends, this made every launch dispatch dynamically. Call `KI.kernel_function` directly there; the branch on `skip_bounds_check` folds at compile time. Per-launch host cost, ns (Julia 1.12, best of 200 batches of 64 launches, median of 6 interleaved runs): CUDA (RTX 5080) 1749 -> 1598 for a 1-d kernel and 1838 -> 1635 for a 3-d kernel (raw `@cuda`: 1468); AMDGPU (gfx1036) 1364 -> 1121 and 1351 -> 1124 (raw `@roc`: 727). A new kernel's first launch takes 6-17 ms less.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #923.
On one CPU core, KA kernels on the POCL back-end were 1.15–2.9x (x86) and 2.6–10.6x (Apple M1) slower than the equivalent serial Julia loop. Two things caused almost all of it:
@kernelis compiled withDynamicCheck, so PoCL's loop over the work-items has a conditional body. On x86 it gets vectorized with masked loads and stores; on AArch64 it doesn't get vectorized at all. Launches whose workgroups cover the ndrange exactly now run a separate kernel compiled without the check. A launch only compiles the kernel it runs: with a tuned workgroup size, that size is predicted from the device's limits (falling back to the checked kernel if the kernel's own limits are smaller and its tuned workgroups don't cover the ndrange). Both kernels share the host code that looks them up and launches them, the unchecked one getting the checked one's context, which only differs in a type parameter; the unchecked kernel is cached separately from the checked one, so launches alternating between both find either on the fast path. Custom iteration spaces keep the check, since they may map work-items outside of the ndrange. Only the POCL back-end changes: on GPUs the check is cheap, and a second kernel would cost compile time.muladddoesn't contract. The SPIR-V back-end can only express contraction withSPV_KHR_float_controls2, which we don't emit yet (POCL: contract fast-math flags lost through SPIR-V, no FMAs emitted #865, SPIR-V: declare FloatControls2 for kernels with SPV_KHR_float_controls2 GPUCompiler.jl#1007) and the PoCL JLL can't read yet (Enable SPV_KHR_float_controls2 for the CPU devices pocl/pocl#2393 needs LLVM 23), so everymuladdbecame a separate multiplication and addition. Until then, as a stopgap,muladdofFloat32/Float64now compiles tofmawhen the host CPU has hardware FMA (x86 withfma, all AArch64), through an overlay table selected by a compiler parameter, so the choice is part of the cache key. GPUCompiler'shave_fmadoesn't answer this, since OpenCL'sfmamay be implemented in software.Queue-path timings, main → this PR (µs; best of 3; serial Julia loop in brackets):
-t 4,0-t 4,0muladds, 1e6Choosing between the two kernels adds 0–6 ms (up to 21M instructions) to the first launch of a new kernel on top of #923, while kernels that PoCL compiles faster without the check launch sooner: the reduction's first divisible launch takes 12–15 ms less. Median over 9 fresh kernels on the Xeon with
-t 4, for an elementwise kernel, a 2D stencil and an AK-style reduction with a fresh closure, each with divisible and non-divisible ndranges: #923 → this PR takes 105–137 ms → 110–143 ms with a workgroup size that is tuned (or, for the reduction, given at launch), and 101–132 ms → 106–135 ms with one in the kernel. Against main, both PRs together take it from 158–192 ms to 110–143 ms, and from 142–174 ms to 106–135 ms.The remaining difference at 1 thread on small arrays is the queue's launch cost. Launches whose tuned workgroup size doesn't divide the ndrange (e.g. 1e5 elements, or a 126³ stencil) keep the check and are unchanged. The
muladdrow on M1 is one of those. Running their full workgroups without the check is part of the host-kernel work (#906).