Repository navigation
Less compilation for a kernel's first launch - #931
Merged
Merged
Conversation
maleadt
added
the
latency
Contributor
Benchmark ResultsShow table
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
This was referenced Oct 27, 2026
Strip singleton kernel functions before `launch_setup`, `select_launch` and `launch_geometry` so they compile once per backend and static sizes. Preserve stateful callables for custom hooks. First launch with a previously launched shape (Julia 1.12): PoCL 1-d 61 -> 48 ms, 3-d with partial work-groups 68 -> 49 ms; CUDA (RTX 5080) 81 -> 76 ms and 98 -> 83 ms.
Build `launch_converted` argument expressions with a vector-based helper so it compiles once across argument counts. The first kernel with a new argument count (3 to 8) launches in 94 instead of 120 ms (Julia 1.12).
Use `fma` in the precompile kernel to precompile LLVM.jl's `llvmcall` generators for OpenCL builtins. A session's first axpy launch drops from 425 to 299 ms (Julia 1.12).
Since #922, `launch_kernel` looks up every kernel through `kernel_function_as`, which hides the kernel's type from inference so that PoCL compiles the lookup once for both checked and unchecked kernels. On other backends, this made every launch dispatch dynamically. Call `KI.kernel_function` directly there; the branch on `skip_bounds_check` folds at compile time. Per-launch host cost, ns (Julia 1.12, best of 200 batches of 64 launches, median of 6 interleaved runs): CUDA (RTX 5080) 1749 -> 1598 for a 1-d kernel and 1838 -> 1635 for a 3-d kernel (raw `@cuda`: 1468); AMDGPU (gfx1036) 1364 -> 1121 and 1351 -> 1124 (raw `@roc`: 727). A new kernel's first launch takes 6-17 ms less.
maleadt
force-pushed
the
tb/launch-cost-main
branch
from
October 27, 2026 20:16
a2dd21b to
7b9b007
Compare
Contributor
Benchmark resultsCompared with 466ad8a16, the merge base.
No unusual changes. All benchmarks
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #931 +/- ##
==========================================
- Coverage 83.30% 82.76% -0.54%
==========================================
Files 31 31
Lines 2569 2524 -45
==========================================
- Hits 2140 2089 -51
- Misses 429 435 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four launch-cost improvements from the stack on #906 that also help main, where launches go through PoCL's queue (2 µs on a Ryzen 9950X, 6 µs on a Xeon E3). Changes that only save a few cycles per launch stay with #906.
@kernelfunctions) beforelaunch_setup,select_launchandlaunch_geometry, so that their code is shared by kernels with the same backend and static sizes. Stateful callables keep their type for custom hooks. Applies to every backend.launch_convertedgenerator uses vectors instead of tuples, so it isn't recompiled for each argument count.muladd#922, kernel lookups hid the kernel's type from inference so PoCL could share lookup code between checked and unchecked kernels. Restore inference on CUDA, AMDGPU, oneAPI, Metal and OpenCL to avoid dynamic dispatch on every launch.fma, which precompiles LLVM.jl'sllvmcallgenerators.First launches, ms, median of 3 interleaved runs (Ryzen 9950X, Julia 1.12,
-t 4):Oceananigans hydrostatic model (36x18x10,
-t 4,0, median of 3): build 18.43 → 18.27 s, first time step 21.72 → 21.47 s.Per-launch host cost on GPUs before and after inferring the kernel's type, ns, best of 200 batches of 64 launches, median of 6 interleaved runs (Julia 1.12):