[maleadt]

Back-ends look up the compiled kernel on every launch. CUDA.jl's cufunction, for example, resolves the method instance, builds the compiler config and a CompilerJob, takes a lock, queries the code cache through cached_results, and scans for the context's CuFunction. On a cache hit that is about 5500 instructions (0.20 µs), a third of the instructions of an asynchronous @cuda launch. For back-ends with cheaper launches, like the POCL host kernels in KernelAbstractions, the lookup is most of the launch.

This adds GPUCompiler.cached_lookup(lookup, V, F, tt, key)::V, which back-ends wrap around that lookup:

function cufunction(f::F, tt::TT=Tuple{}; kwargs...) where {F,TT}
    kw = values(kwargs)
    fun, state = GPUCompiler.cached_lookup(Tuple{CuFunction,KernelState}, F, tt,
                                           (context(), kw)) do
        cufunction_link(methodinstance(F, tt); kw...)
    end
    return HostKernel{F,tt}(f, fun, state)
end

It remembers the result per lookup function type, kernel signature, key type and result type, and reuses it while the task's world age and the back-end's key (here the context and the compiler keyword arguments, compared with isequal) are the same. The cache is a constant that a @generated function embeds in the caller, so a hit is an atomic load and a couple of comparisons, with no lock or hashing. It's an immutable list of the 4 most recent results, so a kernel used on a couple of devices or from tasks in different worlds doesn't keep missing.

The world has to match exactly. Checking that the cached code is still valid in the current world isn't enough: after compiling f(::Number) for an Int, adding f(::Int) leaves that code valid, but the kernel call now dispatches to the new method. In another world the lookup runs again and normally finds the compiled code in GPUCompiler's cache. Only the lookup's result is kept, not the callable, as its captured values have to come from each launch. Nothing is stored while a compile hook is active, so that @device_code_* sees every lookup, or while generating output, as the cache is serialized along with its callers. Back-ends that can recreate sessions should put something in the key that changes when they do; there's no API to clear the cache.

I also looked at making the uncached lookup cheaper for back-ends that don't adopt this, but there is little to gain there. Half of a cufunction hit is jl_rettype_inferred comparing the cache owner token field by field (box identity of the immutable token isn't preserved, so jl_egal's pointer check never hits), and methodinstance is already cheap.

I adopted this locally in two back-ends, measured on an RTX 5080 with Julia 1.12 (minimum time, user-space instructions on the calling thread):

	before	after
CUDA.jl cufunction (hit)	0.20 µs, 5476 insns	0.011 µs, 273 insns
CUDA.jl @cuda (async)	1.63–1.71 µs, 16965 insns	1.35–1.42 µs, 11810 insns
KernelAbstractions POCL clfunction (hit)	97 insns	70 insns

Most of what remains in CUDA.jl's cufunction is fetching the task-local context. The POCL "before" is JuliaGPU/KernelAbstractions.jl#923, which hand-writes the same cache; its tests pass with cached_lookup instead, as do CUDA.jl's execution, exception, reflection and KernelAbstractions tests.

The tests cover the more specific method, callee redefinition, lookups in an older world, captured values and the callable's lifetime, key changes and eviction, compile hooks, exceptions, concurrent use and precompilation. The test suite passes on Julia 1.10 and 1.12.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.42%. Comparing base (ed5ff41) to head (835ec1d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1061      +/-   ##
==========================================
+ Coverage   92.38%   92.42%   +0.03%     
==========================================
  Files          31       31              
  Lines        7285     7323      +38     
==========================================
+ Hits         6730     6768      +38     
  Misses        555      555              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.