Repository navigation
High and variable launch + synchronize latency for small kernels (~56 µs, often 2x) #756
Description
Launching a trivial kernel and synchronizing takes about 56 µs on an Arc A770. A noticeable share of calls take twice that or more:
- the launch itself costs about 18 µs of host time, and still 15 µs per launch when 1000 launches are queued back to back with a single synchronize at the end;
synchronizethen takes a median of 39 µs, with a p90 of 110 µs;- back to back, about 10% of launch+sync calls take 125–155 µs instead of 56 µs. After the host has been idle for 1 ms, the median is 97–117 µs.
synchronize(blocking=true)behaves the same.
For small problems this dominates. We ran into it in JuliaGPU/AcceleratedKernels.jl#194, where medians of back-to-back calls on 10^4 elements flipped by ±30% between runs, depending on the share of slow samples. Earlier, with heavier kernels and some background CPU load, back-to-back calls were bimodal at about 60 µs and 125–155 µs, with each process tending to stay in one mode. I haven't found out whether the cost is in oneAPI.jl's launch path, Level Zero, or the GPU's power management.
launch: p10 17.0 p50 17.9 p90 18.6 µs
sync: p10 37.7 p50 39.0 p90 109.7 µs
1000 back-to-back launches + 1 sync: 15.1 µs per launch
blocking=false gap 0 µs: p10 55.8 p50 56.7 p90 127.8 µs
blocking=false gap 10 µs: p10 56.4 p50 58.4 p90 59.8 µs
blocking=false gap 50 µs: p10 56.6 p50 57.1 p90 59.3 µs
blocking=false gap 1000 µs: p10 69.5 p50 96.9 p90 149.4 µs
blocking=true gap 0 µs: p10 55.6 p50 56.4 p90 126.9 µs
blocking=true gap 1000 µs: p10 70.5 p50 117.2 p90 148.7 µs
Environment: oneAPI.jl 2.10.0, NEO 26.35.39758, libigc 2.41.5, Level Zero API 1.17, Julia 1.13.2, Arc A770, Linux 7.2.11 (Arch), otherwise idle machine.
Reproducers
# Split launch + sync of a trivial kernel: host time of the launch, of the synchronize, and the device time
using oneAPI, Statistics, Printf
z = oneArray(zeros(Float32, 1))
k(a) = (@inbounds a[1] = 1f0; nothing)
for _ in 1:100; @oneapi k(z); oneAPI.synchronize(); end
tl = Float64[]; ts = Float64[]
for _ in 1:1000
t0 = time_ns(); @oneapi k(z); t1 = time_ns(); oneAPI.synchronize(); t2 = time_ns()
push!(tl, (t1 - t0) / 1e3); push!(ts, (t2 - t1) / 1e3)
end
@printf("launch: p10 %5.1f p50 %5.1f p90 %5.1f µs\n", quantile(tl, [0.1, 0.5, 0.9])...)
@printf("sync: p10 %5.1f p50 %5.1f p90 %5.1f µs\n", quantile(ts, [0.1, 0.5, 0.9])...)
# launch-only throughput: 1000 launches, one sync
t = @elapsed (for _ in 1:1000; @oneapi k(z); end; oneAPI.synchronize())
@printf("1000 back-to-back launches + 1 sync: %.1f µs per launch\n", 1e6 * t / 1000)# Wall time of launching a trivial kernel and synchronizing, after the host idles for `gap` µs
using oneAPI, Statistics, Printf
z = oneArray(zeros(Float32, 1))
k(a) = (@inbounds a[1] = 1f0; nothing)
function spin(us)
t = time_ns() + round(UInt64, us * 1000)
while time_ns() < t; end
end
for blocking in (false, true)
for _ in 1:100; @oneapi k(z); oneAPI.synchronize(; blocking); end
for gap in (0, 10, 25, 50, 100, 1000)
ts = Float64[]
for _ in 1:500
spin(gap)
push!(ts, 1e6 * @elapsed (@oneapi k(z); oneAPI.synchronize(; blocking)))
end
q = quantile(ts, [0.1, 0.5, 0.9])
@printf("blocking=%-5s gap %4d µs: p10 %6.1f p50 %6.1f p90 %6.1f µs\n", blocking, gap, q...)
end
end
Activity