[maleadt]

This replaces benchmark/ and its AirspeedVelocity workflow with a perf/ suite like CUDA.jl's, Metal.jl's and AMDGPU.jl's. Its results are published to the history at https://juliagpu.org/perf/, and pull requests get a comment comparing them with it.

The old suite measured steady-state kernel times, mostly saxpy over a grid of sizes, on shared GitHub runners. It missed the regressions of the 0.10 cycle:

• time to the first kernel (Precompile the CPU back-end's compiler: first kernel launch in 0.6 s instead of 15 s #816);
• load time (KernelAbstractions 0.10 doubles load time #889);
• 4-d ranges (CPU backend: a 4-dimensional ndrange makes kernels up to 100× slower #913);
• state across barriers (CPU backend: locals that live across a @synchronize() in a loop make a kernel 14–78× slower #893).

Each benchmark in the new suite isolates one mechanism. There are 50, in three categories:

• latency:
  • Julia start-up, and precompiling and loading KernelAbstractions;
  • the first kernel with a cold and a warm PoCL cache;
  • compiling a new kernel (simple, local memory + barrier, @groupreduce) and launching at a new workgroup size;
  • launch overhead: default and partial workgroups, many arguments, a collective.
• performance:
  • streaming in cache and DRAM;
  • 3-d and 4-d indexing;
  • narrow workgroups (CPU backend: work-item loops with fewer than 16 iterations are not vectorized (runtime SCEV checks + tiny trip count) #868);
  • default vs. explicit workgroup size (CPU backend: the default workgroup size is the device maximum (4096), so launches of up to 4096 work-items run on one thread #859);
  • @Const, exp/sin, contraction vs. muladd (POCL: contract fast-math flags lost through SPIR-V, no FMAs emitted #865), with serial Julia loops for reference;
  • tiled and naive transposes, and private state across barriers;
  • native and fallback @groupreduce, @groupscan, a histogram with atomics;
  • zeros and copyto!.
• applications: three DGSEM volume integrals (moved from benchmark/dgsem.jl), and a 3-d stencil time-stepping loop.

How it measures:

• Fresh processes: each group runs in a fresh process with one Julia thread, and one PoCL worker (serial) or one per physical core (parallel).
• No tuning: fixed sample counts instead of tune!.
• Latency: every sample is its own process, with an empty PoCL kernel cache.
• Checks: every kernel's result is checked before it's measured. Launches that don't allocate today are required to stay that way, because the history compares values by their ratio and can't flag a change from zero. A run that fails a check writes no results, so nothing gets published.

It runs on Buildkite, on gpuci1 (i7-9700K, the CUDA benchmark machine), with Julia 1.13, for commits on main and non-draft PRs ([only benchmarks]/[skip benchmarks]). A run takes about 10.5 minutes there. Three consecutive runs with nothing else on the machine took 623, 624 and 622 s. Across those runs, the minimum of 33 of the 50 rows varied by less than 3%, and every latency row except the launches by less than 2%. A few parallel kernels vary much more: dgsem/fluxdiff-node by 45%, and indexing/linear3d and localmem/transpose-tiled by 27%. The history learns each benchmark's own noise, so those will only flag larger changes. I dropped three launch benchmarks that varied by 11–17%.

Some numbers from gpuci1, as a baseline:

precompile KernelAbstractions	34 s
using KernelAbstractions	0.50 s
first kernel, cold / warm PoCL cache	2.0 s / 0.36 s
compile a new kernel: simple / local memory / @groupreduce	0.12 s / 0.81 s / 2.0 s
launch	3.8 µs (0 allocations); 18.8 µs and 93 allocations with a collective

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.17%. Comparing base (5dd8a01) to head (b06ff1e).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #932      +/-   ##
==========================================
+ Coverage   83.13%   83.17%   +0.03%     
==========================================
  Files          31       31              
  Lines        2568     2568              
==========================================
+ Hits         2135     2136       +1     
+ Misses        433      432       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.