Skip to content

Only test Julia 1.10 and 1.13 on pull requests in CI - #936

Merged
maleadt merged 2 commits into
mainfrom
tb/ci-trim
Oct 27, 2026
Merged

maleadt merged 2 commits into
mainfrom
tb/ci-trim

Conversation

@maleadt

@maleadt maleadt commented Oct 27, 2026

Copy link
Copy Markdown
Member

Every pull request push runs 20 GPU jobs on Buildkite (5 back ends × Julia 1.10–1.13) and 40 jobs on GitHub Actions (5 platforms × 5 Julia versions for KernelAbstractions, 3 × 5 for KernelInterface). This PR trims that down, like CUDA.jl did recently.

Over the last ~3.5 weeks:

  • Julia 1.11 and 1.12 took 167 of the ~300 GPU agent-hours spent on pull requests. None of their failures that didn't also show up on 1.10 or 1.13 were real: they were OpenCL.jl install failures (an unregistered SPIRVIntrinsics version) and a lost agent.
  • On GitHub Actions, a pull request push cost about 8.4 runner-hours, 4.7 of them on macOS. Across ~150 runs, 1.11/1.12/nightly failed on their own only once for a real reason (the POCL launch-allocation counts, on 1.11). The rest were BusyBox download failures and unregistered JLLs. Intel macOS, the slowest runner at 22–28 min per job, never failed on its own.

The changes, on pull requests only:

  • Buildkite tests 1.10 and 1.13. The releases in between run on branch builds, or when selected with [only julia] or the name of a back end.
  • GitHub Actions tests 1.10 and 1.13. Intel macOS only runs 1.13, and the standalone KernelInterface tests only run on Linux, since KernelAbstractions' own suite already runs the KernelInterface testsuite on every platform. Pushes to main and release branches keep the full matrix, including nightly.

That cuts a pull request push to 10 GPU jobs and about 1.9 runner-hours on GitHub Actions (0.6 on macOS).

The second commit fixes the formatting check's push trigger, which listed master and release-, so it never ran on main.

Julia 1.11, 1.12 and nightly took about 55% of the GPU agent time on pull
requests over the last weeks, and on GitHub Actions most of the 8.4
runner-hours a pull request push costs, while only failing on their own
once (an allocation count in a POCL test). Test them on branch builds
instead, like CUDA.jl does. Pull requests also only use the slow Intel
macOS runners for 1.13, and only test KernelInterface standalone on Linux,
since KernelAbstractions' own tests already run its testsuite everywhere.

On Buildkite, `[only julia]` or naming a back end still selects the
intermediate releases on a pull request.
The push trigger listed `master` and `release-` (without wildcard), which
match no branch here, so formatting was only checked on pull requests.
@github-actions

Copy link
Copy Markdown
Contributor

Benchmark results

Compared with d2ada37c6, the latest measured ancestor of the merge base (1 commit before it).
The environment differs from the baseline's: Parsers 3.0.1 → 3.0.2.

Times are the minimum time of every benchmark. A change is unusual when it exceeds both three times the benchmark's typical change between consecutive runs on the default branch and a minimum per category (performance 3%, latency 5%, applications 3%, other 3%). A benchmark that starts or stops allocating is listed however few the allocations. A single run can still be a fluke, so rerun before drawing conclusions.

Geometric mean Unusual: slower Unusual: faster Compared
Performance −0.6% 0 0 32
Latency +0.0% 0 0 13
Applications +10.0% 0 0 5

No unusual changes.

All benchmarks
Benchmark Baseline This PR Change
Performance
julia/serial/arithmetic/muladd-reference 655 µs 655 µs −0.0% noise unknown
julia/serial/math/exp-f64-reference 284 µs 284 µs −0.0% noise unknown
julia/serial/narrow/masked-reference 702 µs 710 µs +1.2% noise unknown
pocl/parallel/atomics/histogram 5.16 ms 5.17 ms +0.3% noise unknown
pocl/parallel/groupreduce/add-f64 1.66 ms 1.66 ms −0.0% noise unknown
pocl/parallel/groupreduce/fallback 18.1 ms 18.1 ms −0.1% noise unknown
pocl/parallel/groupscan/add-i32 2.53 ms 2.53 ms +0.0% noise unknown
pocl/parallel/indexing/cartesian3d 84.8 µs 80.6 µs −5.0% noise unknown
pocl/parallel/indexing/linear3d 84.4 µs 78.9 µs −6.5% noise unknown
pocl/parallel/indexing/ntuple3d 81.4 µs 81.7 µs +0.4% noise unknown
pocl/parallel/indexing/ntuple4d 164 µs 161 µs −1.7% noise unknown
pocl/parallel/localmem/transpose-naive 459 µs 454 µs −1.2% noise unknown
pocl/parallel/localmem/transpose-tiled 101 µs 105 µs +3.4% noise unknown
pocl/parallel/private/across-barrier 164 µs 164 µs +0.3% noise unknown
pocl/parallel/scheduling/default 181 µs 181 µs −0.0% noise unknown
pocl/parallel/scheduling/explicit 181 µs 181 µs −0.1% noise unknown
pocl/parallel/stream/copy/cache 6.06 µs 6.14 µs +1.3% noise unknown
pocl/parallel/stream/copy/dram 6.32 ms 6.06 ms −4.1% noise unknown
pocl/parallel/stream/triad/cache 6.90 µs 6.82 µs −1.2% noise unknown
pocl/parallel/stream/triad/dram 8.94 ms 9.26 ms +3.6% noise unknown
pocl/parallel/util/copyto-small 4.59 µs 4.66 µs +1.7% noise unknown
pocl/parallel/util/zeros-large 998 µs 1.00 ms +0.4% noise unknown
pocl/parallel/util/zeros-small 665 ns 649 ns −2.4% noise unknown
pocl/serial/arithmetic/contract 1.91 ms 1.90 ms −0.4% noise unknown
pocl/serial/arithmetic/muladd 759 µs 767 µs +1.1% noise unknown
pocl/serial/const/marked 728 µs 736 µs +1.1% noise unknown
pocl/serial/const/unmarked 915 µs 910 µs −0.6% noise unknown
pocl/serial/math/exp-f64 147 µs 146 µs −0.3% noise unknown
pocl/serial/math/sin-f64 202 µs 194 µs −4.3% noise unknown
pocl/serial/narrow/masked 854 µs 791 µs −7.3% noise unknown
pocl/serial/narrow/unmasked 765 µs 777 µs +1.6% noise unknown
pocl/serial/stream/triad/dram 8.63 ms 8.59 ms −0.5% noise unknown
Latency
host/import/ka 506 ms 508 ms +0.4% noise unknown
host/precompile/ka 44.9 s 44.6 s −0.6% noise unknown
host/startup 133 ms 131 ms −2.1% noise unknown
pocl/parallel/launch/dynamic-default 4.39 µs 4.47 µs +1.8% noise unknown
pocl/parallel/launch/groupreduce-f64 18.5 µs 18.5 µs −0.1% noise unknown
pocl/parallel/launch/many-args 12.0 µs 12.4 µs +3.4% noise unknown
pocl/parallel/launch/partial-workgroup 4.18 µs 3.93 µs −6.0% noise unknown
pocl/serial/compile/groupreduce 2.06 s 2.07 s +0.4% noise unknown
pocl/serial/compile/localmem-barrier 808 ms 803 ms −0.7% noise unknown
pocl/serial/compile/new-kernel 118 ms 118 ms +0.2% noise unknown
pocl/serial/compile/new-workgroupsize 13.6 ms 14.1 ms +4.2% noise unknown
pocl/serial/first-kernel/cold-cache 2.02 s 2.02 s +0.1% noise unknown
pocl/serial/first-kernel/warm-cache 360 ms 359 ms −0.1% noise unknown
Applications
pocl/parallel/dgsem/fluxdiff-node 291 µs 336 µs +15.5% noise unknown
pocl/parallel/dgsem/weak-element 160 µs 217 µs +36.0% noise unknown
pocl/parallel/dgsem/weak-node-localmem 118 µs 118 µs −0.1% noise unknown
pocl/parallel/stencil/timestep 7.91 ms 8.08 ms +2.2% noise unknown
pocl/serial/stencil/timestep 11.3 ms 11.3 ms +0.3% noise unknown

History: https://juliagpu.org/perf/?pkg=kernelabstractions

@maleadt
maleadt merged commit b3daae8 into main Oct 27, 2026
19 of 20 checks passed
@maleadt
maleadt deleted the tb/ci-trim branch October 27, 2026 20:05
@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.47%. Comparing base (d2ada37) to head (5131288).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #936      +/-   ##
==========================================
- Coverage   82.87%   82.47%   -0.41%     
==========================================
  Files          31       31              
  Lines        2564     2522      -42     
==========================================
- Hits         2125     2080      -45     
- Misses        439      442       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant