T31: oneAPI and Metal extensions

.github/workflows/ci.yml

@@ -72,11 +72,12 @@ jobs:
         os: [ubuntu-latest]
         julia-version: ['1']
         julia-arch: [x64]
-        # Both legs must pass: the AMDGPU extension (T23) runs the suite on
-        # ROCBackend with rocBLAS/rocSOLVER (test/backends.jl adds it when
-        # AMDGPU is functional). Making the amdgpu check required is a branch
-        # ruleset setting of the owner.
-        backend: ['cuda', 'amdgpu']
+        # Every leg must pass: the AMDGPU extension (T23) runs the suite on
+        # ROCBackend with rocBLAS/rocSOLVER, the oneAPI extension (T31) on
+        # oneAPIBackend with oneMKL (the runner `headroom`); test/backends.jl adds
+        # each backend when its package is functional. Making a check required is
+        # a branch ruleset setting of the owner.
+        backend: ['cuda', 'amdgpu', 'oneapi']
     runs-on: [self-hosted, linux, X64, gpu, '${{ matrix.backend }}']
 
     steps:
@@ -95,6 +96,11 @@ jobs:
         if: matrix.backend == 'amdgpu'
         run: |
           julia --project=test -e 'using Pkg; Pkg.add("AMDGPU")'
+      - name: Add oneAPI.jl to the test environment
+        if: matrix.backend == 'oneapi'
+        run: |
+          julia --project=test -e 'using Pkg; Pkg.add("oneAPI")'
+          julia --project=test -e 'using oneAPI; oneAPI.versioninfo()'
       - uses: julia-actions/julia-runtest@latest
       - uses: julia-actions/julia-processcoverage@v1
       - uses: codecov/codecov-action@v4

AGENTS.md

@@ -43,9 +43,10 @@ Three documents define the work. Read them in this order before touching code:
   mechanically.
 * New dependencies need a one-line justification in the Report. Never `dev`
   unregistered or local checkouts into the project (the package requires
-  KernelAbstractions 0.10 and GPUArrays 12, hence CUDA.jl >= 6.5 and
-  AMDGPU.jl >= 2.9; the test allocation budgets carry a per-launch allowance
-  for the 0.10 CPU backend -- see `ka_cpu_alloc_budget`).
+  KernelAbstractions 0.10 and GPUArrays 12, hence CUDA.jl >= 6.5, AMDGPU.jl
+  >= 2.10, Metal.jl >= 1.11 and oneAPI.jl >= 2.10; the test allocation budgets
+  carry a per-launch allowance for the 0.10 CPU backend -- see
+  `ka_cpu_alloc_budget` and #145).
 * The repository is `github.com/exanauts/SparseDirectSolver.jl` (remote
   `origin`, branch `main`). Commit at the end of the task with a message
   starting with the task id, e.g. `T05: symbolic pattern, ordering, etree,
@@ -96,7 +97,8 @@ reviewing agent check; the owner merges by hand. The moving parts:
   `ci.yml` on the PR, and from the `amdgpu` leg of `ci.yml` on the
   self-hosted AMD runner (the AMDGPU extension, T23; no longer
   continue-on-error, a required check once the owner adds it to the branch
-  ruleset).
+  ruleset), and from the `oneapi` leg on the self-hosted Intel GPU runner
+  `headroom` (the oneAPI extension, T31; oneAPI.jl release, compat `2.10`).
 
 ### The owner's machine
 
@@ -108,8 +110,8 @@ reviewing agent check; the owner merges by hand. The moving parts:
   KernelAbstractions CPU backend and on CUDA only. GitHub CI
   (`.github/workflows/ci.yml`, modelled on `../ExaPF.jl`) runs the CPU suite and
   the GPU suite on the exanauts self-hosted runners (CPU suite: the `kkt`
-  machine; GPU suite: the runners labelled `cuda` and `amdgpu`). oneAPI and Metal extension files are written by analogy
-  (task T23) and only precompile-checked.
+  machine; GPU suite: the runners labelled `cuda`, `amdgpu` and `oneapi`; Metal has no CI
+  leg: the hosted `macos-latest` runner could not run the Metal suite).
 * CUDA.jl 6.x is split into packages: `CUDACore` (`CuArray`, `CUDABackend`),
   `cuSPARSE` (`CuSparseMatrixCSR`, `CuSparseMatrixCSC`), `cuBLAS`, `cuSOLVER`.
   Use these as weak dependencies; tests load the umbrella `CUDA`.

Project.toml

@@ -1,6 +1,6 @@
 name = "SparseDirectSolver"
 uuid = "dc8e0fd9-41f1-440b-9624-22817132f700"
-version = "0.1.0"
+version = "0.2.0"
 authors = ["Michel Schanen <Michel.Schanen@gmail.com>"]
 
 [deps]
@@ -20,6 +20,8 @@ AMDGPU = "21141c5a-9bdb-4563-92ae-f87d6854732e"
 CUDACore = "bd0ed864-bdfe-4181-a5ed-ce625a5fdea2"
 Krylov = "ba0b0d4f-ebba-5204-a429-3ac8c609bfb7"
 MadNLP = "2621e9c9-9eb4-46b1-8089-e8c72242dfb6"
+Metal = "dde4c033-4e86-420c-a63e-0dd931031962"
+oneAPI = "8f75cd03-7ff8-4ecb-9b8f-daf728133b1b"
 cuBLAS = "182d3088-87b7-4494-8cad-fc6afaa545bc"
 cuSOLVER = "887afef0-6a32-4de5-add4-7827692ba8fc"
 cuSPARSE = "b26da814-b3bc-49ef-b0ee-c816305aa060"
@@ -29,10 +31,12 @@ SparseDirectSolverAMDGPUExt = "AMDGPU"
 SparseDirectSolverCUDAExt = ["CUDACore", "cuBLAS", "cuSOLVER", "cuSPARSE"]
 SparseDirectSolverKrylovExt = "Krylov"
 SparseDirectSolverMadNLPExt = "MadNLP"
+SparseDirectSolverMetalExt = "Metal"
+SparseDirectSolverOneAPIExt = "oneAPI"
 
 [compat]
 AMD = "0.5"
-AMDGPU = "2.9"
+AMDGPU = "2.10"
 Adapt = "4"
 Atomix = "1"
 CUDACore = "6.5"
@@ -43,9 +47,11 @@ KernelAbstractions = "0.10"
 Krylov = "0.10"
 LinearAlgebra = "1.13"
 MadNLP = "0.11"
+Metal = "1.11"
 Metis = "1"
 SparseArrays = "1.13"
 cuBLAS = "6.5"
 cuSOLVER = "6.5"
 cuSPARSE = "6.5"
 julia = "1.13"
+oneAPI = "2.10"

README.md

@@ -9,8 +9,9 @@ and phases of [CUDSS.jl](https://github.com/exanauts/CUDSS.jl) so that MadNLP
 and other cuDSS users can switch to it mechanically, and targets CUDA, AMDGPU,
 oneAPI, Metal and the KernelAbstractions CPU backend.
 
-**Status: v0.1 under construction.** Working today, on the CPU backend, on
-CUDA and on AMDGPU (ROCm):
+**Status: v0.1 under construction.** Working today on the CPU backend, CUDA,
+AMDGPU (ROCm), Metal (`Float32`, tested on an M4 Max) and oneAPI (tested on an
+Intel Data Center GPU Max 1550):
 
 * host symbolic analysis: AMD or nested dissection (METIS) ordering,
   elimination tree, supernodes with GPU-tuned
@@ -60,8 +61,7 @@ CUDA and on AMDGPU (ROCm):
   through `SDS_LOG_LEVEL`.
 
 Not there yet: non-uniform batches, ND partition-tree export, mixed precision,
-hybrid host memory, delayed pivots, and the oneAPI and Metal
-extensions. Unsupported structures, phases and parameters raise
+hybrid host memory, and delayed pivots. Unsupported structures, phases and parameters raise
 `NotSupportedError` rather than falling back silently. The remaining gap to
 cuDSS is performance, not features (see below and
 [`PERFORMANCE.md`](PERFORMANCE.md)); a MadNLP end-to-end run on CUDA and AMD
@@ -108,7 +108,9 @@ Pkg.add(url = "https://github.com/exanauts/SparseDirectSolver.jl")
 Loading CUDA.jl enables the CUDA extension (`CuSparseMatrixCSR`/`CuSparseMatrixCSC`
 constructors, cuBLAS/cuSOLVER dense kernels); loading AMDGPU.jl enables the
 AMDGPU extension (`ROCSparseMatrixCSR`/`ROCSparseMatrixCSC` constructors,
-rocBLAS/rocSOLVER dense kernels; ROCm must be found by AMDGPU.jl).
+rocBLAS/rocSOLVER dense kernels; ROCm must be found by AMDGPU.jl); loading
+oneAPI.jl enables the oneAPI extension (`oneSparseMatrixCSR`/`oneSparseMatrixCSC`
+constructors, oneMKL BLAS kernels; needs oneAPI.jl with KernelAbstractions 0.10).
 
 ## Usage