From 7c9add421944d4b025f8b88c9f8b5d7676cb67ae Mon Sep 17 00:00:00 2001 From: Dana Robinson Date: Sat, 17 Oct 2026 18:10:03 -0700 Subject: [PATCH] Install only the ROCm that CI needs and pull images in the background MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Build, clang-tidy, IWYU, and MinIO Integration workflows ran in the rocm/dev-ubuntu-26.04:10.1.0-full image (and its 24.04 counterpart), which is 8.9 GB compressed, almost all of it in one layer. Pulling it took 2 to 5 minutes of every job, more than building and testing did. These jobs now run in plain ubuntu:26.04 or ubuntu:24.04 images, and a new composite action, .github/actions/install-rocm, installs only what hipObject needs from AMD's package repository (stable.repo.amd.com): amdrocm-runtime-dev10.1 (HIP's headers, CMake package, and runtime) and amdrocm-llvm10.1 (amdclang). That's about 330 MB to download. The action trusts only the repository key with the pinned fingerprint, and sets ROCM_PATH and LD_LIBRARY_PATH for the steps after it. - These packages install ROCm into /opt/rocm/core-10.1 and don't create the /opt/rocm links that the full images have, so the workflows use the version's own directory, as hipobject-nightly.yml does. - Each job installs build-essential before ROCm, so ROCm's compiler packages use the release's libstdc++ (GCC 15 on 26.04, GCC 13 on 24.04). - In the Build workflow, amdclang is now the default compiler for jobs that don't set cc and cxx, so the ROCm version appears in one place. - The four workflows also run when the action changes. - DEBIAN_FRONTEND=noninteractive keeps apt from waiting for answers in the plain images. - CI now picks up 10.1.x patch releases from the repository, where the image was pinned to 10.1.0. AGENTS.md no longer says CI uses the full image. The five emulated-hardware lanes pulled the ernic image (218 MB) in the foreground when they extracted rocm-ernic from it, and the two GPU lanes pulled the rocjitsu image (1.1 GB) in the foreground when they generated ip_discovery.bin. Both pulls now run as a background task alongside the guest image fetch, and a wait step joins it before the first step that uses them. That took 12 to 15 s off each lane, and about 55 s off the GPU lanes, in recent runs. Testing: built against only these packages, bind-mounted over /opt/rocm so the build saw only /opt/rocm/core-10.1. With amdclang: - RelWithDebInfo with the MinIO bridge and HIPOBJ_WERROR built with no warnings and passed 33/33 tests, including the install, package, and subproject tests. - ASan/UBSan, TSan, and coverage builds each passed 32/32, and the coverage report built (96.05% line coverage). I also checked that the pinned key verifies the repository's signed index and that a wrong fingerprint exports no key. actionlint, shellcheck (including the action's script), and codespell are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- .github/actions/install-rocm/action.yml | 108 ++++++++++++++++++ .github/workflows/hipobject-build.yml | 66 +++++------ .github/workflows/hipobject-clang-tidy.yml | 44 ++++--- .../hipobject-hardware-test-gpu-direct.yml | 13 +++ .../hipobject-hardware-test-minio-bridge.yml | 12 ++ .../hipobject-hardware-test-s3-backend.yml | 12 ++ ...object-hardware-test-two-vm-gpu-direct.yml | 13 +++ ...ipobject-hardware-test-two-vm-protocol.yml | 12 ++ .github/workflows/hipobject-iwyu.yml | 39 +++++-- .../hipobject-minio-integration-check.yml | 28 +++-- AGENTS.md | 12 +- 11 files changed, 288 insertions(+), 71 deletions(-) create mode 100644 .github/actions/install-rocm/action.yml diff --git a/.github/actions/install-rocm/action.yml b/.github/actions/install-rocm/action.yml new file mode 100644 index 00000000..6e8929fe --- /dev/null +++ b/.github/actions/install-rocm/action.yml @@ -0,0 +1,108 @@ +--- +name: Install ROCm +description: >- + Install the HIP runtime and amdclang, and nothing else of ROCm, from + AMD's package repository, in an Ubuntu container job. hipObject is + host-only, so it needs no more, and these packages are a ~330 MB + download, where the rocm/dev-ubuntu full images are ~9 GB. + +inputs: + version: + description: >- + ROCm's major and minor version, such as 10.1. Only ROCm 10.0 and + later, which come from stable.repo.amd.com. + required: true + +outputs: + rocm_path: + description: Directory ROCm is installed in, /opt/rocm/core- + value: ${{ steps.install.outputs.rocm_path }} + +runs: + using: composite + steps: + # The packages install ROCm into /opt/rocm/core-. They don't + # make the links in /opt/rocm that the full images have (ROCm's larger + # packages, such as amdrocm-core, make those), so everything uses the + # version's own directory, as hipobject-nightly.yml does. + # + # apt only trusts the key with the known fingerprint, and only for this + # repository. Exporting the key by its fingerprint from a scratch + # keyring leaves out any other key in the download, and exports nothing + # if the key isn't there. + # + # amdrocm-runtime-dev is HIP's headers and CMake package, and brings in + # the runtime. amdrocm-llvm is amdclang, which the runtime already + # needs; it's named because the builds use it. Their recommendations + # (gcc, g++, and their multilib packages) are left out: the job installs + # the compilers it wants. It should do that first, since the compiler + # packages need some release of libstdc++'s development package, and apt + # otherwise picks the oldest that the Ubuntu release has. + # + # The packages don't add ROCm's lib directory to the loader cache, so + # programs linked with libamdhip64, such as the tests, find it through + # LD_LIBRARY_PATH. + - id: install + shell: bash + env: + ROCM_MAJOR_MINOR: ${{ inputs.version }} + # apt takes the packages' defaults rather than asking questions + DEBIAN_FRONTEND: noninteractive + # The fingerprint of AMD's signing key for the repository, which is + # public + AMD_SIGNER_FINGERPRINT: D0F004A0025A1145C7807FCD0701EAC4D5E02107 + run: | + set -euo pipefail + + if [[ ! "${ROCM_MAJOR_MINOR}" =~ ^([0-9]+)\.[0-9]+$ ]] \ + || (( BASH_REMATCH[1] < 10 )); then + echo "::error::version must be 10.0 or later, as major.minor, not '${ROCM_MAJOR_MINOR}'" + exit 1 + fi + + # The repository has a directory for each Ubuntu release, such as + # ubuntu2604 + ubuntu_version=$(awk -F= '$1 == "VERSION_ID" { gsub(/"/, "", $2); print $2 }' \ + /etc/os-release) + repo="https://stable.repo.amd.com/rocm/core/packages/ubuntu${ubuntu_version//./}" + + # gpg imports the key without gpg-agent, but then exits with status 2 + # because it can't start the agent, so the agent is installed too. + apt-get update + apt-get install -y --no-install-recommends \ + ca-certificates curl gpg gpg-agent + + curl -fsSL https://stable.repo.amd.com/rocm/gpg/packages.gpg \ + -o /tmp/amdrocm.asc + GNUPGHOME=$(mktemp -d) + export GNUPGHOME + gpg --batch --import /tmp/amdrocm.asc + install -d -m 0755 /etc/apt/keyrings + gpg --batch --export "${AMD_SIGNER_FINGERPRINT}" \ + > /etc/apt/keyrings/amdrocm.gpg + if [[ ! -s /etc/apt/keyrings/amdrocm.gpg ]]; then + echo "::error::AMD's ROCm repository key isn't ${AMD_SIGNER_FINGERPRINT}" + exit 1 + fi + echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/amdrocm.gpg]" \ + "${repo} stable main" > /etc/apt/sources.list.d/amdrocm.list + + apt-get update + apt-get install -y --no-install-recommends \ + "amdrocm-runtime-dev${ROCM_MAJOR_MINOR}" \ + "amdrocm-llvm${ROCM_MAJOR_MINOR}" + + rocm_path="/opt/rocm/core-${ROCM_MAJOR_MINOR}" + for compiler in amdclang amdclang++; do + if [[ ! -x "${rocm_path}/llvm/bin/${compiler}" ]]; then + echo "::error::${compiler} isn't in ${rocm_path}/llvm/bin" + exit 1 + fi + done + "${rocm_path}/llvm/bin/amdclang++" --version + + { + echo "ROCM_PATH=${rocm_path}" + echo "LD_LIBRARY_PATH=${rocm_path}/lib" + } >> "${GITHUB_ENV}" + echo "rocm_path=${rocm_path}" >> "${GITHUB_OUTPUT}" diff --git a/.github/workflows/hipobject-build.yml b/.github/workflows/hipobject-build.yml index 04de2206..ed75b1c6 100644 --- a/.github/workflows/hipobject-build.yml +++ b/.github/workflows/hipobject-build.yml @@ -9,10 +9,10 @@ on: # used with HIPOBJ_BUILD_DOCS, which is OFF here (the Documentation # Check lane configures and builds those), the .clang-tidy files only # configure the clang-tidy lane, and .github/ holds the other - # workflows and their configuration. This workflow's own file is added - # back last; a later pattern overrides an earlier one, which is why - # this is a paths list with negations and not paths-ignore, which - # can't add a path back. + # workflows and their configuration. This workflow's own file, and the + # action that installs ROCm, are added back last; a later pattern + # overrides an earlier one, which is why this is a paths list with + # negations and not paths-ignore, which can't add a path back. pull_request: branches: [develop] paths: &paths @@ -27,6 +27,7 @@ on: - '!requirements.txt' - '!.github/**' - '.github/workflows/hipobject-build.yml' + - '.github/actions/install-rocm/**' push: branches: [develop] paths: *paths @@ -61,9 +62,10 @@ jobs: # The jobs are listed one by one rather than made by crossing axes, # since most of them change one thing from the first job, and # crossing would build combinations nobody needs. Every job sets - # label, ubuntu, cc, cxx, and build_type. A job that leaves out one - # of the other keys gets its default: + # label, ubuntu, and build_type. A job that leaves out one of the + # other keys gets its default: # + # cc, cxx ROCm's amdclang and amdclang++ # shared_libs ON. TheRock only builds shared libraries. # sanitizer None. asan is HIPOBJ_USE_SANITIZERS (ASan, UBSan, # and the compiler's other checks), and tsan is @@ -74,65 +76,49 @@ jobs: # HIPOBJ_IONIC # packages None. Extra apt packages, such as a compiler. # - # Every job uses ROCm 10.1.0, the latest ROCm (see the image below). + # Every job uses ROCm 10.1, the latest ROCm (see Install ROCm below). matrix: include: # The core jobs build with ROCm's amdclang on the newest Ubuntu # release, as TheRock will build hipObject. - label: amdclang, RelWithDebInfo ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: RelWithDebInfo # Debug, so the library's assert()s are checked with the # sanitizers on, and the reports aren't muddied by optimization - label: amdclang, Debug, ASan and UBSan ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: Debug sanitizer: asan # RelWithDebInfo, as TheRock's ThreadSanitizer builds are, so the # threads' timing is closer to a normal build's - label: amdclang, RelWithDebInfo, TSan ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: RelWithDebInfo sanitizer: tsan # Debug, so the report's lines match the source. Coverage needs # Clang. - label: amdclang, Debug, coverage ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: Debug coverage: ON # Smoke tests of what the core jobs don't build. Release is # TheRock's default build type. - label: amdclang, Release, static ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: Release shared_libs: OFF - label: amdclang, no NIC backends ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: RelWithDebInfo backends: OFF - label: amdclang, C++23 ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: RelWithDebInfo cxx_standard: 23 # C++26 needs CMake 3.30 or later, which Ubuntu 24.04 doesn't have - label: amdclang, C++26 ubuntu: '26.04' - cc: /opt/rocm/llvm/bin/amdclang - cxx: /opt/rocm/llvm/bin/amdclang++ build_type: RelWithDebInfo cxx_standard: 26 # Upstream Clang, the version Ubuntu installs as clang @@ -169,14 +155,15 @@ jobs: cxx: clang++-18 build_type: RelWithDebInfo packages: clang-18 + # A plain Ubuntu image, with only the ROCm that hipObject needs + # installed (see Install ROCm below) container: - image: rocm/dev-ubuntu-${{ matrix.ubuntu }}:10.1.0-full + image: ubuntu:${{ matrix.ubuntu }} options: --user root env: - ROCM_PATH: /opt/rocm - # The full ROCm image ships libamdhip64 in /opt/rocm/lib but leaves it out - # of the loader cache, so ctest binaries fail to start without this. - LD_LIBRARY_PATH: /opt/rocm/lib + # Packages such as tzdata ask questions when they're installed into a + # plain image; this takes their defaults instead. + DEBIAN_FRONTEND: noninteractive # The tests that load the system's libibverbs fail, rather than skip, # if it doesn't load (see test/unit/require-libibverbs.h) HIPOBJ_TEST_REQUIRE_LIBIBVERBS: '1' @@ -185,11 +172,14 @@ jobs: with: persist-credentials: false + # build-essential is the release's GCC (GCC 15 on 26.04, GCC 13 on + # 24.04) and make. amdclang uses GCC's libstdc++ too. + # # rpm provides rpmbuild, so hipObject can build RPM packages here # as well as DEB packages, and the RPM package test runs. # - # The MinIO bridge needs nlohmann-json3-dev and libpugixml-dev. - # libcurlpp-dev saves downloading and building curlpp. + # The MinIO bridge needs nlohmann-json3-dev, libpugixml-dev, and + # zlib1g-dev. libcurlpp-dev saves downloading and building curlpp. # # A job's packages (EXTRA_PACKAGES) are separated by spaces, so read # splits them into an array with one package in each element. @@ -200,6 +190,7 @@ jobs: read -ra extra_packages <<< "${EXTRA_PACKAGES}" apt-get update apt-get install -y \ + build-essential \ cmake \ git \ libcurl4-openssl-dev \ @@ -209,8 +200,16 @@ jobs: libssl-dev \ nlohmann-json3-dev \ rpm \ + zlib1g-dev \ "${extra_packages[@]}" + # After the compilers, so the libstdc++ that ROCm's compiler + # packages need is the one already installed + - name: Install ROCm + uses: ./.github/actions/install-rocm + with: + version: '10.1' + # HIPOBJ_WERROR makes a new warning fail the build. It only # covers hipObject's own targets, not the code FetchContent pulls in. # @@ -219,14 +218,17 @@ jobs: # package tests include it. The coverage report only measures src/, # so the bridge would only slow that job down. # - # CMake picks the compilers up from CC and CXX. + # CMake picks the compilers up from CC and CXX. A job that doesn't + # set them gets amdclang, which is in ROCm's directory. - name: Configure env: CC: ${{ matrix.cc }} CXX: ${{ matrix.cxx }} run: | + export CC="${CC:-${ROCM_PATH}/llvm/bin/amdclang}" + export CXX="${CXX:-${ROCM_PATH}/llvm/bin/amdclang++}" cmake -B build \ - -DCMAKE_PREFIX_PATH=/opt/rocm \ + -DCMAKE_PREFIX_PATH="${ROCM_PATH}" \ -DBUILD_TESTING=ON \ -DHIPOBJ_WERROR=ON \ -DHIPOBJ_BNXT=${{ matrix.backends || 'ON' }} \ diff --git a/.github/workflows/hipobject-clang-tidy.yml b/.github/workflows/hipobject-clang-tidy.yml index b8d829e8..490094cf 100644 --- a/.github/workflows/hipobject-clang-tidy.yml +++ b/.github/workflows/hipobject-clang-tidy.yml @@ -8,7 +8,7 @@ name: hipObject - clang-tidy # This uses upstream clang-tidy rather than ROCm's, which is built # without the clang-analyzer-* checks, as hipFile's clang-tidy workflow # does. The compiler is still ROCm's amdclang++. Keep LLVM_VERSION at -# the LLVM major version of the image's ROCm where apt.llvm.org has a +# the LLVM major version of the job's ROCm where apt.llvm.org has a # release of it, so clang-tidy understands the compiler's flags. ROCm # 10.1 is LLVM 24, which only has snapshot builds, so this uses LLVM 23, # the newest release. That works while the build uses no warning flag @@ -16,7 +16,8 @@ name: hipObject - clang-tidy on: # The same triggers and paths as the Build workflow, since anything that - # it compiles can have findings, plus this workflow's own file. The + # it compiles can have findings, plus this workflow's own file and the + # action that installs ROCm. The # .clang-tidy files are matched by '**'. The paths are a list with # negations and not paths-ignore, so the last entry can add a path back # (see hipobject-build.yml). @@ -33,6 +34,7 @@ on: - '!requirements.txt' - '!.github/**' - '.github/workflows/hipobject-clang-tidy.yml' + - '.github/actions/install-rocm/**' push: branches: [develop] paths: *paths @@ -60,11 +62,15 @@ jobs: clang-tidy: name: clang-tidy runs-on: ubuntu-latest + # A plain Ubuntu image, with only the ROCm that hipObject needs + # installed (see Install ROCm below) container: - image: rocm/dev-ubuntu-26.04:10.1.0-full + image: ubuntu:26.04 options: --user root env: - ROCM_PATH: /opt/rocm + # Packages such as tzdata ask questions when they're installed into a + # plain image; this takes their defaults instead. + DEBIAN_FRONTEND: noninteractive LLVM_VERSION: 23 # The Ubuntu release of the image, for apt.llvm.org UBUNTU_CODENAME: resolute @@ -75,15 +81,16 @@ jobs: with: persist-credentials: false - # Ninja keeps each command's output together, so clang-tidy's - # output for one file isn't interleaved with another's in the - # build log. The MinIO bridge needs nlohmann-json3-dev and - # libpugixml-dev, and libcurlpp-dev saves downloading and building - # curlpp. + # build-essential is GCC, whose libstdc++ amdclang uses. Ninja keeps + # each command's output together, so clang-tidy's output for one + # file isn't interleaved with another's in the build log. The MinIO + # bridge needs nlohmann-json3-dev, libpugixml-dev, and zlib1g-dev, + # and libcurlpp-dev saves downloading and building curlpp. - name: Install dependencies run: | apt-get update apt-get install -y \ + build-essential \ ca-certificates \ cmake \ curl \ @@ -95,7 +102,15 @@ jobs: libpugixml-dev \ libssl-dev \ ninja-build \ - nlohmann-json3-dev + nlohmann-json3-dev \ + zlib1g-dev + + # After the compilers, so the libstdc++ that ROCm's compiler + # packages need is the one already installed + - name: Install ROCm + uses: ./.github/actions/install-rocm + with: + version: '10.1' # apt only trusts the key with the known fingerprint, and only for # this repository. Exporting the key by its fingerprint from a @@ -130,13 +145,14 @@ jobs: # # HIPOBJ_WERROR is on, so a compiler warning fails this job too, as # it does the Build workflow's jobs. + # + # CMake picks the compilers up from CC and CXX. - name: Configure - env: - CC: /opt/rocm/llvm/bin/amdclang - CXX: /opt/rocm/llvm/bin/amdclang++ run: | + export CC="${ROCM_PATH}/llvm/bin/amdclang" + export CXX="${ROCM_PATH}/llvm/bin/amdclang++" cmake -B build -G Ninja \ - -DCMAKE_PREFIX_PATH=/opt/rocm \ + -DCMAKE_PREFIX_PATH="${ROCM_PATH}" \ -DBUILD_TESTING=ON \ -DHIPOBJ_WERROR=ON \ -DHIPOBJ_BNXT=ON \ diff --git a/.github/workflows/hipobject-hardware-test-gpu-direct.yml b/.github/workflows/hipobject-hardware-test-gpu-direct.yml index 6ae86598..dde983db 100644 --- a/.github/workflows/hipobject-hardware-test-gpu-direct.yml +++ b/.github/workflows/hipobject-hardware-test-gpu-direct.yml @@ -292,6 +292,16 @@ jobs: fi EOF + - name: Start pulling the ernic and rocjitsu images + run: | + # In the background, alongside the guest image fetch, so the steps + # that run them don't wait for the pull. "Wait for the images" + # joins it before the first of them. + bash ci/ernic/background-task.sh start images << 'EOF' + docker pull "$ERNIC_IMAGE" + docker pull "$ROCJITSU_IMAGE_GPU" + EOF + - name: Fetch the ionic driver sources id: ionic_sources uses: $/.github/actions/fetch-ionic-sources @@ -333,6 +343,9 @@ jobs: - name: Wait for deps run: bash ci/ernic/background-task.sh wait deps + - name: Wait for the images + run: bash ci/ernic/background-task.sh wait images + - name: Extract rocm-ernic binary from ernic image run: | docker create --name ernic-tmp-gpu "$ERNIC_IMAGE" diff --git a/.github/workflows/hipobject-hardware-test-minio-bridge.yml b/.github/workflows/hipobject-hardware-test-minio-bridge.yml index 5ab20563..6855a038 100644 --- a/.github/workflows/hipobject-hardware-test-minio-bridge.yml +++ b/.github/workflows/hipobject-hardware-test-minio-bridge.yml @@ -242,6 +242,15 @@ jobs: fi EOF + - name: Start pulling the ernic image + run: | + # In the background, alongside the guest image fetch, so the step + # that extracts rocm-ernic from it doesn't wait for the pull. "Wait + # for the image" joins it before then. + bash ci/ernic/background-task.sh start images << 'EOF' + docker pull "$ERNIC_IMAGE" + EOF + - name: Fetch the ionic driver sources id: ionic_sources uses: $/.github/actions/fetch-ionic-sources @@ -284,6 +293,9 @@ jobs: - name: Wait for deps run: bash ci/ernic/background-task.sh wait deps + - name: Wait for the image + run: bash ci/ernic/background-task.sh wait images + - name: Extract rocm-ernic binary from ernic image run: | docker create --name ernic-tmp-2vm "$ERNIC_IMAGE" diff --git a/.github/workflows/hipobject-hardware-test-s3-backend.yml b/.github/workflows/hipobject-hardware-test-s3-backend.yml index 6505c2f5..989c99c1 100644 --- a/.github/workflows/hipobject-hardware-test-s3-backend.yml +++ b/.github/workflows/hipobject-hardware-test-s3-backend.yml @@ -212,6 +212,15 @@ jobs: fi EOF + - name: Start pulling the ernic image + run: | + # In the background, alongside the guest image fetch, so the step + # that extracts rocm-ernic from it doesn't wait for the pull. "Wait + # for the image" joins it before then. + bash ci/ernic/background-task.sh start images << 'EOF' + docker pull "$ERNIC_IMAGE" + EOF + - name: Fetch the ionic driver sources id: ionic_sources uses: $/.github/actions/fetch-ionic-sources @@ -253,6 +262,9 @@ jobs: - name: Wait for deps run: bash ci/ernic/background-task.sh wait deps + - name: Wait for the image + run: bash ci/ernic/background-task.sh wait images + - name: Extract rocm-ernic binary from ernic image run: | docker create --name ernic-tmp "$ERNIC_IMAGE" diff --git a/.github/workflows/hipobject-hardware-test-two-vm-gpu-direct.yml b/.github/workflows/hipobject-hardware-test-two-vm-gpu-direct.yml index bc0c1975..b36d0039 100644 --- a/.github/workflows/hipobject-hardware-test-two-vm-gpu-direct.yml +++ b/.github/workflows/hipobject-hardware-test-two-vm-gpu-direct.yml @@ -270,6 +270,16 @@ jobs: fi EOF + - name: Start pulling the ernic and rocjitsu images + run: | + # In the background, alongside the guest image fetch, so the steps + # that run them don't wait for the pull. "Wait for the images" + # joins it before the first of them. + bash ci/ernic/background-task.sh start images << 'EOF' + docker pull "$ERNIC_IMAGE" + docker pull "$ROCJITSU_IMAGE_GPU" + EOF + - name: Fetch the ionic driver sources id: ionic_sources uses: $/.github/actions/fetch-ionic-sources @@ -312,6 +322,9 @@ jobs: - name: Wait for deps run: bash ci/ernic/background-task.sh wait deps + - name: Wait for the images + run: bash ci/ernic/background-task.sh wait images + - name: Extract rocm-ernic binary from ernic image run: | docker create --name ernic-tmp-2vm "$ERNIC_IMAGE" diff --git a/.github/workflows/hipobject-hardware-test-two-vm-protocol.yml b/.github/workflows/hipobject-hardware-test-two-vm-protocol.yml index b76c2bfe..cdb99acf 100644 --- a/.github/workflows/hipobject-hardware-test-two-vm-protocol.yml +++ b/.github/workflows/hipobject-hardware-test-two-vm-protocol.yml @@ -238,6 +238,15 @@ jobs: fi EOF + - name: Start pulling the ernic image + run: | + # In the background, alongside the guest image fetch, so the step + # that extracts rocm-ernic from it doesn't wait for the pull. "Wait + # for the image" joins it before then. + bash ci/ernic/background-task.sh start images << 'EOF' + docker pull "$ERNIC_IMAGE" + EOF + - name: Fetch the ionic driver sources id: ionic_sources uses: $/.github/actions/fetch-ionic-sources @@ -280,6 +289,9 @@ jobs: - name: Wait for deps run: bash ci/ernic/background-task.sh wait deps + - name: Wait for the image + run: bash ci/ernic/background-task.sh wait images + - name: Extract rocm-ernic binary from ernic image run: | docker create --name ernic-tmp-2vm "$ERNIC_IMAGE" diff --git a/.github/workflows/hipobject-iwyu.yml b/.github/workflows/hipobject-iwyu.yml index 1709d4d4..d15a905e 100644 --- a/.github/workflows/hipobject-iwyu.yml +++ b/.github/workflows/hipobject-iwyu.yml @@ -11,7 +11,8 @@ name: hipObject - IWYU on: # The same triggers and paths as the Build workflow, since anything that - # it compiles can break IWYU, plus this workflow's own file and scripts. + # it compiles can break IWYU, plus this workflow's own file and scripts, + # and the action that installs ROCm. # The paths are a list with negations and not paths-ignore, so the last # entries can add paths back (see hipobject-build.yml). pull_request: @@ -29,6 +30,7 @@ on: - '!.github/**' - 'ci/iwyu/**' - '.github/workflows/hipobject-iwyu.yml' + - '.github/actions/install-rocm/**' push: branches: [develop] paths: *paths @@ -56,24 +58,30 @@ jobs: iwyu: name: include-what-you-use (advisory) runs-on: ubuntu-latest + # A plain Ubuntu image, with only the ROCm that hipObject needs + # installed (see Install ROCm below) container: - image: rocm/dev-ubuntu-26.04:10.1.0-full + image: ubuntu:26.04 options: --user root env: - ROCM_PATH: /opt/rocm + # Packages such as tzdata ask questions when they're installed into a + # plain image; this takes their defaults instead. + DEBIAN_FRONTEND: noninteractive steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - # Ninja keeps each command's output together, so IWYU's output for - # one file isn't interleaved with another's in the build log. The - # MinIO bridge needs nlohmann-json3-dev and libpugixml-dev, and + # build-essential is GCC, whose libstdc++ amdclang uses. Ninja keeps + # each command's output together, so IWYU's output for one file isn't + # interleaved with another's in the build log. The MinIO bridge needs + # nlohmann-json3-dev, libpugixml-dev, and zlib1g-dev, and # libcurlpp-dev saves downloading and building curlpp. - name: Install dependencies run: | apt-get update apt-get install -y \ + build-essential \ cmake \ git \ iwyu \ @@ -83,7 +91,15 @@ jobs: libpugixml-dev \ libssl-dev \ ninja-build \ - nlohmann-json3-dev + nlohmann-json3-dev \ + zlib1g-dev + + # After the compilers, so the libstdc++ that ROCm's compiler + # packages need is the one already installed + - name: Install ROCm + uses: ./.github/actions/install-rocm + with: + version: '10.1' # CMake hides IWYU's failures (see ci/iwyu/iwyu-wrapper.sh), so it # runs IWYU through a wrapper that records every run for the check @@ -95,13 +111,14 @@ jobs: # # HIPOBJ_WERROR is off: the Build workflow checks for warnings, and # this job is only about IWYU. + # + # CMake picks the compilers up from CC and CXX. - name: Configure - env: - CC: /opt/rocm/llvm/bin/amdclang - CXX: /opt/rocm/llvm/bin/amdclang++ run: | + export CC="${ROCM_PATH}/llvm/bin/amdclang" + export CXX="${ROCM_PATH}/llvm/bin/amdclang++" cmake -B build -G Ninja \ - -DCMAKE_PREFIX_PATH=/opt/rocm \ + -DCMAKE_PREFIX_PATH="${ROCM_PATH}" \ -DBUILD_TESTING=ON \ -DHIPOBJ_BNXT=ON \ -DHIPOBJ_IONIC=ON \ diff --git a/.github/workflows/hipobject-minio-integration-check.yml b/.github/workflows/hipobject-minio-integration-check.yml index fce99667..22b94fee 100644 --- a/.github/workflows/hipobject-minio-integration-check.yml +++ b/.github/workflows/hipobject-minio-integration-check.yml @@ -16,6 +16,7 @@ on: - "shared/**" - "src/**" - ".github/workflows/hipobject-minio-integration-check.yml" + - ".github/actions/install-rocm/**" - "!cmake/HIPOBJClangTidy.cmake" - "!**/.clang-tidy" - "!cmake/HIPOBJDocumentation.cmake" @@ -65,24 +66,27 @@ jobs: name: MinIO Integration Check runs-on: ubuntu-24.04 timeout-minutes: 30 + # A plain Ubuntu image, with only the ROCm that hipObject needs + # installed (see Install ROCm below) container: - image: rocm/dev-ubuntu-26.04:10.1.0-full + image: ubuntu:26.04 options: --user root env: - ROCM_PATH: /opt/rocm - # The full ROCm image ships libamdhip64 in /opt/rocm/lib but leaves it out - # of the loader cache, so ctest binaries fail to start without this. - LD_LIBRARY_PATH: /opt/rocm/lib + # Packages such as tzdata ask questions when they're installed into a + # plain image; this takes their defaults instead. + DEBIAN_FRONTEND: noninteractive steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false + # build-essential is GCC, whose libstdc++ amdclang uses, and make - name: Install build dependencies run: | apt-get update apt-get install -y \ + build-essential \ cmake \ curl \ git \ @@ -90,6 +94,13 @@ jobs: libssl-dev \ libibverbs-dev + # After the compilers, so the libstdc++ that ROCm's compiler + # packages need is the one already installed + - name: Install ROCm + uses: ./.github/actions/install-rocm + with: + version: '10.1' + - name: Install mc (MinIO client) shell: bash run: | @@ -139,12 +150,11 @@ jobs: # CMake picks the compilers up from CC and CXX - name: Build hipObject with curl support - env: - CC: /opt/rocm/llvm/bin/amdclang - CXX: /opt/rocm/llvm/bin/amdclang++ run: | + export CC="${ROCM_PATH}/llvm/bin/amdclang" + export CXX="${ROCM_PATH}/llvm/bin/amdclang++" cmake -B build \ - -DCMAKE_PREFIX_PATH=/opt/rocm \ + -DCMAKE_PREFIX_PATH="${ROCM_PATH}" \ -DBUILD_TESTING=ON \ -DHIPOBJ_BNXT=ON \ -DHIPOBJ_IONIC=ON diff --git a/AGENTS.md b/AGENTS.md index 7822114c..a41c12de 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,9 +52,9 @@ one or edit the other project without asking. 7.11+ layout) and to `/opt/rocm` otherwise; setting `ROCM_VERSION` changes this (see [docs/building.rst](docs/building.rst)). Put `/opt/rocm/bin` on `PATH` and include `/opt/rocm/lib` in `LD_LIBRARY_PATH` when running binaries - you build. The `rocm/dev-ubuntu` full images that CI uses leave - `libamdhip64` out of the loader cache, so test binaries won't start there - without it. + you build. ROCm's packages, and the `rocm/dev-ubuntu` images built from + them, leave `libamdhip64` out of the loader cache, so test binaries won't + start without it. - **Compiler**: hipObject is host-only C/C++ (no GPU kernels), so any C++20 compiler works, though changes have to be checked with both GCC and Clang (see [C and C++](#c-and-c)). The HIP runtime headers and library come from @@ -84,8 +84,10 @@ Notes: ([hipobject-build.yml](.github/workflows/hipobject-build.yml)) focuses on the latest ROCm and its amdclang, which is how TheRock will build hipObject. Every job uses ROCm 10.1.0, with `HIPOBJ_WERROR` on. Its core jobs build - this configuration with amdclang on Ubuntu 26.04 (the - `rocm/dev-ubuntu-26.04:10.1.0-full` image): `RelWithDebInfo`, then Debug + this configuration with amdclang on Ubuntu 26.04 (the `ubuntu:26.04` + image, with only ROCm's HIP runtime and amdclang packages installed, by + [`install-rocm`](.github/actions/install-rocm/action.yml)): + `RelWithDebInfo`, then Debug with `-DHIPOBJ_USE_SANITIZERS=ON` (every sanitizer except ThreadSanitizer), `RelWithDebInfo` with `-DHIPOBJ_USE_THREAD_SANITIZER=ON`, and Debug with `-DHIPOBJ_USE_CODE_COVERAGE=ON`. Single jobs smoke-test the rest: with