From 6fc6b0968833ca1dd1fe5e37c98ba2c5701a99b1 Mon Sep 17 00:00:00 2001 From: hujc Date: Mon, 12 Oct 2026 19:46:42 -0700 Subject: [PATCH 1/3] [CI] Run ARM CI through the x86 test action and re-enable ARM tests (#8233) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ARM CI restores compatible Warp and OVRTX caches and runs Newton/OVPhysX marker coverage through the shared x86 test action. Camera smoke coverage stays enabled, and previously skipped ARM checks are restored. 1. Route ARM marker tests through `run-package-tests`, with explicit `linux/arm64` passed through image lookup, dependency hashing and fallback builds. PRs restore ARM caches; protected-branch pushes publish them. Test files run in four slots like the x86 test jobs; rendering files still run one at a time. 2. Run the existing state-training, camera-rendering and camera-training Cartpole probes as their own ARM step after the marker tests, reusing the restored Warp and OVRTX cache directories that `run-package-tests` now exposes as outputs. Installation CI now covers state training separately, so the camera probes remain necessary. The step checks the probe report explicitly, since `run-tests` records failures in the report rather than its exit code. 3. Restore shadow-hand kitless rendering and operational-space controller coverage on ARM. Carry the Isaac Sim PhysX training re-enable into the replacement uv workspace test rather than restoring a deleted installation test, and preserve the required ARM system OpenMP preload when the probe adds the Isaac Sim shim. Remove the Kuka visual-material ARM marker to match x86 rendering selection. Supersedes #8230. 4. Build the image in the runner's Docker daemon when no registry cache is configured, writing new layers with zstd. The disposable `docker-container` builder exported and reloaded the ~29 GB image every build and discarded its layer cache. Only ARM takes this path today: x86 callers resolve ECR and keep the `docker-container` builder. The action is part of the dependency-cache hash, so the first build after merge misses that cache once. 5. Count kitless rendering files that use `rendering_test_utils` as renderers, so their cold shader compiles no longer run concurrently. 1. `uv run --frozen --extra test isaaclab -f`: all checks passed. 2. Docker-storage and file-scheduler coverage: 19 tests passed. Actionlint passed for ARM and x86 workflows; 50 local action calls validated against their declared inputs. 3. A real subprocess regression check failed before the preload fix and passes afterward for ARM with/without the shim and x86 with the shim. 4. On `7b73dafe1fe`, [ARM CI](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182263) passed 71 marker cases (3 expected skips) and all 3 state/camera smoke probes. [ARM installation](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182231) passed 3/3 attempts on three NVIDIA GB10 runners, with all 8 contracts including PhysX executed each time; x86 installation also passed all 8 contracts. 5. On the final head `01a3ff4737d`, the [x86 Docker suite](https://github.com/isaac-sim/IsaacLab/actions/runs/38312079847) passed: 30 jobs succeeded and 3 post-merge-only jobs were skipped as expected. Hosted pre-commit, Tools Tests, Docs, kitless image validation, installation and multi-GPU CI also passed. 6. ARM job time, all on NVIDIA GB10 with driver 580.159.03. A temporary commit (since reverted) let PR runs save caches for this measurement: | Run | Build | Tests | Total | | --- | ---: | ---: | ---: | | Before, [cold](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182263/job/112609307602) | 8m30 | 16m52 | 25m42 | | [Warm, new runner](https://github.com/isaac-sim/IsaacLab/actions/runs/38298855064) (3 runs) | 3m06–3m23 | 5m22–5m38 | 9m01–9m06 | | Warm, runner with layer cache | 0m56 | 5m42 | 6m57 | | [Final head, new runner](https://github.com/isaac-sim/IsaacLab/actions/runs/38312048044) | 3m31 | 9m22 | 13m11 | The first four rows also included the Newton RTX shader-cache fix, now split into #8504; the final head does not. Warm per-file times: Cartpole 54 s (was 297 s), Kuka 51 s, Shadow Hand 41 s, operational space 118 s alongside them. Newton RTX takes 254 s without #8504 and 27 s with it. Every run passed 71 marker cases (3 expected skips) and 3 smoke probes. Image export on the runners' containerd store fell from 243 s with gzip to 44–46 s with zstd. Protected-branch publication remains unmeasured before merge. 1. Bug fix (non-breaking CI and test coverage changes). - [x] Backport this pull request to the active release branch after it merges into `develop` - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/develop/source/refs/contributing.html) - [x] I have run the `pre-commit` checks with `uv run isaaclab --format` - [x] Documentation describing the smoke-test caller matches the current workflow; rendered documentation is unaffected - [x] No new project warning sites were added; CI still reports existing dependency and action deprecations - [x] I have validated the changed GPU behavior with focused coverage and applied the test-audit criteria to test changes - [x] I have added changelog fragments for the touched source packages - [x] My name already exists in `CONTRIBUTORS.md` (cherry picked from commit 8519738d0c984a2a5bfddc161f60f59606c3b394) --- .github/actions/docker-build/action.yml | 25 +++- .../actions/ecr-build-push-pull/action.yml | 6 + .github/actions/run-package-tests/action.yml | 22 ++- .github/workflows/arm-ci.yml | 59 +++++--- .../jichuanh-arm-ci-align-x86.skip | 0 .../controllers/test_operational_space.py | 4 +- source/isaaclab/test/install_ci/conftest.py | 140 ++++++++++++++++++ .../misc/cartpole_training_smoke.py | 5 +- .../test/install_ci/test_workspace.py | 63 ++++++++ .../jichuanh-arm-ci-align-x86.skip | 0 ...a_visual_material_randomization_kitless.py | 1 - .../test_rendering_shadow_hand_kitless.py | 4 +- tools/conftest.py | 11 +- 13 files changed, 299 insertions(+), 41 deletions(-) create mode 100644 source/isaaclab/changelog.d/jichuanh-arm-ci-align-x86.skip create mode 100644 source/isaaclab/test/install_ci/test_workspace.py create mode 100644 source/isaaclab_tasks/changelog.d/jichuanh-arm-ci-align-x86.skip diff --git a/.github/actions/docker-build/action.yml b/.github/actions/docker-build/action.yml index 038deac54e98..182da3ef608a 100644 --- a/.github/actions/docker-build/action.yml +++ b/.github/actions/docker-build/action.yml @@ -168,13 +168,28 @@ runs: BUILD_ARGS+=( --cache-to "${{ inputs.cache-to }}" ) fi - BUILDER_NAME="docker-build-${{ github.run_id }}-${{ github.job }}" - docker buildx create --use --driver docker-container --name "${BUILDER_NAME}" \ - || docker buildx use "${BUILDER_NAME}" - trap 'docker buildx rm "${BUILDER_NAME}" || true' EXIT + # Registry cache export needs a docker-container builder, whose image must + # then be exported and loaded into the daemon. Callers without a registry + # cache (ARM, which has no ECR) build in the daemon itself instead, so the + # image, layer cache and uv cache mount stay on the runner. A containerd + # image store still compresses new layers; zstd level 1 is several times + # faster than the default gzip for the multi-GB venv layer. + if [ -n "${{ inputs.cache-from }}${{ inputs.cache-to }}" ]; then + BUILDER_NAME="docker-build-${{ github.run_id }}-${{ github.job }}" + docker buildx create --driver docker-container --name "${BUILDER_NAME}" \ + || true + trap 'docker buildx rm "${BUILDER_NAME}" || true' EXIT + BUILD_ARGS+=( --builder "${BUILDER_NAME}" --load ) + else + BUILD_ARGS+=( + --builder default + --provenance=false + --output "type=image,name=${{ inputs.image-tag }},compression=zstd,compression-level=1" + ) + fi echo "🔵 Building ${{ inputs.image-tag }}..." - docker buildx build --load "${BUILD_ARGS[@]}" "${{ inputs.context-path }}" + docker buildx build "${BUILD_ARGS[@]}" "${{ inputs.context-path }}" echo "was-built=true" >> "$GITHUB_OUTPUT" ##### 6: Layer the CI pytest harness onto the built image ##### diff --git a/.github/actions/ecr-build-push-pull/action.yml b/.github/actions/ecr-build-push-pull/action.yml index 6c6b7eb6f069..7ff7a10ded17 100644 --- a/.github/actions/ecr-build-push-pull/action.yml +++ b/.github/actions/ecr-build-push-pull/action.yml @@ -23,6 +23,10 @@ inputs: description: 'Path to Dockerfile, relative to the repository root' default: 'docker/Dockerfile.base' required: false + platform: + description: 'Target platform for the build and its dependency-cache identity' + default: 'linux/amd64' + required: false ecr-url: description: > ECR repository URL (e.g. "123456789.dkr.ecr.us-west-2.amazonaws.com/my-repo"). @@ -208,6 +212,7 @@ runs: dockerfile-path: ${{ inputs.dockerfile-path }} isaacsim-base-image: ${{ inputs.isaacsim-base-image }} isaacsim-version: ${{ inputs.isaacsim-version }} + platform: ${{ inputs.platform }} - name: Check deps cache id: deps-cache @@ -250,6 +255,7 @@ runs: isaacsim-base-image: ${{ inputs.isaacsim-base-image }} isaacsim-version: ${{ inputs.isaacsim-version }} dockerfile-path: ${{ inputs.dockerfile-path }} + platform: ${{ inputs.platform }} cache-from: ${{ steps.resolve-ecr.outputs.available == 'true' && format('type=registry,ref={0}', env.CACHE_IMAGE) || '' }} cache-to: ${{ steps.resolve-ecr.outputs.available == 'true' && format('type=registry,ref={0},mode=max', env.CACHE_IMAGE) || '' }} deps-hash: ${{ steps.deps-hash.outputs.hash }} diff --git a/.github/actions/run-package-tests/action.yml b/.github/actions/run-package-tests/action.yml index e04859354305..bf46296df977 100644 --- a/.github/actions/run-package-tests/action.yml +++ b/.github/actions/run-package-tests/action.yml @@ -27,6 +27,14 @@ inputs: description: 'ECR cache tag' default: 'cache-base' required: false + ci-marker: + description: 'Optional pytest -m marker expression applied to every test file (e.g. arm_ci)' + default: '' + required: false + platform: + description: 'Target platform if the image has to be built' + default: 'linux/amd64' + required: false filter-pattern: description: >- Pattern to filter test files (e.g., isaaclab_tasks); test files whose path @@ -111,14 +119,15 @@ inputs: newest compatible snapshot and never writes; every test job uses this, so a pull request can never publish kernels built from unmerged code. 'save' restores the same way and writes the result back as a new snapshot, making - post-merge a cumulative writeback; only warm-warp-cache uses it. + post-merge a cumulative writeback; only trusted post-merge cache producers + use it. default: '' required: false ovrtx-shader-cache: description: >- OVRTX shader cache mode. Empty disables the cache. 'restore' reads the newest compatible snapshot and never writes. 'save' restores and writes - back a new snapshot; only warm-ovrtx-cache uses it. + back a new snapshot; only trusted post-merge cache producers use it. default: '' required: false ovrtx-shader-cache-trees: @@ -146,6 +155,13 @@ inputs: container-name: description: 'Docker container name prefix (run-id is appended automatically)' required: true +outputs: + warp-cache-host-dir: + description: 'Host directory mounted as the Warp kernel cache; empty when the cache is disabled' + value: ${{ steps.warp-cache-key.outputs.host-dir }} + ovrtx-shader-cache-host-dir: + description: 'Host directory mounted as the OVRTX shader cache; empty when the cache is disabled' + value: ${{ steps.ovrtx-cache.outputs.host-dir }} runs: using: composite steps: @@ -197,6 +213,7 @@ runs: isaacsim-version: ${{ inputs.isaacsim-version }} dockerfile-path: ${{ inputs.dockerfile-path }} cache-tag: ${{ inputs.cache-tag }} + platform: ${{ inputs.platform }} - name: Report pull duration if: always() @@ -348,6 +365,7 @@ runs: include-files: ${{ inputs.include-files }} test-node-ids-file: ${{ inputs.test-node-ids-file }} test-node-ids-key: ${{ inputs.test-node-ids-key }} + ci-marker: ${{ inputs.ci-marker }} volume-mount-source: ${{ github.workspace }} extra-pip-packages: ${{ inputs.extra-pip-packages }} extra-uv-packages: ${{ inputs.extra-uv-packages }} diff --git a/.github/workflows/arm-ci.yml b/.github/workflows/arm-ci.yml index a45a180a4815..ddedccb0b560 100644 --- a/.github/workflows/arm-ci.yml +++ b/.github/workflows/arm-ci.yml @@ -175,40 +175,53 @@ jobs: uses: ./.github/actions/resolve-ov-pins id: ov_pins + # Same action as the x86 test jobs, with four file slots; rendering files still + # run one at a time. With no ECR on the Spark, it picks up the image built above from the local + # store. Pushes to protected branches publish the arm64 Warp and OVRTX caches + # that pull requests restore, the role x86's cache-warm jobs play. - name: Run arm_ci marker tests - uses: ./.github/actions/run-tests + id: tests + uses: ./.github/actions/run-package-tests with: - test-path: tools - result-file: arm-ci-report.xml - container-name: isaac-lab-arm-ci-${{ github.run_id }}-${{ github.run_attempt }} image-tag: ${{ needs.config.outputs.ci_image_tag }}-arm64 - extra-pip-packages: "${{ steps.ov_pins.outputs.ovrtx }} ${{ steps.ov_pins.outputs.ovphysx }}" - # ovstage rendering cases are post-merge only on x86 as well. - test-k-expr: not ovphysx and not ovstage + platform: linux/arm64 + isaacsim-base-image: ${{ needs.config.outputs.isaacsim_image_name }} + isaacsim-version: ${{ needs.config.outputs.isaacsim_image_tag }} ci-marker: arm_ci - volume-mount-source: ${{ github.workspace }} - - - name: Run arm_ci OVPhysX marker tests - uses: ./.github/actions/run-tests - with: - test-path: tools - result-file: arm-ci-ovphysx-report.xml - container-name: isaac-lab-arm-ci-ovphysx-${{ github.run_id }}-${{ github.run_attempt }} - image-tag: ${{ needs.config.outputs.ci_image_tag }}-arm64 + # ovstage rendering cases are post-merge only on x86 as well. + test-k-expr: not ovstage + test-jobs: "4" extra-pip-packages: "${{ steps.ov_pins.outputs.ovrtx }} ${{ steps.ov_pins.outputs.ovphysx }}" - test-k-expr: ovphysx and not ovstage - ci-marker: arm_ci - volume-mount-source: ${{ github.workspace }} - + warp-cache: ${{ github.event_name == 'push' && 'save' || 'restore' }} + ovrtx-shader-cache: ${{ github.event_name == 'push' && 'save' || 'restore' }} + ovrtx-shader-cache-trees: kitless + result-file: arm-ci-report.xml + container-name: isaac-lab-arm-ci + omni-github-test-type: arm-ci - - name: Run shared Cartpole smoke + # The test orchestrator skips install_ci/, so the Cartpole state/camera probes + # run here in the same image, reusing the restored Warp and OVRTX caches. + - name: Run Cartpole smoke tests uses: ./.github/actions/run-tests with: test-path: source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py - result-file: arm-ci-cartpole-smoke-report.xml - container-name: isaac-lab-arm-ci-cartpole-${{ github.run_id }}-${{ github.run_attempt }} + result-file: arm-ci-smoke-report.xml + container-name: isaac-lab-arm-ci-smoke-${{ github.run_id }}-${{ github.run_attempt }} image-tag: ${{ needs.config.outputs.ci_image_tag }}-arm64 volume-mount-source: ${{ github.workspace }} + warp-cache-host-dir: ${{ steps.tests.outputs.warp-cache-host-dir }} + ovrtx-shader-cache-host-dir: ${{ steps.tests.outputs.ovrtx-shader-cache-host-dir }} + omni-github-test-type: arm-ci + + # run-tests records failures in the report rather than its exit code. + - name: Check Cartpole smoke results + shell: bash + run: | + report=reports/arm-ci-smoke-report.xml + if [ ! -f "$report" ] || grep -qE '(failures|errors)="[1-9][0-9]*"' "$report"; then + echo "::error::Cartpole smoke tests failed or produced no report: $report" + exit 1 + fi - name: Upload test reports if: always() diff --git a/source/isaaclab/changelog.d/jichuanh-arm-ci-align-x86.skip b/source/isaaclab/changelog.d/jichuanh-arm-ci-align-x86.skip new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/source/isaaclab/test/controllers/test_operational_space.py b/source/isaaclab/test/controllers/test_operational_space.py index 76ed5a42165f..8a98daeed8f3 100644 --- a/source/isaaclab/test/controllers/test_operational_space.py +++ b/source/isaaclab/test/controllers/test_operational_space.py @@ -13,7 +13,7 @@ import pytest import torch -pytestmark = pytest.mark.arm_ci +pytestmark = [pytest.mark.integration, pytest.mark.arm_ci] import isaaclab.envs.mdp as mdp import isaaclab.sim as sim_utils @@ -52,8 +52,6 @@ from isaaclab_assets import FRANKA_PANDA_LEGACY_CFG, G1_29DOF_CFG # isort:skip -pytestmark = pytest.mark.integration - @pytest.fixture def sim(): diff --git a/source/isaaclab/test/install_ci/conftest.py b/source/isaaclab/test/install_ci/conftest.py index 7cc322fe5df1..11aa6168af7e 100644 --- a/source/isaaclab/test/install_ci/conftest.py +++ b/source/isaaclab/test/install_ci/conftest.py @@ -9,11 +9,14 @@ import os import platform +import shutil import subprocess import sys +from collections.abc import Callable, Iterator from pathlib import Path import pytest +import tomllib # Make ``utils`` (sibling of this conftest) importable from tests living in # subdirectories (``cli/``, ``misc/``). Required because ``pytest.ini`` uses @@ -96,6 +99,137 @@ def wheel_path() -> Path | None: return None +@pytest.fixture(scope="session") +def checkout() -> Path: + return Path(__file__).resolve().parents[4] + + +@pytest.fixture(scope="session") +def run() -> Callable[..., str]: + """Run with bounded capture and no inherited Python environment or package indexes.""" + clean_env = { + key: value + for key, value in os.environ.items() + if not key.startswith(("UV_", "PIP_", "CONDA_")) + and key + not in ( + "VIRTUAL_ENV", + "PYTHONPATH", + "PYTHONHOME", + "ISAAC_PATH", + "ISAACSIM_PATH", + "CARB_APP_PATH", + "EXP_PATH", + "PXR_PLUGINPATH_NAME", + "LD_LIBRARY_PATH", + "LD_PRELOAD", + ) + } + clean_env["OMNI_KIT_ACCEPT_EULA"] = "yes" + if platform.machine().lower() in ("aarch64", "arm64"): + clean_env["LD_PRELOAD"] = "/lib/aarch64-linux-gnu/libgomp.so.1" + + def execute(*args: str, cwd: Path, env: dict[str, str] | None = None, timeout: int = 900) -> str: + command_env = {**clean_env, **(env or {})} + # Keep platform-required libraries when a probe adds another preload. + preloads = [clean_env.get("LD_PRELOAD", ""), (env or {}).get("LD_PRELOAD", "")] + if any(preloads): + command_env["LD_PRELOAD"] = ":".join(preload for preload in preloads if preload) + result = subprocess.run( + args, + cwd=cwd, + env=command_env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=timeout, + check=False, + ) + assert result.returncode == 0, f"{args} exited with {result.returncode}:\n{result.stdout}" + return result.stdout + + return execute + + +@pytest.fixture(scope="session") +def wheel() -> Path: + value = os.environ.get("ISAACLAB_WHEEL") + if not value: + pytest.fail("Supply ISAACLAB_WHEEL or use tools/run_install_ci.py --build-wheel.") + path = Path(value).resolve() + assert path.is_file(), f"Wheel not found: {path}" + return path + + +@pytest.fixture(scope="session", params=["base", "isaacsim"]) +def workspace( + request: pytest.FixtureRequest, tmp_path_factory: pytest.TempPathFactory, checkout: Path, run: Callable[..., str] +) -> Iterator[tuple[str, Path, dict[str, str]]]: + """Each supported runtime installs once, then shares its environment with probes.""" + assert not (checkout / "_isaac_sim").exists(), "Run installation CI from a checkout without a local Kit build" + directory = tmp_path_factory.mktemp(request.param) + env = {"UV_PROJECT_ENVIRONMENT": str(directory / "venv")} + extras = ["--extra", "isaacsim"] if request.param == "isaacsim" else [] + # Validate the committed lock rather than silently resolving or rewriting it. + run("uv", "sync", "--locked", *extras, cwd=checkout, env=env, timeout=4500) + yield request.param, directory, env + shutil.rmtree(directory / "venv") + + +@pytest.fixture(scope="session", params=["base", "isaacsim"]) +def installed_wheel( + request: pytest.FixtureRequest, + tmp_path_factory: pytest.TempPathFactory, + checkout: Path, + wheel: Path, + run: Callable[..., str], +) -> Iterator[tuple[str, Path, Path]]: + directory = tmp_path_factory.mktemp(f"wheel-{request.param}") + python = directory / "venv" / ("Scripts/python.exe" if os.name == "nt" else "bin/python") + run("uv", "venv", "--python", "3.12", str(directory / "venv"), cwd=directory) + requirement = str(wheel) + ("[isaacsim]" if request.param == "isaacsim" else "") + run( + "uv", + "--no-config", + "pip", + "install", + "--python", + str(python), + requirement, + "--overrides", + str(checkout / "tools/wheel_builder/uv-overrides.txt"), + "--extra-index-url", + "https://pypi.nvidia.com", + "--index-strategy", + "unsafe-best-match", + cwd=directory, + timeout=4500, + ) + # Wheel consumers select their CUDA index explicitly; project sources do not propagate. + with (checkout / "pyproject.toml").open("rb") as file: + versions = tomllib.load(file)["tool"]["isaaclab"]["versions"] + run( + "uv", + "--no-config", + "pip", + "install", + "--python", + str(python), + f"torch=={versions['torch']}", + f"torchvision=={versions['torchvision']}", + "--reinstall-package", + "torch", + "--reinstall-package", + "torchvision", + "--index-url", + "https://download.pytorch.org/whl/cu130", + cwd=directory, + timeout=4500, + ) + yield request.param, directory, python + shutil.rmtree(directory / "venv") + + # Markers @@ -109,6 +243,7 @@ def pytest_addoption(parser: pytest.Parser) -> None: dest="noskip", help="Strip all @pytest.mark.skip markers for this run (does not affect skipif). Alias: --force.", ) + parser.addoption("--run-gpu", action="store_true", help="Run GPU training after installation.") def pytest_configure(config: pytest.Config) -> None: @@ -195,6 +330,11 @@ def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item items[:] = remaining config.hook.pytest_deselected(items=deselected) + if not config.getoption("--run-gpu"): + for item in items: + if "gpu" in item.keywords: + item.add_marker(pytest.mark.skip(reason="pass --run-gpu to exercise GPU training")) + if config.getoption("--noskip"): for item in items: item.own_markers = [m for m in item.own_markers if m.name != "skip"] diff --git a/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py b/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py index 862c0467eb8a..59f12e3d8ee1 100644 --- a/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py +++ b/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py @@ -5,9 +5,8 @@ """Shared state-training, camera-rendering, and camera-training Cartpole smoke probes. -The caller owns environment preparation. Installation CI executes this file -inside the environment it just installed, while architecture CI executes the -same probes inside its prepared image. +The caller owns environment preparation. Architecture CI executes all probes +inside its prepared image, while kit-less image validation runs the state-training probe. """ from __future__ import annotations diff --git a/source/isaaclab/test/install_ci/test_workspace.py b/source/isaaclab/test/install_ci/test_workspace.py new file mode 100644 index 000000000000..939848cb223a --- /dev/null +++ b/source/isaaclab/test/install_ci/test_workspace.py @@ -0,0 +1,63 @@ +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +"""The committed uv workspace installs and launches its supported runtimes.""" + +import platform +from collections.abc import Callable +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.slow + + +def test_workspace_install( + workspace: tuple[str, Path, dict[str, str]], checkout: Path, run: Callable[..., str] +) -> None: + runtime, _, env = workspace + probe = "import isaaclab, isaaclab_tasks; from isaaclab.sim import SimulationContext" + if runtime == "isaacsim": + probe += "; from isaaclab_physx.app import KitLauncher" + else: + probe += ( + "; import importlib.metadata as m; " + "assert not any(d.metadata['Name'] == 'isaacsim' for d in m.distributions())" + ) + run("uv", "run", "--no-sync", "python", "-c", probe, cwd=checkout, env=env) + output = run("uv", "run", "--no-sync", "isaaclab", "--help", cwd=checkout, env=env) + assert "train" in output + + +@pytest.mark.gpu +def test_workspace_trains(workspace: tuple[str, Path, dict[str, str]], checkout: Path, run: Callable[..., str]) -> None: + runtime, directory, env = workspace + if runtime == "isaacsim": + if platform.system() == "Linux": + shim = directory / "venv/lib/python3.12/site-packages/isaacsim/kit/kernel/plugins/libcarb.env.shim.so" + assert shim.is_file() + env = {**env, "LD_PRELOAD": str(shim)} + physics = "isaacsim_physx" if runtime == "isaacsim" else "newton_mjwarp" + output = run( + "uv", + "run", + "--no-sync", + "isaaclab", + "train", + "--rl_library", + "rsl_rl", + "--task", + "Isaac-Cartpole-Direct", + "--num_envs", + "16", + "--max_iterations", + "5", + f"physics={physics}", + cwd=checkout, + env=env, + timeout=900, + ) + assert "Training time:" in output + assert "Traceback (most recent call last):" not in output diff --git a/source/isaaclab_tasks/changelog.d/jichuanh-arm-ci-align-x86.skip b/source/isaaclab_tasks/changelog.d/jichuanh-arm-ci-align-x86.skip new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/source/isaaclab_tasks/test/core/test_rendering_kuka_visual_material_randomization_kitless.py b/source/isaaclab_tasks/test/core/test_rendering_kuka_visual_material_randomization_kitless.py index a7f2aacd7ced..fbeaf8a2894a 100644 --- a/source/isaaclab_tasks/test/core/test_rendering_kuka_visual_material_randomization_kitless.py +++ b/source/isaaclab_tasks/test/core/test_rendering_kuka_visual_material_randomization_kitless.py @@ -18,7 +18,6 @@ ) pytestmark = [ - pytest.mark.arm_ci, pytest.mark.integration, pytest.mark.rendering, pytest.mark.kitless, diff --git a/source/isaaclab_tasks/test/core/test_rendering_shadow_hand_kitless.py b/source/isaaclab_tasks/test/core/test_rendering_shadow_hand_kitless.py index ea44307f02c2..b17e9c41a0c3 100644 --- a/source/isaaclab_tasks/test/core/test_rendering_shadow_hand_kitless.py +++ b/source/isaaclab_tasks/test/core/test_rendering_shadow_hand_kitless.py @@ -19,8 +19,8 @@ rendering_test_shadow_hand, ) -# no arm_ci marker: intermittently stalls the arm64 runner mid-file (renderer hang after a few -# passing cases, killed at the per-file timeout); x86 kitless coverage remains in place +pytestmark = pytest.mark.arm_ci + _RENDERING_PARAMS = group_rendering_params( make_kitless_rendering_params(MINIMAL_KITLESS_PHYSICS_RENDERER_AOV_COMBINATIONS) ) diff --git a/tools/conftest.py b/tools/conftest.py index 3844e56f69fa..4980bd221253 100644 --- a/tools/conftest.py +++ b/tools/conftest.py @@ -1404,8 +1404,15 @@ def _run_one_pass( def _starts_renderer(test_content: str) -> bool: - """Return whether a test file's source starts an RTX renderer: Kit's, by enabling cameras, or OVRTX.""" - return "enable_cameras=True" in test_content or "ovrtx" in test_content.lower() + """Return whether a test file's source starts an RTX renderer: Kit's, by enabling cameras, or OVRTX. + + Files built on ``rendering_test_utils`` pick their renderer through its helpers, so they name neither. + """ + return ( + "enable_cameras=True" in test_content + or "ovrtx" in test_content.lower() + or "rendering_test_utils" in test_content + ) def _test_file_job(test_file: str, max_slots: int) -> TestFileJob: From 502db9eddec5bbecec51de46747806ea22e62308 Mon Sep 17 00:00:00 2001 From: Antoine RICHARD Date: Tue, 13 Oct 2026 13:16:04 +0200 Subject: [PATCH 2/3] Map #8233 install_ci changes onto the release test layout release/3.0.0 still has the cli/, uv_pip/ and uv_run/ install_ci layout, so the conflicting #8233 hunks have no direct target. The proposed resolution instead ported develop's workspace fixtures, test_workspace.py and a --run-gpu gate that nothing on this branch passes. Its module-level tomllib import broke x86 test collection on Python 3.10, and the gate skipped existing ARM training tests. Restore conftest.py, drop test_workspace.py, and carry the actual change: re-enable Isaac Sim PhysX training on aarch64 in the uv_run test that test_workspace.py replaced on develop. That test already preserves the ARM OpenMP preload when it adds the Carbonite shim. --- source/isaaclab/test/install_ci/conftest.py | 140 ------------------ .../test/install_ci/test_workspace.py | 63 -------- .../test_uv_run_isaacsim_trains_cartpole.py | 8 - 3 files changed, 211 deletions(-) delete mode 100644 source/isaaclab/test/install_ci/test_workspace.py diff --git a/source/isaaclab/test/install_ci/conftest.py b/source/isaaclab/test/install_ci/conftest.py index 11aa6168af7e..7cc322fe5df1 100644 --- a/source/isaaclab/test/install_ci/conftest.py +++ b/source/isaaclab/test/install_ci/conftest.py @@ -9,14 +9,11 @@ import os import platform -import shutil import subprocess import sys -from collections.abc import Callable, Iterator from pathlib import Path import pytest -import tomllib # Make ``utils`` (sibling of this conftest) importable from tests living in # subdirectories (``cli/``, ``misc/``). Required because ``pytest.ini`` uses @@ -99,137 +96,6 @@ def wheel_path() -> Path | None: return None -@pytest.fixture(scope="session") -def checkout() -> Path: - return Path(__file__).resolve().parents[4] - - -@pytest.fixture(scope="session") -def run() -> Callable[..., str]: - """Run with bounded capture and no inherited Python environment or package indexes.""" - clean_env = { - key: value - for key, value in os.environ.items() - if not key.startswith(("UV_", "PIP_", "CONDA_")) - and key - not in ( - "VIRTUAL_ENV", - "PYTHONPATH", - "PYTHONHOME", - "ISAAC_PATH", - "ISAACSIM_PATH", - "CARB_APP_PATH", - "EXP_PATH", - "PXR_PLUGINPATH_NAME", - "LD_LIBRARY_PATH", - "LD_PRELOAD", - ) - } - clean_env["OMNI_KIT_ACCEPT_EULA"] = "yes" - if platform.machine().lower() in ("aarch64", "arm64"): - clean_env["LD_PRELOAD"] = "/lib/aarch64-linux-gnu/libgomp.so.1" - - def execute(*args: str, cwd: Path, env: dict[str, str] | None = None, timeout: int = 900) -> str: - command_env = {**clean_env, **(env or {})} - # Keep platform-required libraries when a probe adds another preload. - preloads = [clean_env.get("LD_PRELOAD", ""), (env or {}).get("LD_PRELOAD", "")] - if any(preloads): - command_env["LD_PRELOAD"] = ":".join(preload for preload in preloads if preload) - result = subprocess.run( - args, - cwd=cwd, - env=command_env, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - timeout=timeout, - check=False, - ) - assert result.returncode == 0, f"{args} exited with {result.returncode}:\n{result.stdout}" - return result.stdout - - return execute - - -@pytest.fixture(scope="session") -def wheel() -> Path: - value = os.environ.get("ISAACLAB_WHEEL") - if not value: - pytest.fail("Supply ISAACLAB_WHEEL or use tools/run_install_ci.py --build-wheel.") - path = Path(value).resolve() - assert path.is_file(), f"Wheel not found: {path}" - return path - - -@pytest.fixture(scope="session", params=["base", "isaacsim"]) -def workspace( - request: pytest.FixtureRequest, tmp_path_factory: pytest.TempPathFactory, checkout: Path, run: Callable[..., str] -) -> Iterator[tuple[str, Path, dict[str, str]]]: - """Each supported runtime installs once, then shares its environment with probes.""" - assert not (checkout / "_isaac_sim").exists(), "Run installation CI from a checkout without a local Kit build" - directory = tmp_path_factory.mktemp(request.param) - env = {"UV_PROJECT_ENVIRONMENT": str(directory / "venv")} - extras = ["--extra", "isaacsim"] if request.param == "isaacsim" else [] - # Validate the committed lock rather than silently resolving or rewriting it. - run("uv", "sync", "--locked", *extras, cwd=checkout, env=env, timeout=4500) - yield request.param, directory, env - shutil.rmtree(directory / "venv") - - -@pytest.fixture(scope="session", params=["base", "isaacsim"]) -def installed_wheel( - request: pytest.FixtureRequest, - tmp_path_factory: pytest.TempPathFactory, - checkout: Path, - wheel: Path, - run: Callable[..., str], -) -> Iterator[tuple[str, Path, Path]]: - directory = tmp_path_factory.mktemp(f"wheel-{request.param}") - python = directory / "venv" / ("Scripts/python.exe" if os.name == "nt" else "bin/python") - run("uv", "venv", "--python", "3.12", str(directory / "venv"), cwd=directory) - requirement = str(wheel) + ("[isaacsim]" if request.param == "isaacsim" else "") - run( - "uv", - "--no-config", - "pip", - "install", - "--python", - str(python), - requirement, - "--overrides", - str(checkout / "tools/wheel_builder/uv-overrides.txt"), - "--extra-index-url", - "https://pypi.nvidia.com", - "--index-strategy", - "unsafe-best-match", - cwd=directory, - timeout=4500, - ) - # Wheel consumers select their CUDA index explicitly; project sources do not propagate. - with (checkout / "pyproject.toml").open("rb") as file: - versions = tomllib.load(file)["tool"]["isaaclab"]["versions"] - run( - "uv", - "--no-config", - "pip", - "install", - "--python", - str(python), - f"torch=={versions['torch']}", - f"torchvision=={versions['torchvision']}", - "--reinstall-package", - "torch", - "--reinstall-package", - "torchvision", - "--index-url", - "https://download.pytorch.org/whl/cu130", - cwd=directory, - timeout=4500, - ) - yield request.param, directory, python - shutil.rmtree(directory / "venv") - - # Markers @@ -243,7 +109,6 @@ def pytest_addoption(parser: pytest.Parser) -> None: dest="noskip", help="Strip all @pytest.mark.skip markers for this run (does not affect skipif). Alias: --force.", ) - parser.addoption("--run-gpu", action="store_true", help="Run GPU training after installation.") def pytest_configure(config: pytest.Config) -> None: @@ -330,11 +195,6 @@ def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item items[:] = remaining config.hook.pytest_deselected(items=deselected) - if not config.getoption("--run-gpu"): - for item in items: - if "gpu" in item.keywords: - item.add_marker(pytest.mark.skip(reason="pass --run-gpu to exercise GPU training")) - if config.getoption("--noskip"): for item in items: item.own_markers = [m for m in item.own_markers if m.name != "skip"] diff --git a/source/isaaclab/test/install_ci/test_workspace.py b/source/isaaclab/test/install_ci/test_workspace.py deleted file mode 100644 index 939848cb223a..000000000000 --- a/source/isaaclab/test/install_ci/test_workspace.py +++ /dev/null @@ -1,63 +0,0 @@ -# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). -# All rights reserved. -# -# SPDX-License-Identifier: BSD-3-Clause - -"""The committed uv workspace installs and launches its supported runtimes.""" - -import platform -from collections.abc import Callable -from pathlib import Path - -import pytest - -pytestmark = pytest.mark.slow - - -def test_workspace_install( - workspace: tuple[str, Path, dict[str, str]], checkout: Path, run: Callable[..., str] -) -> None: - runtime, _, env = workspace - probe = "import isaaclab, isaaclab_tasks; from isaaclab.sim import SimulationContext" - if runtime == "isaacsim": - probe += "; from isaaclab_physx.app import KitLauncher" - else: - probe += ( - "; import importlib.metadata as m; " - "assert not any(d.metadata['Name'] == 'isaacsim' for d in m.distributions())" - ) - run("uv", "run", "--no-sync", "python", "-c", probe, cwd=checkout, env=env) - output = run("uv", "run", "--no-sync", "isaaclab", "--help", cwd=checkout, env=env) - assert "train" in output - - -@pytest.mark.gpu -def test_workspace_trains(workspace: tuple[str, Path, dict[str, str]], checkout: Path, run: Callable[..., str]) -> None: - runtime, directory, env = workspace - if runtime == "isaacsim": - if platform.system() == "Linux": - shim = directory / "venv/lib/python3.12/site-packages/isaacsim/kit/kernel/plugins/libcarb.env.shim.so" - assert shim.is_file() - env = {**env, "LD_PRELOAD": str(shim)} - physics = "isaacsim_physx" if runtime == "isaacsim" else "newton_mjwarp" - output = run( - "uv", - "run", - "--no-sync", - "isaaclab", - "train", - "--rl_library", - "rsl_rl", - "--task", - "Isaac-Cartpole-Direct", - "--num_envs", - "16", - "--max_iterations", - "5", - f"physics={physics}", - cwd=checkout, - env=env, - timeout=900, - ) - assert "Training time:" in output - assert "Traceback (most recent call last):" not in output diff --git a/source/isaaclab/test/install_ci/uv_run/test_uv_run_isaacsim_trains_cartpole.py b/source/isaaclab/test/install_ci/uv_run/test_uv_run_isaacsim_trains_cartpole.py index 851cbba6b1f5..2b065072e99d 100644 --- a/source/isaaclab/test/install_ci/uv_run/test_uv_run_isaacsim_trains_cartpole.py +++ b/source/isaaclab/test/install_ci/uv_run/test_uv_run_isaacsim_trains_cartpole.py @@ -15,7 +15,6 @@ from __future__ import annotations -import platform import shutil import pytest @@ -52,13 +51,6 @@ class Test_Uv_Run_Isaacsim_Trains_Cartpole: def setup_class(cls): if not shutil.which("uv"): pytest.skip("uv is not available") - # PhysX training currently fails at env creation on aarch64 (DGX Spark): - # "[omni.physx.tensors.plugin] Simulation view object is invalidated and cannot - # be used again to call updateArticulationsKinematic". The environment itself - # installs fine from the lock; this matches the x86-only scope of the workflow - # this test replaces. Re-enable once PhysX training works on aarch64. - if platform.machine().lower() in ("aarch64", "arm64"): - pytest.skip("PhysX training is not functional on aarch64 yet") @pytest.mark.docker @pytest.mark.smoke From 3e661d2cfcb6768ee5d58fa10e212ebeb633e266 Mon Sep 17 00:00:00 2001 From: Antoine RICHARD Date: Tue, 13 Oct 2026 13:28:38 +0200 Subject: [PATCH 3/3] Keep installation CI as a smoke probe caller on release #8233 dropped installation CI from the probe's caller list because develop's install suite no longer runs it. On release/3.0.0 the uv pip wheel install test still executes this file through the cartpole_smoke_script fixture, so name all three callers. --- .../isaaclab/test/install_ci/misc/cartpole_training_smoke.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py b/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py index 59f12e3d8ee1..f4772a84ddde 100644 --- a/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py +++ b/source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py @@ -5,8 +5,9 @@ """Shared state-training, camera-rendering, and camera-training Cartpole smoke probes. -The caller owns environment preparation. Architecture CI executes all probes -inside its prepared image, while kit-less image validation runs the state-training probe. +The caller owns environment preparation. Installation CI executes this file +inside the environment it just installed, architecture CI executes all probes +inside its prepared image, and kit-less image validation runs the state-training probe. """ from __future__ import annotations