Repository navigation
Conversation
9 of 11 tasks
ooctipus
pushed a commit
that referenced
this pull request
Oct 1, 2026
## Review Map - A stacked PR's page shows the cumulative diff of its dependency chain; the **Exact changes** link pins that PR's own commit range. - SHA links can go stale after a branch update; #8229's copy is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | 📌 #8229 Restore ARM CI build on the DGX Spark pool (1/2) (draft) |  | — | — | | #8230 Re-enable two disabled ARM tests (2/2) (draft) |  | #8229 | https://github.com/isaac-sim/IsaacLab/pull/8230/changes/6e8879060e9b10e7bface139676524da3139cfab..6c5c43d1353009723b884ba09e08780c95a1362b | ## Summary - ARM CI builds again: the Isaac Sim CI pin goes back to the multi-arch manifest list, so the arm64 base pull stops receiving an amd64 image. - A failing ARM CI now shows red instead of green, and the two pre-existing ARM failures it exposed are fixed; it stays non-required, so it does not block merges. # Description ## 1. Changes 1. `.github/workflows/config.yaml`: pin `latest-develop@sha256:e78f716a1cae…`, the manifest list of the same 6.2.0-alpha.12 build. #8097 pinned its x86_64 child `a6c53e29d6f1`, so every arm64 build failed with `InvalidBaseImagePlatform … pulled with platform "linux/amd64"` → `exec format error` (e.g. [run 36811233504](https://github.com/isaac-sim/IsaacLab/actions/runs/36811233504)). x86 resolves to the identical `a6c53e29d6f1` image. 2. `.github/workflows/arm-ci.yml`: drop the job-level `continue-on-error`. It marked the workflow run successful while `Build & Test` failed, so the broken build above looked green in the run list. 3. `isaaclab/cli/utils.py`: on aarch64, preload the system `libgomp.so.1` by its full `/lib/*/` path instead of the bare soname. `isaacsim` exits at import unless each such path is listed verbatim in `LD_PRELOAD` (it never reads `/etc/ld.so.preload`), and the CLI launches every Python process, including ARM CI's pytest, so `test_operational_space.py` exited before running (`aarch_preload_checking()` → `SystemExit`, [run 36812221963](https://github.com/isaac-sim/IsaacLab/actions/runs/36812221963)). Same failure for anyone running the CLI from a virtual environment on a Spark. Passing that path to the test container made the `arm_ci` step pass ([run 36816068195](https://github.com/isaac-sim/IsaacLab/actions/runs/36816068195)); the fix now lives in the CLI, with a regression test in `test_install.py`. 4. `.github/workflows/arm-ci.yml`: deselect `ovstage` cases in both ARM test steps. The ovstage variants of the `arm_ci` kit-less rendering tests fail image comparison on arm64 (rgb diff 1.99 % / 7.04 % / 25.10 % over 1.5–3 % thresholds) and fail on every x86 develop run, where ovstage is post-merge only; the legacy cases pass. The two ARM tests disabled for real failures (shadow-hand kit-less rendering, Isaac Sim PhysX Cartpole) are re-enabled separately in #8230 (ARM test re-enables), after repeated DGX Spark runs. ## 2. Runner capacity | Metric | 07-28..08-04 (2 runners) | 09-26..10-01 (18 runners) | |---|---|---| | ARM jobs never given a runner | 723 / 919 | 0 | | Queue for jobs that ran | p50 300 min, max 605 min | p50 0.0 min, max 12 min | | Fleet utilization | saturated | ~6% | ARM CI ran p50 31 / p90 45 min; at the July trigger rate it adds about 20% utilization. The `ARM CI` workflow, disabled manually on 08-05, was re-enabled separately (repository setting). ## 3. Test plan - [x] arm64 base image builds on DGX Spark with the list pin ([run 36812221963](https://github.com/isaac-sim/IsaacLab/actions/runs/36812221963), `Build base image (linux/arm64)` passed) - [x] ARM CI `Build & Test` green on this PR ([run 36817584603](https://github.com/isaac-sim/IsaacLab/actions/runs/36817584603); `test_operational_space.py` now imports, but its tests stay deselected on ARM because a later `pytestmark = pytest.mark.integration` overrides its `arm_ci` marker) - [x] pre-commit clean ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Release backport - [ ] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` ## Checklist Docker and GPU tests run on demand. Push the commits you want tested, then comment `run-ci` on the pull request. - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
…s on ARM Both were disabled on ARM for real failures: the shadow-hand kit-less rendering file stalled mid-run on arm64 (isaac-sim#6581), and Isaac Sim PhysX training failed at environment creation on aarch64 with an invalidated simulation view. Restore the arm_ci marker and drop the aarch64 skip so DGX Spark runs both again.
hujc7
force-pushed
the
jichuanh/arm-ci-reenable-tests
branch
from
October 1, 2026 05:48
6c5c43d to
4a2ca2d
Compare
8 tasks done
hujc7
added a commit
that referenced
this pull request
Oct 13, 2026
…8233) ## Summary ARM CI restores compatible Warp and OVRTX caches and runs Newton/OVPhysX marker coverage through the shared x86 test action. Camera smoke coverage stays enabled, and previously skipped ARM checks are restored. # Description 1. Route ARM marker tests through `run-package-tests`, with explicit `linux/arm64` passed through image lookup, dependency hashing and fallback builds. PRs restore ARM caches; protected-branch pushes publish them. Test files run in four slots like the x86 test jobs; rendering files still run one at a time. 2. Run the existing state-training, camera-rendering and camera-training Cartpole probes as their own ARM step after the marker tests, reusing the restored Warp and OVRTX cache directories that `run-package-tests` now exposes as outputs. Installation CI now covers state training separately, so the camera probes remain necessary. The step checks the probe report explicitly, since `run-tests` records failures in the report rather than its exit code. 3. Restore shadow-hand kitless rendering and operational-space controller coverage on ARM. Carry the Isaac Sim PhysX training re-enable into the replacement uv workspace test rather than restoring a deleted installation test, and preserve the required ARM system OpenMP preload when the probe adds the Isaac Sim shim. Remove the Kuka visual-material ARM marker to match x86 rendering selection. Supersedes #8230. 4. Build the image in the runner's Docker daemon when no registry cache is configured, writing new layers with zstd. The disposable `docker-container` builder exported and reloaded the ~29 GB image every build and discarded its layer cache. Only ARM takes this path today: x86 callers resolve ECR and keep the `docker-container` builder. The action is part of the dependency-cache hash, so the first build after merge misses that cache once. 5. Count kitless rendering files that use `rendering_test_utils` as renderers, so their cold shader compiles no longer run concurrently. ## Validation 1. `uv run --frozen --extra test isaaclab -f`: all checks passed. 2. Docker-storage and file-scheduler coverage: 19 tests passed. Actionlint passed for ARM and x86 workflows; 50 local action calls validated against their declared inputs. 3. A real subprocess regression check failed before the preload fix and passes afterward for ARM with/without the shim and x86 with the shim. 4. On `7b73dafe1fe`, [ARM CI](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182263) passed 71 marker cases (3 expected skips) and all 3 state/camera smoke probes. [ARM installation](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182231) passed 3/3 attempts on three NVIDIA GB10 runners, with all 8 contracts including PhysX executed each time; x86 installation also passed all 8 contracts. 5. On the final head `01a3ff4737d`, the [x86 Docker suite](https://github.com/isaac-sim/IsaacLab/actions/runs/38312079847) passed: 30 jobs succeeded and 3 post-merge-only jobs were skipped as expected. Hosted pre-commit, Tools Tests, Docs, kitless image validation, installation and multi-GPU CI also passed. 6. ARM job time, all on NVIDIA GB10 with driver 580.159.03. A temporary commit (since reverted) let PR runs save caches for this measurement: | Run | Build | Tests | Total | | --- | ---: | ---: | ---: | | Before, [cold](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182263/job/112609307602) | 8m30 | 16m52 | 25m42 | | [Warm, new runner](https://github.com/isaac-sim/IsaacLab/actions/runs/38298855064) (3 runs) | 3m06–3m23 | 5m22–5m38 | 9m01–9m06 | | Warm, runner with layer cache | 0m56 | 5m42 | 6m57 | | [Final head, new runner](https://github.com/isaac-sim/IsaacLab/actions/runs/38312048044) | 3m31 | 9m22 | 13m11 | The first four rows also included the Newton RTX shader-cache fix, now split into #8504; the final head does not. Warm per-file times: Cartpole 54 s (was 297 s), Kuka 51 s, Shadow Hand 41 s, operational space 118 s alongside them. Newton RTX takes 254 s without #8504 and 27 s with it. Every run passed 71 marker cases (3 expected skips) and 3 smoke probes. Image export on the runners' containerd store fell from 243 s with gzip to 44–46 s with zstd. Protected-branch publication remains unmeasured before merge. ## Type of change 1. Bug fix (non-breaking CI and test coverage changes). ## Release backport - [x] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/develop/source/refs/contributing.html) - [x] I have run the `pre-commit` checks with `uv run isaaclab --format` - [x] Documentation describing the smoke-test caller matches the current workflow; rendered documentation is unaffected - [x] No new project warning sites were added; CI still reports existing dependency and action deprecations - [x] I have validated the changed GPU behavior with focused coverage and applied the test-audit criteria to test changes - [x] I have added changelog fragments for the touched source packages - [x] My name already exists in `CONTRIBUTORS.md`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Description
1. Changes
test_rendering_shadow_hand_kitless.py: restore thearm_cimarker removed in [CI] Drop shadow-hand kitless rendering test from arm-ci to stop intermittent hangs #6581 (shadow-hand arm64 stalls). The file stalled mid-run on arm64 in 15 of 81 ARM CI runs (renderer hang, killed at the per-file timeout); x86 hit the same stall at about half that rate. Since then OVRTX moved from 0.3 to 0.5, and the file passed 20 of 20 recent x86 PR runs.test_uv_run_isaacsim_trains_cartpole.py: drop the aarch64 skip. Isaac Sim PhysX training failed at environment creation on DGX Spark withSimulation view object is invalidated and cannot be used again to call updateArticulationsKinematic.2. Repeat runs on DGX Spark
Only the ARM jobs are repeated; results are filled in as runs finish.
Build & TestInstallation Tests (ARM)3. Test plan
Installation Tests (ARM)runs (5 / 5 including the runs before the restack)Type of change
Release backport
developChecklist
Docker and GPU tests run on demand. Push the commits you want tested, then
comment
run-cion the pull request.pre-commitchecks withuv run isaaclab --formatsource/<pkg>/changelog.d/for every touched package (.skip: test-only change)CONTRIBUTORS.mdor my name already exists there