Repository navigation
[Backport release/3.0.0] [CI] Restore ARM CI on the expanded DGX Spark pool (#8229) - #8242
Conversation
## Review Map - A stacked PR's page shows the cumulative diff of its dependency chain; the **Exact changes** link pins that PR's own commit range. - SHA links can go stale after a branch update; #8229's copy is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | 📌 #8229 Restore ARM CI build on the DGX Spark pool (1/2) (draft) |  | — | — | | #8230 Re-enable two disabled ARM tests (2/2) (draft) |  | #8229 | https://github.com/isaac-sim/IsaacLab/pull/8230/changes/6e8879060e9b10e7bface139676524da3139cfab..6c5c43d1353009723b884ba09e08780c95a1362b | ## Summary - ARM CI builds again: the Isaac Sim CI pin goes back to the multi-arch manifest list, so the arm64 base pull stops receiving an amd64 image. - A failing ARM CI now shows red instead of green, and the two pre-existing ARM failures it exposed are fixed; it stays non-required, so it does not block merges. # Description ## 1. Changes 1. `.github/workflows/config.yaml`: pin `latest-develop@sha256:e78f716a1cae…`, the manifest list of the same 6.2.0-alpha.12 build. #8097 pinned its x86_64 child `a6c53e29d6f1`, so every arm64 build failed with `InvalidBaseImagePlatform … pulled with platform "linux/amd64"` → `exec format error` (e.g. [run 36811233504](https://github.com/isaac-sim/IsaacLab/actions/runs/36811233504)). x86 resolves to the identical `a6c53e29d6f1` image. 2. `.github/workflows/arm-ci.yml`: drop the job-level `continue-on-error`. It marked the workflow run successful while `Build & Test` failed, so the broken build above looked green in the run list. 3. `isaaclab/cli/utils.py`: on aarch64, preload the system `libgomp.so.1` by its full `/lib/*/` path instead of the bare soname. `isaacsim` exits at import unless each such path is listed verbatim in `LD_PRELOAD` (it never reads `/etc/ld.so.preload`), and the CLI launches every Python process, including ARM CI's pytest, so `test_operational_space.py` exited before running (`aarch_preload_checking()` → `SystemExit`, [run 36812221963](https://github.com/isaac-sim/IsaacLab/actions/runs/36812221963)). Same failure for anyone running the CLI from a virtual environment on a Spark. Passing that path to the test container made the `arm_ci` step pass ([run 36816068195](https://github.com/isaac-sim/IsaacLab/actions/runs/36816068195)); the fix now lives in the CLI, with a regression test in `test_install.py`. 4. `.github/workflows/arm-ci.yml`: deselect `ovstage` cases in both ARM test steps. The ovstage variants of the `arm_ci` kit-less rendering tests fail image comparison on arm64 (rgb diff 1.99 % / 7.04 % / 25.10 % over 1.5–3 % thresholds) and fail on every x86 develop run, where ovstage is post-merge only; the legacy cases pass. The two ARM tests disabled for real failures (shadow-hand kit-less rendering, Isaac Sim PhysX Cartpole) are re-enabled separately in #8230 (ARM test re-enables), after repeated DGX Spark runs. ## 2. Runner capacity | Metric | 07-28..08-04 (2 runners) | 09-26..10-01 (18 runners) | |---|---|---| | ARM jobs never given a runner | 723 / 919 | 0 | | Queue for jobs that ran | p50 300 min, max 605 min | p50 0.0 min, max 12 min | | Fleet utilization | saturated | ~6% | ARM CI ran p50 31 / p90 45 min; at the July trigger rate it adds about 20% utilization. The `ARM CI` workflow, disabled manually on 08-05, was re-enabled separately (repository setting). ## 3. Test plan - [x] arm64 base image builds on DGX Spark with the list pin ([run 36812221963](https://github.com/isaac-sim/IsaacLab/actions/runs/36812221963), `Build base image (linux/arm64)` passed) - [x] ARM CI `Build & Test` green on this PR ([run 36817584603](https://github.com/isaac-sim/IsaacLab/actions/runs/36817584603); `test_operational_space.py` now imports, but its tests stay deselected on ARM because a later `pytestmark = pytest.mark.integration` overrides its `arm_ci` marker) - [x] pre-commit clean ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Release backport - [ ] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` ## Checklist Docker and GPU tests run on demand. Push the commits you want tested, then comment `run-ci` on the pull request. - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
Isaac Lab Review Bot
Backport restores ARM CI by pinning the multi-architecture Isaac Sim manifest, surfacing ARM job failures, excluding known-failing ARM ovstage cases, and preloading the full system libgomp.so.1 path for aarch64 CLI launches. Two minor maintainability/portability fixes remain.
- Design and architecture: The workflow changes and private
_aarch64_libgomp_envhelper keep CI policy separate from CLI environment construction. The helper remains stateless and focused; however, its library discovery currently assumes only Debian-style multiarch paths. - API: No public symbol, signature, CLI flag, default, or configuration key changes.
_aarch64_libgomp_envremains private and is consumed through the existingrun_python_commandpath. - Compatibility and deprecation: Breaking changes: none identified. Existing
LD_PRELOADentries remain preserved, and non-Linux/non-aarch64 execution remains unchanged. On aarch64 distributions where system libgomp is located under/lib64or/usr/lib64, the new discovery can fail to add the preload that the previous bare-soname behavior supplied; broaden the search paths. - Implementation: The preload path composition is deterministic and idempotent for matches found under
/lib/*/libgomp.so.1. The exact changed glob misses common non-multiarch locations such as/lib64/libgomp.so.1and/usr/lib64/libgomp.so.1, causing the helper to return the environment unchanged on affected aarch64 systems. Include those established layouts before treating libgomp as absent. - Style consistency: The import ordering, updated helper docstring, and changelog fragment follow repository conventions. In
arm-ci.yml, the comment claiming x86 post-merge parity contradicts the unconditionalnot ovstagefilter across both pull-request and protected-branch push runs. Document the actual known ARM failure and make the rationale clear for both filtered test invocations. - Test quality: The added test protects the distinct aarch64
run_python_commandcontract by asserting the exact composedLD_PRELOADvalue and would fail with the prior bare-soname behavior. It does not duplicate the supplied CLI tests. Its mocked Debian-style path does not cover the portability gap in the production glob.
Minor fixes needed. Posted 2 actionable findings inline.
Automated review; human maintainers own approval decisions.
| image-tag: ${{ needs.config.outputs.ci_image_tag }}-arm64 | ||
| extra-pip-packages: "${{ steps.ov_pins.outputs.ovrtx }} ${{ steps.ov_pins.outputs.ovphysx }}" | ||
| test-k-expr: not ovphysx | ||
| # ovstage rendering cases are post-merge only on x86 as well. |
There was a problem hiding this comment.
🔵 Suggestion · Style Consistency — ovstage comment contradicts the applied filter
The comment explains the skip as x86 post-merge parity, but this workflow also runs on push to main/develop/release/**, and not ovstage applies there too, so these cases never run on ARM. The PR's actual reason is that the ARM ovstage rendering cases are known-failing. State that reason here (and the duplicate filter on line 199 carries no comment at all) so the exclusion can be revisited.
| preload = merged.get("LD_PRELOAD", "") | ||
| if any(os.path.basename(entry) == libgomp for entry in preload.split(":") if entry): | ||
| preload = [entry for entry in merged.get("LD_PRELOAD", "").split(":") if entry] | ||
| missing = [path for path in sorted(glob.glob("/lib/*/libgomp.so.1")) if path not in preload] |
There was a problem hiding this comment.
🔵 Suggestion · Implementation — Glob misses non-multiarch libgomp layouts
/lib/*/libgomp.so.1 requires an intermediate directory, so it matches Debian-style /lib/aarch64-linux-gnu/ but not /lib64/libgomp.so.1 or /usr/lib64/libgomp.so.1 used by RHEL-family aarch64 images. There the helper now returns env unchanged and sets no preload at all, where it previously always added an entry. Extend the search to cover /lib64 and /usr/lib* layouts before concluding no system libgomp exists.
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| preload = merged.get("LD_PRELOAD", "") | ||
| if any(os.path.basename(entry) == libgomp for entry in preload.split(":") if entry): | ||
| preload = [entry for entry in merged.get("LD_PRELOAD", "").split(":") if entry] | ||
| missing = [path for path in sorted(glob.glob("/lib/*/libgomp.so.1")) if path not in preload] |
There was a problem hiding this comment.
On Linux aarch64, the conda activation hook can put Torch’s libgomp.so.1 in LD_PRELOAD. This exact-path check does not recognize it and adds the system library too. The installation guide warns that preloading both can produce libgomp warnings. Handle the existing Torch entry when building the child environment.
| extra-pip-packages: "${{ steps.ov_pins.outputs.ovrtx }} ${{ steps.ov_pins.outputs.ovphysx }}" | ||
| test-k-expr: not ovphysx | ||
| # ovstage rendering cases are post-merge only on x86 as well. | ||
| test-k-expr: not ovphysx and not ovstage |
There was a problem hiding this comment.
The not ovstage filter applies on protected-branch pushes as well as PRs, and the other ARM test shard excludes ovstage too. Unlike x86 CI, ARM CI has no separate post-merge shard, so those ARM rendering cases no longer run after merge. Retain post-merge ARM coverage for them when they can run reliably.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
Description
Backports #8229 onto
release/3.0.0.This restores ARM CI on the expanded DGX Spark pool by pinning the multi-architecture Isaac Sim image manifest, reporting ARM CI failures as failed workflow runs, skipping the known-failing
ovstageARM rendering cases, and preloading the systemlibgomp.so.1path for aarch64 CLI launches.This is an exact, conflict-free cherry-pick of the merged source change. The original regression test and
isaaclabchangelog fragment are included unchanged.Type of change
Release backport
release/3.0.0directly.Screenshots
Not applicable.
Validation
ac7312366230803ebddd56fc65e0a5d3e96e5e62.1 passed.release/3.0.0tip.uv run isaaclab -fentry point is unavailable on this macOS ARM64 host because the lockfile supports Linux and Windows only; the equivalent direct pre-commit run passed.Checklist
isaaclabchangelog fragment.CONTRIBUTORS.md.Original PR: #8229
Cherry-picked merge commit:
1341561896f4f9b650c1f837547d4eb544c2d780Backport commit:
6d062631084ce8e681cd023ea187f295248156b9