Repository navigation
[CI] Restore ARM CI on the expanded DGX Spark pool - #8229
Conversation
… ARM The ARM pool grew from two DGX Spark runners to eighteen, so the two tests dropped from ARM coverage run again: the shadow-hand kit-less rendering test regains its arm_ci marker, and the uv-run Isaac Sim PhysX Cartpole install test no longer skips on aarch64.
The 6.2.0-alpha.12 bump pinned the image's x86_64 manifest digest instead of its manifest list, so the arm64 base build pulled an amd64 image and failed with an exec format error. Pin the list digest of the same build; x86 still resolves to the identical amd64 image.
The job-level continue-on-error marked every run successful even when the arm64 build or tests failed, so a broken ARM CI looked green in the run list. The check is not required, so a failure still does not block merges.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
Isaac Lab Review Bot
The ARM CI restoration is narrowly scoped: it switches the Isaac Sim pin to a multi-architecture manifest, makes ARM job failures visible, and re-enables two ARM test paths. One minor style fix is needed in the workflow header comment.
- Design and architecture: Image selection remains centralized in
.github/workflows/config.yaml, while ARM test selection continues to use the established pytest marker mechanism. Removing job-levelcontinue-on-erroris consistent with the workflow's stated role as a visible but non-required check. - API: No public Python API, import path, signature, task ID, CLI option, configuration schema, or serialized format changes. The interface-shaped edits are limited to CI workflow behavior and pytest routing.
- Compatibility and deprecation: Breaking changes: none identified. The patch changes CI image resolution, check reporting, and ARM test execution without altering supported user-facing contracts.
- Implementation: The removed aarch64 skip also removes its now-unused
platformimport, while the existinguvavailability guard remains intact. The restoredarm_cimodule marker matches the routing pattern used by adjacent kit-less rendering tests, and both touched source packages include.skipchangelog fragments. - Style consistency: The manifest comment records useful platform-selection intent, and the stale explanation for omitting the ARM marker was removed. However,
.github/workflows/arm-ci.ymlline 18 exceeds the surrounding help block's established wrapping width and should be reflowed for consistency. - Test quality: The two changed test files widen platform execution rather than adding test cases: the Shadow Hand rendering test is routed back to ARM CI, and the Isaac Sim PhysX cartpole test no longer skips aarch64. No new functions, parameter rows, fixtures, scene builds, or test-only seams were introduced, and the ARM paths provide platform-specific coverage distinct from existing x86 execution.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
| # arm-ci is currently informational (continue-on-error), but it is wired to be | ||
| # promotable to a REQUIRED status check: it triggers on every PR and gates the | ||
| # arm-ci is not a required check, so a failure reports red without blocking the | ||
| # merge. It is wired to be promotable to one: it triggers on every PR and gates the |
There was a problem hiding this comment.
🔵 Suggestion · Style Consistency — Comment exceeds block wrap width
The surrounding help block (lines 9-24) wraps at or below 80 columns; this rewritten line runs to 83 and breaks the block's visual alignment. Re-wrap the two sentences so the added lines match the existing width.
There was a problem hiding this comment.
Rewrapped the header paragraph to ≤80 columns in 1ec2ba3.
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| # no arm_ci marker: intermittently stalls the arm64 runner mid-file (renderer hang after a few | ||
| # passing cases, killed at the per-file timeout); x86 kitless coverage remains in place | ||
| pytestmark = pytest.mark.arm_ci |
There was a problem hiding this comment.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
The shadow-hand arm_ci re-enable moved out of this PR into the stacked #8230, where ARM CI is re-run repeatedly on DGX Spark before it merges. First Spark run: the file passed (3 passed, 2 motion-vector cases skipped) — run 36812221963.
Comments Outside DiffThese findings could not be posted inline.
|
Keep this change to the ARM CI build fix; the shadow-hand rendering and Isaac Sim PhysX Cartpole re-enables land separately after repeated Spark runs.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
Isaac Sim exits on aarch64 unless /lib/aarch64-linux-gnu/libgomp.so.1 loads before the copy bundled with torch, so Kit-based tests such as test_operational_space failed at import in ARM CI. Add it to /etc/ld.so.preload on arm64, as the image already does for the Carbonite shim because python.sh resets LD_PRELOAD.
The ovstage variants of the arm_ci kit-less rendering tests fail image comparison on arm64 and fail on x86 develop as well, where they run post-merge only. Deselect them in both ARM test steps so ARM CI reports the legacy path the required x86 job covers.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
|
Re the Greptile P1 on |
Isaac Sim's aarch64 check reads the LD_PRELOAD variable, not /etc/ld.so.preload, so the preload did not stop test_operational_space from exiting at import.
isaacsim exits at import on aarch64 unless /lib/aarch64-linux-gnu/libgomp.so.1 is listed in LD_PRELOAD, so the Kit-based arm_ci tests failed before running. Pass it to the test container on aarch64 hosts, matching the install tests' aarch64 environment.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
The CLI is the layer that already preloads libgomp for every Python it launches, including ARM CI's pytest; fix it there instead of in the CI test runner.
isaacsim exits at import on aarch64 unless each system /lib/*/libgomp.so.1 path is listed verbatim in LD_PRELOAD. The CLI preloaded the bare libgomp.so.1 soname, which the check rejects, so venv-based runs such as ARM CI's Kit tests still exited. Prepend the full system paths instead.
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
…k pool (#8229) (#8242) # Description Backports #8229 onto `release/3.0.0`. This restores ARM CI on the expanded DGX Spark pool by pinning the multi-architecture Isaac Sim image manifest, reporting ARM CI failures as failed workflow runs, skipping the known-failing `ovstage` ARM rendering cases, and preloading the system `libgomp.so.1` path for aarch64 CLI launches. This is an exact, conflict-free cherry-pick of the merged source change. The original regression test and `isaaclab` changelog fragment are included unchanged. ## Type of change - Bug fix (non-breaking change which fixes an issue) - Release backport ## Release backport - [x] Not applicable: this PR targets `release/3.0.0` directly. ## Screenshots Not applicable. ## Validation - Exact-patch validation passed: the source and backport patch IDs both resolve to `ac7312366230803ebddd56fc65e0a5d3e96e5e62`. - Focused CLI regression test passed: `1 passed`. - Release-aware changelog validation passed against the current upstream `release/3.0.0` tip. - All repository pre-commit hooks passed over the backport range. - ARM image and workflow behavior were validated on DGX Spark in #8229 ([arm64 image build](https://github.com/isaac-sim/IsaacLab/actions/runs/36812221963), [ARM Build & Test](https://github.com/isaac-sim/IsaacLab/actions/runs/36817584603)). - The project-managed `uv run isaaclab -f` entry point is unavailable on this macOS ARM64 host because the lockfile supports Linux and Windows only; the equivalent direct pre-commit run passed. ## Checklist - [x] I have read and understood the contribution guidelines. - [x] I have run the applicable formatting and lint checks. - [x] Documentation changes are not required for this CI/runtime fix. - [x] My changes generate no new warnings. - [x] I have validated the changed behavior with focused coverage and applied the test-audit criteria. - [x] I have included the required `isaaclab` changelog fragment. - [x] The original contributor is already listed in `CONTRIBUTORS.md`. Original PR: #8229 Cherry-picked merge commit: `1341561896f4f9b650c1f837547d4eb544c2d780` Backport commit: `6d062631084ce8e681cd023ea187f295248156b9` Co-authored-by: hujc <jichuanh@nvidia.com>
Review Map
Summary
Description
1. Changes
.github/workflows/config.yaml: pinlatest-develop@sha256:e78f716a1cae…, the manifest list of the same 6.2.0-alpha.12 build. [CI] Bump Isaac Sim image to 6.2.0-alpha.12 #8097 pinned its x86_64 childa6c53e29d6f1, so every arm64 build failed withInvalidBaseImagePlatform … pulled with platform "linux/amd64"→exec format error(e.g. run 36811233504). x86 resolves to the identicala6c53e29d6f1image..github/workflows/arm-ci.yml: drop the job-levelcontinue-on-error. It marked the workflow run successful whileBuild & Testfailed, so the broken build above looked green in the run list.isaaclab/cli/utils.py: on aarch64, preload the systemlibgomp.so.1by its full/lib/*/path instead of the bare soname.isaacsimexits at import unless each such path is listed verbatim inLD_PRELOAD(it never reads/etc/ld.so.preload), and the CLI launches every Python process, including ARM CI's pytest, sotest_operational_space.pyexited before running (aarch_preload_checking()→SystemExit, run 36812221963). Same failure for anyone running the CLI from a virtual environment on a Spark. Passing that path to the test container made thearm_cistep pass (run 36816068195); the fix now lives in the CLI, with a regression test intest_install.py..github/workflows/arm-ci.yml: deselectovstagecases in both ARM test steps. The ovstage variants of thearm_cikit-less rendering tests fail image comparison on arm64 (rgb diff 1.99 % / 7.04 % / 25.10 % over 1.5–3 % thresholds) and fail on every x86 develop run, where ovstage is post-merge only; the legacy cases pass.The two ARM tests disabled for real failures (shadow-hand kit-less rendering, Isaac Sim PhysX Cartpole) are re-enabled separately in #8230 (ARM test re-enables), after repeated DGX Spark runs.
2. Runner capacity
ARM CI ran p50 31 / p90 45 min; at the July trigger rate it adds about 20% utilization. The
ARM CIworkflow, disabled manually on 08-05, was re-enabled separately (repository setting).3. Test plan
Build base image (linux/arm64)passed)Build & Testgreen on this PR (run 36817584603;test_operational_space.pynow imports, but its tests stay deselected on ARM because a laterpytestmark = pytest.mark.integrationoverrides itsarm_cimarker)Type of change
Release backport
developChecklist
Docker and GPU tests run on demand. Push the commits you want tested, then
comment
run-cion the pull request.pre-commitchecks withuv run isaaclab --formatsource/<pkg>/changelog.d/for every touched packageCONTRIBUTORS.mdor my name already exists there