[CI] Run ARM CI through the x86 test action and re-enable ARM tests by hujc7 · Pull Request #8233 · isaac-sim/IsaacLab

hujc7 · 2026-10-01T06:11:51Z

Summary

ARM CI restores compatible Warp and OVRTX caches and runs Newton/OVPhysX marker coverage through the shared x86 test action. Camera smoke coverage stays enabled, and previously skipped ARM checks are restored.

Description

Route ARM marker tests through run-package-tests, with explicit linux/arm64 passed through image lookup, dependency hashing and fallback builds. PRs restore ARM caches; protected-branch pushes publish them. Test files run in four slots like the x86 test jobs; rendering files still run one at a time.
Run the existing state-training, camera-rendering and camera-training Cartpole probes as their own ARM step after the marker tests, reusing the restored Warp and OVRTX cache directories that run-package-tests now exposes as outputs. Installation CI now covers state training separately, so the camera probes remain necessary. The step checks the probe report explicitly, since run-tests records failures in the report rather than its exit code.
Restore shadow-hand kitless rendering and operational-space controller coverage on ARM. Carry the Isaac Sim PhysX training re-enable into the replacement uv workspace test rather than restoring a deleted installation test, and preserve the required ARM system OpenMP preload when the probe adds the Isaac Sim shim. Remove the Kuka visual-material ARM marker to match x86 rendering selection. Supersedes [CI] Re-enable the shadow-hand rendering and Isaac Sim PhysX Cartpole tests on ARM #8230.
Build the image in the runner's Docker daemon when no registry cache is configured, writing new layers with zstd. The disposable docker-container builder exported and reloaded the ~29 GB image every build and discarded its layer cache. Only ARM takes this path today: x86 callers resolve ECR and keep the docker-container builder. The action is part of the dependency-cache hash, so the first build after merge misses that cache once.
Count kitless rendering files that use rendering_test_utils as renderers, so their cold shader compiles no longer run concurrently.

Validation

uv run --frozen --extra test isaaclab -f: all checks passed.
Docker-storage and file-scheduler coverage: 19 tests passed. Actionlint passed for ARM and x86 workflows; 50 local action calls validated against their declared inputs.
A real subprocess regression check failed before the preload fix and passes afterward for ARM with/without the shim and x86 with the shim.
On 7b73dafe1fe, ARM CI passed 71 marker cases (3 expected skips) and all 3 state/camera smoke probes. ARM installation passed 3/3 attempts on three NVIDIA GB10 runners, with all 8 contracts including PhysX executed each time; x86 installation also passed all 8 contracts.
On the final head 01a3ff4737d, the x86 Docker suite passed: 30 jobs succeeded and 3 post-merge-only jobs were skipped as expected. Hosted pre-commit, Tools Tests, Docs, kitless image validation, installation and multi-GPU CI also passed.

ARM job time, all on NVIDIA GB10 with driver 580.159.03. A temporary commit (since reverted) let PR runs save caches for this measurement:

Run	Build	Tests	Total
Before, cold	8m30	16m52	25m42
Warm, new runner (3 runs)	3m06–3m23	5m22–5m38	9m01–9m06
Warm, runner with layer cache	0m56	5m42	6m57
Final head, new runner	3m31	9m22	13m11

The first four rows also included the Newton RTX shader-cache fix, now split into Redirect the Newton RTX visualizer shader cache #8504; the final head does not. Warm per-file times: Cartpole 54 s (was 297 s), Kuka 51 s, Shadow Hand 41 s, operational space 118 s alongside them. Newton RTX takes 254 s without Redirect the Newton RTX visualizer shader cache #8504 and 27 s with it. Every run passed 71 marker cases (3 expected skips) and 3 smoke probes. Image export on the runners' containerd store fell from 243 s with gzip to 44–46 s with zstd. Protected-branch publication remains unmeasured before merge.

Type of change

Bug fix (non-breaking CI and test coverage changes).

Release backport

Backport this pull request to the active release branch after it merges into develop

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with uv run isaaclab --format
Documentation describing the smoke-test caller matches the current workflow; rendered documentation is unaffected
No new project warning sites were added; CI still reports existing dependency and action deprecations
I have validated the changed GPU behavior with focused coverage and applied the test-audit criteria to test changes
I have added changelog fragments for the touched source packages
My name already exists in CONTRIBUTORS.md

hujc7 · 2026-10-01T06:12:04Z

run-ci

hujc7 · 2026-10-01T06:21:28Z

run-ci

hujc7 · 2026-10-01T06:36:44Z

run-ci

hujc7 · 2026-10-01T06:46:01Z

run-ci

hujc7 · 2026-10-01T06:46:34Z

run-ci

hujc7 · 2026-10-01T07:29:36Z

run-ci

ARM CI called run-tests directly in two containers split by OVPhysX, without the Warp and OVRTX shader caches x86 restores. Let run-package-tests use an image already built on the runner and pass a pytest marker through, then run the arm_ci tests in one step with the x86 kit-less settings: OVPhysX and OVRTX cases together, ovstage deselected, caches restored on pull requests and published on pushes.

Installation Tests (ARM) already runs the same three Cartpole probes on the Spark pool, and x86's Docker CI has no such step, so ARM CI's separate smoke step only repeated them.

…s on ARM Both were disabled on ARM for real failures: the shadow-hand kit-less rendering file stalled mid-run on arm64 (isaac-sim#6581), and Isaac Sim PhysX training failed at environment creation on aarch64 with an invalidated simulation view. Restore the arm_ci marker and drop the aarch64 skip so DGX Spark runs both again.

test_operational_space.py set pytestmark to arm_ci and then reassigned it to integration, so ARM CI's -m arm_ci deselected all of its tests. Declare both markers in one list at the top of the file.

No x86 job selects this file: the task package jobs exclude rendering tests and the kit-less rendering job lists its files explicitly without it. Drop its arm_ci marker so ARM CI runs the same rendering files as x86.

hujc7 · 2026-10-01T08:39:00Z

run-ci

This reverts commit 3db7d91.

…align-x86

hujc7 · 2026-10-12T21:40:04Z

run-ci

greptile-apps · 2026-10-12T22:22:26Z

Confidence Score: 4/5

[High impact] Restructures CI build and test infrastructure for ARM support.

The PR appears safe to merge, with non-blocking improvements to cache publication order and rendering-file detection.

Findings

Smoke caches miss publication ▶
Helper tests occupy rendering lane ▶

Summary

ARM CI now uses the shared package-test action with four file slots and architecture-aware image lookup. It restores or publishes Warp and OVRTX caches, shares their directories with Cartpole smoke probes, and restores ARM test coverage.

ARM CI runs marker tests and Cartpole smoke probes through shared test actions.
ARM marker coverage includes shadow-hand rendering and operational-space controller tests.
ARM installation checks run the Isaac Sim workspace training test.
ARM image builds use the runner's local builder when no registry cache is set.
The test scheduler recognizes files that use rendering helpers as renderer jobs.

Intentional change acknowledged by hujc7: removing the Kuka visual-material ARM marker matches x86 rendering selection.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Build ARM image] --> B[Restore ARM caches]
  B --> C[Run marker tests in four slots]
  C --> D[Publish caches on pushes]
  D --> E[Run Cartpole smoke with shared directories]
  E --> F[Check smoke report]
  F --> G[Upload reports]

_{Reviews (1) · Last reviewed commit: "Merge remote-tracking branch 'upstream/d..." · Reviewed by Greptile}

greptile-apps · 2026-10-12T22:22:29Z

+    - name: Run Cartpole smoke tests
      uses: ./.github/actions/run-tests


Smoke caches miss publication

On protected-branch pushes, run-package-tests saves both caches before this Cartpole step runs. Any cache files added by the state or camera probes miss the published snapshot, so later PRs may compile them again. Run these probes before the action saves the caches, as the PR description intends.

greptile-apps · 2026-10-12T22:22:29Z

+    return (
+        "enable_cameras=True" in test_content
+        or "ovrtx" in test_content.lower()
+        or "rendering_test_utils" in test_content


Helper tests occupy rendering lane

Importing rendering_test_utils does not mean a test starts a renderer. This now marks test_maybe_save_stage_golden.py as a rendering file, although it writes USD files and mocks isaaclab.sim.save_stage. The scheduler unnecessarily blocks it from running beside real rendering files and gives it the cold-shader timeout allowance. Use a narrower check and cover this helper-only test in the classifier tests.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

isaaclab-review-bot

Isaac Lab Review Bot

Needs attention: Design and architecture (1). Other reviewed categories look good.

PR description: Performance evidence supplies revisions, workloads, hardware, driver, and timings, but item 2 misstates cache ordering; smoke probes run after run-package-tests, so correct the claim or move them before cache saving.
Design and architecture: Inferring daemon export from empty cache inputs breaks documented no-ECR fallbacks on runners without containerd image storage; add an explicit builder or export-mode input.
Scope and over-engineering: No over-engineering issues.
API: API changes look good.
Compatibility and deprecation: No breaking changes identified.
Implementation: Implementation looks good.
Style consistency: Style consistency looks good.
Test quality: Test quality looks good.

Minor fixes needed. Posted 1 inline finding.

Automated review; human maintainers own approval decisions.

isaaclab-review-bot · 2026-10-12T22:25:16Z

+        # image, layer cache and uv cache mount stay on the runner. A containerd
+        # image store still compresses new layers; zstd level 1 is several times
+        # faster than the default gzip for the multi-GB venv layer.
+        if [ -n "${{ inputs.cache-from }}${{ inputs.cache-to }}" ]; then


🟡 Warning · Design Architecture — Builder choice inferred from cache inputs

Empty cache inputs also occur on ecr-build-push-pull's documented no-ECR fallback, so those builds now export via --output type=image on the default builder, which fails without a containerd image store. Gate on an explicit input.

release/3.0.0 still has the cli/, uv_pip/ and uv_run/ install_ci layout, so the conflicting #8233 hunks have no direct target. The proposed resolution instead ported develop's workspace fixtures, test_workspace.py and a --run-gpu gate that nothing on this branch passes. Its module-level tomllib import broke x86 test collection on Python 3.10, and the gate skipped existing ARM training tests. Restore conftest.py, drop test_workspace.py, and carry the actual change: re-enable Isaac Sim PhysX training on aarch64 in the uv_run test that test_workspace.py replaced on develop. That test already preserves the ARM OpenMP preload when it adds the Carbonite shim.

#8233 dropped installation CI from the probe's caller list because develop's install suite no longer runs it. On release/3.0.0 the uv pip wheel install test still executes this file through the cartpole_smoke_script fixture, so name all three callers.

Backports #8233 to `release/3.0.0`. The original cherry-pick conflicted. An NVIDIA inference model proposed this resolution, and deterministic validation confirmed that it changes no paths outside the original PR. Because conflict resolution cannot be certified as an exact patch replay, this PR is intentionally a draft and requires release-maintainer review. | Field | Commit | |---|---| | Original merged change | `8519738d0c984a2a5bfddc161f60f59606c3b394` | | Release base used | `8ce1df03bf5256f0d8fc2e76d021c2d186e36a02` | | Proposed backport | `6fc6b0968833ca1dd1fe5e37c98ba2c5701a99b1` | --------- Co-authored-by: hujc <jichuanh@nvidia.com> Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>

github-actions Bot added the infrastructure label Oct 1, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 1, 2026

github-actions Bot added the isaac-lab Related to Isaac Lab team label Oct 1, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 1, 2026

hujc7 changed the title ~~[CI] Run ARM CI tests through the x86 test action~~ [CI] Run ARM CI through the x86 test action and re-enable two ARM tests Oct 1, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 1, 2026

hujc7 changed the title ~~[CI] Run ARM CI through the x86 test action and re-enable two ARM tests~~ [CI] Run ARM CI through the x86 test action and re-enable ARM tests Oct 1, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 1, 2026

hujc7 added 5 commits October 1, 2026 01:38

Drop ARM CI's duplicate Cartpole training smoke

a018260

Installation Tests (ARM) already runs the same three Cartpole probes on the Spark pool, and x86's Docker CI has no such step, so ARM CI's separate smoke step only repeated them.

Run the operational-space controller tests on ARM

adccfae

test_operational_space.py set pytestmark to arm_ci and then reassigned it to integration, so ARM CI's -m arm_ci deselected all of its tests. Declare both markers in one list at the top of the file.

Stop running the kuka visual-material kit-less rendering test on ARM

8938bf9

No x86 job selects this file: the task package jobs exclude rendering tests and the kit-less rendering job lists its files explicitly without it. Drop its arm_ci marker so ARM CI runs the same rendering files as x86.

hujc7 force-pushed the jichuanh/arm-ci-align-x86 branch from 657a8c5 to 8938bf9 Compare October 1, 2026 08:38

isaaclab-bot Bot added the ci:run-docker Trigger the on-demand Docker and GPU CI workflow label Oct 1, 2026

hujc7 added 4 commits October 12, 2026 14:37

Revert "Redirect the Newton RTX visualizer shader cache"

62226c7

This reverts commit 3db7d91.

Run ARM Cartpole smoke tests outside the shared test action

5a940d4

State which callers build in the runner daemon

d02aedd

Merge remote-tracking branch 'upstream/develop' into jichuanh/arm-ci-…

01a3ff4

…align-x86

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 12, 2026

hujc7 mentioned this pull request Oct 12, 2026

Redirect the Newton RTX visualizer shader cache #8504

Draft

5 tasks done

hujc7 marked this pull request as ready for review October 12, 2026 22:16

hujc7 requested a review from a team October 12, 2026 22:16

hujc7 requested review from AntoineRichard, StafaH, fatimaanes, kellyguo11, myurasov-nv, ooctipus and pbarejko as code owners October 12, 2026 22:16

greptile-apps Bot reviewed Oct 12, 2026

View reviewed changes

isaaclab-review-bot Bot reviewed Oct 12, 2026

View reviewed changes

StafaH approved these changes Oct 12, 2026

View reviewed changes

ooctipus approved these changes Oct 12, 2026

View reviewed changes

hujc7 enabled auto-merge (squash) October 12, 2026 23:46

kellyguo11 approved these changes Oct 13, 2026

View reviewed changes

hujc7 merged commit 8519738 into isaac-sim:develop Oct 13, 2026
61 checks passed

isaaclab-bot Bot mentioned this pull request Oct 13, 2026

[Backport] PR #8233 to release/3.0.0 #8509

Merged

hujc7 mentioned this pull request Oct 14, 2026

Improve backport conflict resolution context and models #8556

Open

		- name: Run Cartpole smoke tests
		uses: ./.github/actions/run-tests

Conversation

hujc7 commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Validation

Type of change

Release backport

Checklist

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 1, 2026

Uh oh!

hujc7 commented Oct 12, 2026

Uh oh!

greptile-apps Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

Isaac Lab Review Bot

Uh oh!

isaaclab-review-bot Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hujc7 commented Oct 1, 2026 •

edited

Loading

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading