[Backport] PR #8233 to release/3.0.0 by isaaclab-bot[bot] · Pull Request #8509 · isaac-sim/IsaacLab

isaaclab-bot · 2026-10-13T02:48:25Z

Backports #8233 to release/3.0.0.

The original cherry-pick conflicted. An NVIDIA inference model proposed this resolution, and deterministic validation confirmed that it changes no paths outside the original PR. Because conflict resolution cannot be certified as an exact patch replay, this PR is intentionally a draft and requires release-maintainer review.

Field	Commit
Original merged change	`8519738d0c984a2a5bfddc161f60f59606c3b394`
Release base used	`8ce1df03bf5256f0d8fc2e76d021c2d186e36a02`
Proposed backport	`6fc6b0968833ca1dd1fe5e37c98ba2c5701a99b1`

…8233) ARM CI restores compatible Warp and OVRTX caches and runs Newton/OVPhysX marker coverage through the shared x86 test action. Camera smoke coverage stays enabled, and previously skipped ARM checks are restored. 1. Route ARM marker tests through `run-package-tests`, with explicit `linux/arm64` passed through image lookup, dependency hashing and fallback builds. PRs restore ARM caches; protected-branch pushes publish them. Test files run in four slots like the x86 test jobs; rendering files still run one at a time. 2. Run the existing state-training, camera-rendering and camera-training Cartpole probes as their own ARM step after the marker tests, reusing the restored Warp and OVRTX cache directories that `run-package-tests` now exposes as outputs. Installation CI now covers state training separately, so the camera probes remain necessary. The step checks the probe report explicitly, since `run-tests` records failures in the report rather than its exit code. 3. Restore shadow-hand kitless rendering and operational-space controller coverage on ARM. Carry the Isaac Sim PhysX training re-enable into the replacement uv workspace test rather than restoring a deleted installation test, and preserve the required ARM system OpenMP preload when the probe adds the Isaac Sim shim. Remove the Kuka visual-material ARM marker to match x86 rendering selection. Supersedes #8230. 4. Build the image in the runner's Docker daemon when no registry cache is configured, writing new layers with zstd. The disposable `docker-container` builder exported and reloaded the ~29 GB image every build and discarded its layer cache. Only ARM takes this path today: x86 callers resolve ECR and keep the `docker-container` builder. The action is part of the dependency-cache hash, so the first build after merge misses that cache once. 5. Count kitless rendering files that use `rendering_test_utils` as renderers, so their cold shader compiles no longer run concurrently. 1. `uv run --frozen --extra test isaaclab -f`: all checks passed. 2. Docker-storage and file-scheduler coverage: 19 tests passed. Actionlint passed for ARM and x86 workflows; 50 local action calls validated against their declared inputs. 3. A real subprocess regression check failed before the preload fix and passes afterward for ARM with/without the shim and x86 with the shim. 4. On `7b73dafe1fe`, [ARM CI](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182263) passed 71 marker cases (3 expected skips) and all 3 state/camera smoke probes. [ARM installation](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182231) passed 3/3 attempts on three NVIDIA GB10 runners, with all 8 contracts including PhysX executed each time; x86 installation also passed all 8 contracts. 5. On the final head `01a3ff4737d`, the [x86 Docker suite](https://github.com/isaac-sim/IsaacLab/actions/runs/38312079847) passed: 30 jobs succeeded and 3 post-merge-only jobs were skipped as expected. Hosted pre-commit, Tools Tests, Docs, kitless image validation, installation and multi-GPU CI also passed. 6. ARM job time, all on NVIDIA GB10 with driver 580.159.03. A temporary commit (since reverted) let PR runs save caches for this measurement: | Run | Build | Tests | Total | | --- | ---: | ---: | ---: | | Before, [cold](https://github.com/isaac-sim/IsaacLab/actions/runs/37401182263/job/112609307602) | 8m30 | 16m52 | 25m42 | | [Warm, new runner](https://github.com/isaac-sim/IsaacLab/actions/runs/38298855064) (3 runs) | 3m06–3m23 | 5m22–5m38 | 9m01–9m06 | | Warm, runner with layer cache | 0m56 | 5m42 | 6m57 | | [Final head, new runner](https://github.com/isaac-sim/IsaacLab/actions/runs/38312048044) | 3m31 | 9m22 | 13m11 | The first four rows also included the Newton RTX shader-cache fix, now split into #8504; the final head does not. Warm per-file times: Cartpole 54 s (was 297 s), Kuka 51 s, Shadow Hand 41 s, operational space 118 s alongside them. Newton RTX takes 254 s without #8504 and 27 s with it. Every run passed 71 marker cases (3 expected skips) and 3 smoke probes. Image export on the runners' containerd store fell from 243 s with gzip to 44–46 s with zstd. Protected-branch publication remains unmeasured before merge. 1. Bug fix (non-breaking CI and test coverage changes). - [x]  Backport this pull request to the active release branch after it merges into `develop` - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/develop/source/refs/contributing.html) - [x] I have run the `pre-commit` checks with `uv run isaaclab --format` - [x] Documentation describing the smoke-test caller matches the current workflow; rendered documentation is unaffected - [x] No new project warning sites were added; CI still reports existing dependency and action deprecations - [x] I have validated the changed GPU behavior with focused coverage and applied the test-audit criteria to test changes - [x] I have added changelog fragments for the touched source packages - [x] My name already exists in `CONTRIBUTORS.md` (cherry picked from commit 8519738)

isaaclab-bot · 2026-10-13T02:48:26Z

run-ci

greptile-apps · 2026-10-13T02:55:10Z

Confidence Score: 4/5

[High impact] Modifies CI build and test infrastructure for ARM support.

Restore GPU training coverage in installation CI before merging.

Findings

GPU training checks stop running ▶
New wheel installer goes unused ▶
Caches save before smoke tests ▶

Summary

This backport moves ARM tests onto the shared package-test action.

ARM CI shares its test, cache, and smoke-check path.
ARM CI selects its kitless rendering and controller tests.
Installation CI checks both supported uv workspace runtimes.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Build ARM image] --> B[Restore caches]
  B --> C[Run ARM marker tests]
  C --> D[Save caches on push]
  D --> E[Run Cartpole smoke with shared caches]
  E --> F[Check smoke report]
  G[Installation CI exposes GPU] --> H[Pytest lacks run-gpu option]
  H --> I[GPU training checks skipped]

_{Reviews (1) · Last reviewed commit: "[CI] Run ARM CI through the x86 test act..." · Reviewed by Greptile}

greptile-apps · 2026-10-13T02:55:13Z

+        warp-cache-host-dir: ${{ steps.tests.outputs.warp-cache-host-dir }}
+        ovrtx-shader-cache-host-dir: ${{ steps.tests.outputs.ovrtx-shader-cache-host-dir }}


Caches save before smoke tests

The new cache snapshots are saved inside run-package-tests, before the Cartpole smoke step runs. Any kernels or shaders first built by the smoke tests miss those snapshots, so later ARM runs can rebuild that work. Publish the final caches after the smoke step so later runs can reuse it too.

isaaclab-review-bot

Isaac Lab Review Bot

Needs attention: Implementation (1), Style consistency (2). Other reviewed categories look good.

PR description: The body omits resulting ARM behavior and support for “deterministic validation” and “several times faster”; add commands, revisions, ARM hardware, cache state, baseline/candidate build times, and test results.
Design and architecture: Architecture looks good.
Scope and over-engineering: No over-engineering issues.
API: API changes look good.
Compatibility and deprecation: No breaking changes identified.
Implementation: The default-off --run-gpu gate skips all GPU-marked install_ci tests; remove the gate and route GPU coverage through pytest marker selection.
Style consistency: Fixture ownership and caller documentation are inconsistent: checkout duplicates isaaclab_root, and smoke-probe prose conflicts with install_ci’s fixture contract.
Test quality: Test quality looks good.

Minor fixes needed. Posted 3 inline findings.

Automated review; human maintainers own approval decisions.

release/3.0.0 still has the cli/, uv_pip/ and uv_run/ install_ci layout, so the conflicting #8233 hunks have no direct target. The proposed resolution instead ported develop's workspace fixtures, test_workspace.py and a --run-gpu gate that nothing on this branch passes. Its module-level tomllib import broke x86 test collection on Python 3.10, and the gate skipped existing ARM training tests. Restore conftest.py, drop test_workspace.py, and carry the actual change: re-enable Isaac Sim PhysX training on aarch64 in the uv_run test that test_workspace.py replaced on develop. That test already preserves the ARM OpenMP preload when it adds the Carbonite shim.

AntoineRichard · 2026-10-13T11:23:49Z

run-ci

#8233 dropped installation CI from the probe's caller list because develop's install suite no longer runs it. On release/3.0.0 the uv pip wheel install test still executes this file through the cartpole_smoke_script fixture, so name all three callers.

AntoineRichard · 2026-10-13T11:45:29Z

@hujc7 @kellyguo11 I modified this PR since it looked quite different from what we had originally in develop. These backporting agents are not very reliable. Is there anyway to use Claude or Codex for this?

kellyguo11 · 2026-10-15T14:53:05Z

is this one ready? do we still want this on the release branch?

AntoineRichard · 2026-10-15T14:58:55Z

@kellyguo11 the original PR looked quite important. @hujc7 can you confirm we need this? (original PR: #8233)

Update: it's ready, and I think we should have it on release. It's CI/test-only and release ARM CI is green without it, but @hujc7 opted it in, and it re-enables ARM coverage for 3.0 (Isaac Sim training from the committed lockfile, OSC, shadow-hand kitless rendering), which all passes now. Worth taking while 3.0 is still taking backports.

kellyguo11 · 2026-10-15T17:35:03Z

run-ci

github-actions Bot added isaac-lab Related to Isaac Lab team infrastructure labels Oct 13, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 13, 2026

kellyguo11 marked this pull request as ready for review October 13, 2026 02:49

kellyguo11 requested a review from a team October 13, 2026 02:49

kellyguo11 requested review from AntoineRichard, StafaH, fatimaanes, hujc7, kellyguo11, myurasov-nv, ooctipus and pbarejko as code owners October 13, 2026 02:49

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

isaaclab-review-bot Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread source/isaaclab/test/install_ci/conftest.py Outdated

Comment thread source/isaaclab/test/install_ci/conftest.py Outdated

Comment thread source/isaaclab/test/install_ci/misc/cartpole_training_smoke.py Outdated

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 13, 2026

hujc7 mentioned this pull request Oct 14, 2026

Improve backport conflict resolution context and models #8556

Open

isaaclab-bot Bot added the ci:run-docker Trigger the on-demand Docker and GPU CI workflow label Oct 15, 2026

isaaclab-bot Bot removed the ci:run-docker Trigger the on-demand Docker and GPU CI workflow label Oct 15, 2026

kellyguo11 merged commit 027c0a1 into release/3.0.0 Oct 15, 2026
58 checks passed

kellyguo11 deleted the backport/release/3.0.0/pr-8233 branch October 15, 2026 19:37

Navigation Menu

Repository navigation

[Backport] PR #8233 to release/3.0.0 - #8509

[Backport] PR #8233 to release/3.0.0#8509
kellyguo11 merged 3 commits into
release/3.0.0from
backport/release/3.0.0/pr-8233

isaaclab-bot Bot commented Oct 13, 2026

Uh oh!

isaaclab-bot Bot commented Oct 13, 2026

Uh oh!

greptile-apps Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Oct 13, 2026

Uh oh!

isaaclab-review-bot Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AntoineRichard commented Oct 13, 2026

Uh oh!

AntoineRichard commented Oct 13, 2026

Uh oh!

kellyguo11 commented Oct 15, 2026

Uh oh!

AntoineRichard commented Oct 15, 2026 •

edited

Loading

Uh oh!

kellyguo11 commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		warp-cache-host-dir: ${{ steps.tests.outputs.warp-cache-host-dir }}
		ovrtx-shader-cache-host-dir: ${{ steps.tests.outputs.ovrtx-shader-cache-host-dir }}

Conversation

isaaclab-bot Bot commented Oct 13, 2026

Uh oh!

isaaclab-bot Bot commented Oct 13, 2026

Uh oh!

greptile-apps Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

Isaac Lab Review Bot

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AntoineRichard commented Oct 13, 2026

Uh oh!

AntoineRichard commented Oct 13, 2026

Uh oh!

kellyguo11 commented Oct 15, 2026

Uh oh!

AntoineRichard commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kellyguo11 commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greptile-apps Bot commented Oct 13, 2026 •

edited

Loading

AntoineRichard commented Oct 15, 2026 •

edited

Loading