Skip to content

[CI] Drop shadow-hand kitless rendering test from arm-ci to stop intermittent hangs - #6581

Merged
kellyguo11 merged 1 commit into
isaac-sim:developfrom
hujc7:jichuanh/disable-shadow-hand-kitless-arm
Jul 22, 2026
Merged

kellyguo11 merged 1 commit into
isaac-sim:developfrom
hujc7:jichuanh/disable-shadow-hand-kitless-arm

Conversation

@hujc7

@hujc7 hujc7 commented Jul 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

1. Summary

Removes the arm_ci marker from test_rendering_shadow_hand_kitless.py. Since the test landed on arm-ci (2026-07-10, #5698) it has been implicated in 15 of 81 concluded arm-ci runs (~19%) and 15 of that job's 18 failures — including 3 failures on develop itself — typically stalling mid-file after several passing render cases until the per-file timeout kills the job. x86 coverage is unchanged: the file still runs in rendering-correctness-kitless with its full backend matrix.

2. Failure data (all concluded runs since 2026-07-10)

Job Concluded Failures Implicating this test
arm-ci (blocking on PRs) 81 18 15 (3 on develop)
rendering-correctness-kitless x86 (continue-on-error on PRs) 85 32 10 (3 timeouts, 7 file failures)

The instability is not arm-exclusive — x86 hits the same file at roughly half the rate — but the x86 job is advisory on PRs while arm-ci is blocking, making this test the dominant source of spurious red arm-ci checks. test_rendering_dexsuite_kuka_homo_kitless.py shows the same signature on arm-ci at a much lower rate (2 of 18 failures) and stays enabled for now.

3. Test plan

  • Pre-commit clean (./isaaclab.sh -f)
  • arm-ci green on this PR with the file no longer collected

4. Out of scope

Root-causing the renderer stall (all observed hangs sit in the ovrtx segment of the parametrized cases, between per-case env create/teardown cycles); the two remaining arm_ci-marked rendering files (cartpole, kuka-homo) stay enabled.

The test intermittently stalls the arm64 runner mid-file: several
parametrized render cases pass, then the renderer hangs and the file
is killed at the per-file timeout, failing the arm-ci job. The same
signature hit unrelated branches within the same hour while develop
passed, so the instability is platform-side, not change-related.

Keep the test running in the x86 rendering-correctness-kitless job,
which retains full backend coverage including ovphysx parameters that
arm-ci already excluded.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 17, 2026
@hujc7
hujc7 marked this pull request as ready for review July 21, 2026 16:36
@hujc7
hujc7 requested a review from a team July 21, 2026 16:36
@greptile-apps

greptile-apps Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR removes the pytest.mark.arm_ci marker from test_rendering_shadow_hand_kitless.py to stop it from blocking arm64 CI runs, where it was responsible for ~83% of all arm-ci failures since it was added. x86 coverage is unaffected — the test continues to run in the rendering-correctness-kitless job.

  • Test marker removed: pytestmark drops pytest.mark.arm_ci; an inline comment documents the renderer stall root cause.
  • Changelog fragment added: disable-shadow-hand-kitless-arm.skip follows the existing convention for test-only/CI-only entries (.skip extension matches several other fragments in changelog.d/).

Confidence Score: 5/5

Safe to merge — a one-line marker removal with no impact on test logic or x86 coverage.

The change removes a single pytest marker and adds a comment plus a changelog fragment. No test logic, no application code, and no x86 coverage is affected. The .skip changelog extension matches the existing convention for test-only entries.

No files require special attention.

Important Files Changed

Filename Overview
source/isaaclab_tasks/test/core/test_rendering_shadow_hand_kitless.py Removes pytest.mark.arm_ci from pytestmark; adds inline comment explaining the arm64 renderer stall issue. No logic changes.
source/isaaclab_tasks/changelog.d/disable-shadow-hand-kitless-arm.skip New changelog fragment with .skip extension, consistent with other test-only/CI-only entries in the same directory.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[test_rendering_shadow_hand_kitless.py] -->|Before PR| B["pytestmark = [isaacsim_ci, arm_ci]"]
    A -->|After PR| C["pytestmark = [isaacsim_ci]"]
    B --> D[Collected by arm-ci job]
    B --> E[Collected by rendering-correctness-kitless x86]
    C --> F[NOT collected by arm-ci job]
    C --> G[Collected by rendering-correctness-kitless x86]
    D -->|~19% of runs| H[Intermittent hang / timeout]
    G --> I[Full backend matrix coverage retained]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[test_rendering_shadow_hand_kitless.py] -->|Before PR| B["pytestmark = [isaacsim_ci, arm_ci]"]
    A -->|After PR| C["pytestmark = [isaacsim_ci]"]
    B --> D[Collected by arm-ci job]
    B --> E[Collected by rendering-correctness-kitless x86]
    C --> F[NOT collected by arm-ci job]
    C --> G[Collected by rendering-correctness-kitless x86]
    D -->|~19% of runs| H[Intermittent hang / timeout]
    G --> I[Full backend matrix coverage retained]
Loading

Reviews (1): Last reviewed commit: "Remove arm_ci marker from shadow-hand ki..." | Re-trigger Greptile

@kellyguo11
kellyguo11 merged commit d8e1523 into isaac-sim:develop Jul 22, 2026
42 checks passed
@hujc7 hujc7 mentioned this pull request Jul 23, 2026
3 tasks done
matthewtrepte pushed a commit to matthewtrepte/IsaacLab that referenced this pull request Aug 4, 2026
…rmittent hangs (isaac-sim#6581)

# Description

## 1. Summary

Removes the `arm_ci` marker from
`test_rendering_shadow_hand_kitless.py`. Since the test landed on arm-ci
(2026-07-10, isaac-sim#5698) it has been implicated in 15 of 81 concluded arm-ci
runs (~19%) and 15 of that job's 18 failures — including 3 failures on
`develop` itself — typically stalling mid-file after several passing
render cases until the per-file timeout kills the job. x86 coverage is
unchanged: the file still runs in `rendering-correctness-kitless` with
its full backend matrix.

## 2. Failure data (all concluded runs since 2026-07-10)

| Job | Concluded | Failures | Implicating this test |
|---|---|---|---|
| `arm-ci` (blocking on PRs) | 81 | 18 | 15 (3 on `develop`) |
| `rendering-correctness-kitless` x86 (continue-on-error on PRs) | 85 |
32 | 10 (3 timeouts, 7 file failures) |

The instability is not arm-exclusive — x86 hits the same file at roughly
half the rate — but the x86 job is advisory on PRs while `arm-ci` is
blocking, making this test the dominant source of spurious red arm-ci
checks. `test_rendering_dexsuite_kuka_homo_kitless.py` shows the same
signature on arm-ci at a much lower rate (2 of 18 failures) and stays
enabled for now.

## 3. Test plan

- [x] Pre-commit clean (`./isaaclab.sh -f`)
- [x] arm-ci green on this PR with the file no longer collected

## 4. Out of scope

Root-causing the renderer stall (all observed hangs sit in the ovrtx
segment of the parametrized cases, between per-case env create/teardown
cycles); the two remaining `arm_ci`-marked rendering files (cartpole,
kuka-homo) stay enabled.
hujc7 added a commit to hujc7/IsaacLab that referenced this pull request Oct 1, 2026
…s on ARM

Both were disabled on ARM for real failures: the shadow-hand kit-less rendering file
stalled mid-run on arm64 (isaac-sim#6581), and Isaac Sim PhysX training failed at environment
creation on aarch64 with an invalidated simulation view. Restore the arm_ci marker and
drop the aarch64 skip so DGX Spark runs both again.
hujc7 added a commit to hujc7/IsaacLab that referenced this pull request Oct 1, 2026
…s on ARM

Both were disabled on ARM for real failures: the shadow-hand kit-less rendering file
stalled mid-run on arm64 (isaac-sim#6581), and Isaac Sim PhysX training failed at environment
creation on aarch64 with an invalidated simulation view. Restore the arm_ci marker and
drop the aarch64 skip so DGX Spark runs both again.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants