Repository navigation

[CI] Multi-GPU pytest: dynamic work-stealing + shutdown-hang stack capture - #5875

Closed

hujc7 wants to merge 77 commits into

isaac-sim:developfrom

hujc7:jichuanh/multi-gpu-dynamic-sharding

Closed

[CI] Multi-GPU pytest: dynamic work-stealing + shutdown-hang stack capture#5875
hujc7 wants to merge 77 commits into
isaac-sim:developfrom
hujc7:jichuanh/multi-gpu-dynamic-sharding

Commits on May 26, 2026
- Enable FabricFrameView cuda:1 tests in multi-GPU CI
  hujc7
  committed
  d9dccba
Commits on May 28, 2026
- Multi-GPU pytest framework + P0 migration
  hujc7
  committed
  30667b8
- Expand multi-GPU pytest scope and auto-discover opt-in files
  hujc7
  committed
  880b4f5
- Install pytest in multi-GPU pytest workflow
  hujc7
  committed
  d6b2934
- Install flaky/pytest-mock; expand SKIP; accept pytest exit 5
  hujc7
  committed
  f1fa016
- Expand multi-GPU SKIP to cover cuda:1 hangs
  hujc7
  committed
  042c409
- Honor ISAACLAB_SIM_DEVICE env in AppLauncher; lift cuda:1 SKIPs
  hujc7
  committed
  1546520
- Restore SKIPs for cuda:1 failures with infra/upstream causes
  hujc7
  committed
  feb8e21
- Add changelog fragment for multi-GPU CI helpers
  hujc7
  committed
  6eeda64
- Convert multi-GPU pytest to ECR-pulled docker image
  hujc7
  committed
  e3d328b
- Fix docker entrypoint + pytest --ignore in multi-GPU workflow
  hujc7
  committed
  3c5248d
- Explicitly pull ECR image before docker run
  hujc7
  committed
  4ff85f4
- Re-auth to ECR before pulling image
  hujc7
  committed
  85baf4f
- Use fresh DOCKER_CONFIG with credsStore disabled for ECR pull
  hujc7
  committed
  9ff909d
- Run docker container as host uid:gid
  hujc7
  committed
  c45604c
- Point HOME / XDG_CACHE_HOME at a writable tmp dir
  hujc7
  committed
  6706690
- Add ovphysx tests to SKIP (module-level skipif + exit-1 wrapper)
  hujc7
  committed
  328878a
- TEMP: force run_docker_tests=false while iterating multi-GPU CI
  hujc7
  committed
  665f0c3
- Refactor multi-GPU pytest to use run-package-tests action
  hujc7
  committed
  1b5036f
- Add build job before multi-GPU test to populate ECR exact-tag
  hujc7
  committed
  f66b101
- Revert "TEMP: force run_docker_tests=false while iterating multi-GPU CI"
  hujc7
  committed
  5d29bb0
- TEMP: re-disable heavy CI + retest fabric on docker path
  hujc7
  committed
  3c985eb
- Add FabricFrameView cuda:1 tests to multi-GPU CI scope
  hujc7
  committed
  bdcb527
- TEMP: re-apply run_docker_tests=false until PR 5823 lands
  hujc7
  committed
  d6d69c4
- Move cuda_test_devices to isaaclab.test.utils
  hujc7
  committed
  ea919b7
- Move SKIPs into test files via cuda_test_devices(skip=...)
  hujc7
  committed
  61a9daa
- Enable all cuda:1-skipped tests to revalidate on docker path
  hujc7
  committed
  0118ea7
- Re-skip the cuda:1 tests still broken on docker path
  hujc7
  committed
  9befb6a
- Wrap long line in PhysX no-friction skip reason
  hujc7
  committed
  c37b0f9
- Fix Newton/Warp init-order on cuda:1 (issue #5132)
  hujc7
  committed
  101a47a
Commits on May 29, 2026
- ATTEMPT: parallelize multi-GPU pytest across all non-default GPUs
  hujc7
  committed
  9cb9b32
- Run shards in parallel on one runner (drop matrix)
  hujc7
  committed
  9dc722d
- Fix parallel-shard race + missing env vars by mirroring run-tests setup
  hujc7
  committed
  52da4a7
- Install pytest test-deps in each parallel shard container
  hujc7
  committed
  3941b42
- Stream per-shard output live (tagged) + raise diag timeout to 45m
  hujc7
  committed
  612a019
- Fix broadcast: give each shard its own round-robin file slice
  hujc7
  committed
  ae7ddad
- Add changelog fragments for newton fix and test-only packages
  hujc7
  committed
  b99bc97
- Rework test device selection into scope/budget masks
  hujc7
  committed
  a81e3ad
- Rename test_devices env-var constant to runtime devices
  hujc7
  committed
  b2bb7a1
- Add per-test run time and per-file device to test summary
  hujc7
  committed
  1547a9c
Commits on May 30, 2026
- Simplify test_devices to scope/runtime intersection
  hujc7
  committed
  a1fa279
- Add dynamic work-stealing across multi-GPU shards
  hujc7
  committed
  1e3697f
- Merge remote-tracking branch 'origin/develop' into jichuanh/multi-gpu-dynamic-sharding
  hujc7
  committed
  d513281
- Skip Newton/PhysX jacobian tests on non-default GPUs under sharding
  hujc7
  committed
  c8c04a8
- Ignore spurious SIGHUP in per-file test subprocess
  hujc7
  committed
  084ce63
- Block SIGHUP via signal mask in test subprocess
  hujc7
  committed
  319f0ae
- Run device-parametrized articulation tests on the shard GPU
  hujc7
  committed
  f9603ff
- Add temporary SIGHUP-sender probe for multi-GPU CI
  hujc7
  committed
  b21df58
- Gate fragile test_articulation off the multi-GPU lane
  hujc7
  committed
  f1762a5
- Merge remote-tracking branch 'origin/develop' into jichuanh/multi-gpu-dynamic-sharding
  hujc7
  committed
  5295e33
- Run jacobian tests on multi-GPU shards
  hujc7
  committed
  162ac6a
- Re-gate test_articulation off multi-GPU lane
  hujc7
  committed
  3060db5
- Replace workflow exclusion list with file-level marker
  hujc7
  committed
  74d1ad1
- TEMP: re-enable test_articulation on multi-GPU + bump fd limit
  hujc7
  committed
  e2bfa1b
- Revert "TEMP: re-enable test_articulation on multi-GPU + bump fd limit"
  hujc7
  committed
  5464c1e
- TEMP: 2-shard cap workaround for Kit lifecycle bug
  hujc7
  committed
  e7d1af7
- Revert "TEMP: 2-shard cap workaround for Kit lifecycle bug"
  hujc7
  committed
  4bc8b33
- Honor device kwarg over sim_cfg.device in build_simulation_context
  hujc7
  committed
  3d3f136
- Re-enable test_articulation in multi-GPU CI
  hujc7
  committed
  3bd01c4
Commits on May 31, 2026
Commits on Jun 3, 2026

Commits

Commits on May 26, 2026

Commits on May 28, 2026

Commits on May 29, 2026

Commits on May 30, 2026

Commits on May 31, 2026

Commits on Jun 3, 2026