Skip to content

[Bug] Root cause mgpu problem with --/physics/fabricUseGPUInterop #6382

Open
Open
[Bug] Root cause mgpu problem with --/physics/fabricUseGPUInterop#6382
Assignees
hujc7
Labels
bugSomething isn't working

Description

@hujc7
hujc7
opened on Jul 7, 2026

Context

ISAACLAB_FABRIC_USE_GPU_INTEROP was added in

#5933 — [MGPU] App: default Kit renderer to one GPU

(merged 2026-07-07, commit 7b0fe6aae27) as an explicitly temporary workaround. Its
changelog fragment commits to removing it "after the underlying Kit/PhysX problem is
fixed". This issue tracks that root cause.

Where the workaround lives

  • .github/actions/multi-gpu/multi_gpu_host_launcher.sh:107 — passes
    -e ISAACLAB_FABRIC_USE_GPU_INTEROP=0 into the shard container.
  • .github/workflows/test-multi-gpu-pytest.yaml:262 — the Multi-GPU pytest lane that
    invokes that launcher. It runs on PRs today.
  • source/isaaclab/isaaclab/app/app_launcher.py:45,1276-1283 — translates the env var
    into --/physics/fabricUseGPUInterop=false, unless an explicit sys.argv / kit_args
    setting of the same key already exists.

Because the lane always forces the setting off, a green Multi-GPU pytest run carries no
information about the unset case.

Original symptom (2026-07-07)

From the A/B recorded in

#6300 — [STRESS / DO NOT MERGE] Validate Fabric GPU interop necessity on multi-GPU CI

(closed, never merged), on 4x A10G:

  • Negative arm (setting omitted), attempt 1: one trial hit a 120 s startup hang in
    test_simulation_context.py, passing only via the built-in retry.
  • Negative arm, attempt 2 (same SHA / image / runner): the A/B step was terminated
    after 526 s with exit 143 (SIGTERM); runner cleanup terminated an orphaned Docker
    process. No workflow cancellation, timeout, or lost-runner message.
  • Controls (5 runs, setting explicitly false): all clean, 17/17 files, zero JUnit
    failures.

That comment's own conclusion was deliberately narrow: it validated the flag as a
practical CI mitigation and did not establish Fabric interop as the sole root cause.

Retest 2026-09-03 — the failure condition no longer reproduces

Re-ran the negative condition against current develop (8ca264bd8). See the comment
below for the full method, caveats, and per-run table.

Result: 6/6 runs clean (3 negative + 3 control, alternating). Negative arm 54/54
passing, 0 failing, 0 crashed, 0 startup-hang, 0 timeout — indistinguishable from control,
and marginally faster. Zero Traceback, SIGTERM, exit 143, or device mismatch across
all 24 logs.

The arms were verified genuinely differential, so the clean negative arm is meaningful:

  • All four Kit experiences set the setting ON, so "env unset" is not "disabled":
    apps/isaaclab.python.kit:270, apps/isaaclab.python.headless.kit:174,
    apps/isaaclab.python.rendering.kit:119,
    apps/isaaclab.python.headless.rendering.kit:149.
  • Runtime read-back confirmed /physics/fabricUseGPUInterop resolves to True in the
    negative arm and False in the control arm.

This was not the CI lane. Different GPUs (4x L40 vs 4x A10G), different host (DGXC
node vs GitHub self-hosted runner), and a different image (daily
internal/isaac-lab:latest-develop plus the CI pytest-dep layer, not the lane's
per-commit isaac-lab-ci built from docker/Dockerfile.base on pinned
isaacsim@sha256:1bbd249c). The July failure was also intermittent, so a small number of
clean reps bounds the failure rate without excluding a moderate one.

Not this bug

Two nearby items look related and are not:

  • The libcarb.cudainterop.plugin.so SIGSEGV at >=5 ranks (nvbug 6565122 / 6576053) is a
    distinct defect in the Carb CUDA interop plugin, not /physics/fabricUseGPUInterop.
  • .github/workflows/test-fabric-multi-gpu.yaml covers the FabricFrameView class and is
    unrelated to Fabric GPU interop.

What would close this

Run the negative condition on the real lane: a stress PR that drops the
-e ISAACLAB_FABRIC_USE_GPU_INTEROP=0 line and dispatches Multi-GPU pytest N times on the
CI runner and the per-commit CI image. If those are clean, remove the CI override, the env
var, its docstring, and the two tests in source/isaaclab/test/app/test_env_var_launch.py.

Removing the override on the strength of a green lane that still sets it would be circular,
which is why the local retest above is offered as evidence to justify that stress run — not
as a substitute for it.

Activity

  1. hujc7
    added this to Isaac Labon Jul 7, 2026
  2. hujc7
    self-assigned this
    on Jul 7, 2026
  3. kellyguo11
    moved this to Backlog in Isaac Labon Jul 8, 2026
  4. RandomOakForest
    added
    bugSomething isn't working
    on Jul 8, 2026

  5. hujc7 commented on Sep 3, 2026

    @hujc7
    hujc7
    on Sep 3, 2026
    CollaboratorAuthor

    Retest of the negative condition — 2026-09-03

    Re-ran the exact condition that failed on 2026-07-07: the multi-GPU pytest shard fanout
    with ISAACLAB_FABRIC_USE_GPU_INTEROP unset, so Kit's configured value stands.

    Setup
    Host 4x NVIDIA L40, idle at launch
    Source develop @ 8ca264bd8
    Image nvcr.io/.../internal/isaac-lab:latest-develop (daily) + the CI pytest-dep layer
    Per run CI shard fanout reproduced: 18 discovered opt-in files seeded into the directory work queue, 3 pytest shards (cuda:1..3), queue reconciled afterwards
    Design 3 reps x 2 arms, strictly sequential, arms alternating so drift hits both equally

    Test discovery used the workflow's own device_scope_pattern logic and matched CI exactly:
    18 files, 0 skipped, including source/isaaclab/test/sim/test_simulation_context.py —
    the file that hung in July.

    Results

    run arm interop rc elapsed done queue_left inflight_left
    rep1 negative enabled 0 503s 18/18 0 0
    rep1 control disabled 0 520s 18/18 0 0
    rep2 negative enabled 0 505s 18/18 0 0
    rep2 control disabled 0 505s 18/18 0 0
    rep3 negative enabled 0 508s 18/18 0 0
    rep3 control disabled 0 518s 18/18 0 0

    Aggregated harness stats per arm (18 files x 3 reps):

    arm passing failing crashed startup hang timeout
    negative 54 0 0 0 0
    control 54 0 0 0 0

    queue_left / inflight_left are the lane's own silent-drop assertion — both zero on every
    run, so nothing was claimed-but-never-finished. Zero Traceback, SIGTERM, exit 143,
    Fatal, or device mismatch across all 24 run and shard logs. The negative arm was
    marginally faster than the control.

    Why the clean negative arm is meaningful

    A clean negative arm proves nothing if both arms were effectively identical, so that was
    checked two independent ways.

    1. All four Kit experiences set the setting ON explicitly, so "env unset" is not the same
      as "disabled": apps/isaaclab.python.kit:270,
      apps/isaaclab.python.headless.kit:174, apps/isaaclab.python.rendering.kit:119,
      apps/isaaclab.python.headless.rendering.kit:149.
    2. Runtime read-back through AppLauncher in each arm:
    arm=negative (env unset)
  RESOLVED /physics/fabricUseGPUInterop = True
  KIT_ARGS = ['--/exts/isaacsim.core.simulation_manager/enable_default_callbacks=false']

arm=control  (ISAACLAB_FABRIC_USE_GPU_INTEROP=0)
  RESOLVED /physics/fabricUseGPUInterop = False
  KIT_ARGS = ['--/physics/fabricUseGPUInterop=false',
              '--/exts/isaacsim.core.simulation_manager/enable_default_callbacks=false']

    Limits of this evidence

    • Not the CI lane. 4x L40 rather than the 4x A10G runner, a DGXC node rather than the
      GitHub self-hosted runner, and the daily internal/isaac-lab image plus a locally built
      CI pytest-dep layer rather than the lane's per-commit isaac-lab-ci image built from
      docker/Dockerfile.base on pinned isaacsim@sha256:1bbd249c.
    • The July failure was intermittent — one recovered hang and one SIGTERM across two
      attempts. Three clean negative reps bound the per-run failure rate only loosely; they
      rule out a high rate, not a moderate one. An extended negative-only series is running to
      tighten that bound and will be posted here.
    • The first launch of this harness aborted (6 runs, ModuleNotFoundError: junitparser)
      because the daily image lacks the pytest deps that CI layers on at
      .github/actions/docker-build/action.yml:199. Those aborts were discarded and the
      identical dep layer rebuilt before the runs above.

    Suggested next step

    This is evidence to justify a stress run on the real lane, not a substitute for one: a
    DO NOT MERGE PR dropping -e ISAACLAB_FABRIC_USE_GPU_INTEROP=0 from
    .github/actions/multi-gpu/multi_gpu_host_launcher.sh:107, dispatching Multi-GPU pytest N
    times. If that is clean, remove the override, the env var, its docstring, and the two tests
    in source/isaaclab/test/app/test_env_var_launch.py.

  6. hujc7 commented on Sep 3, 2026

    @hujc7
    hujc7
    on Sep 3, 2026
    CollaboratorAuthor

    Extended negative-arm series — 13/13 clean

    Follow-up to the retest above, tightening the bound on an intermittent failure rate. Same
    host, source, image, and harness; negative arm only (ISAACLAB_FABRIC_USE_GPU_INTEROP
    unset, so /physics/fabricUseGPUInterop resolves to True).

    metric value
    negative runs 13
    non-zero exit 0
    runs not completing 18/18 files 0
    runs with queue_left or inflight_left != 0 0
    elapsed min 499s, max 516s, mean 505.0s

    Aggregated harness stats across the 13 runs (18 files x 13 reps):

    passing failing crashed startup hang timeout
    234 0 0 0 0

    Failure-signature scan across all negative-arm run and shard logs — Traceback, SIGTERM,
    exit 143, Fatal, device mismatch, cudainterop, Segmentation: 0 files each.

    What this bounds

    By the rule of three, 13 consecutive clean runs put the 95% upper bound on the per-run
    failure rate at roughly 21%. For comparison, both negative attempts in the July A/B
    showed an anomaly; had the rate stayed anywhere near that, 13 clean runs would be
    vanishingly unlikely (p ~ 0.0001 at a 50% rate, ~1% at a 30% rate).

    The elapsed-time spread is also worth noting: 499-516s across 13 runs, with no outlier.
    The July symptom announced itself as a 120 s startup hang and a 526 s termination, so a
    tight, outlier-free distribution is direct evidence against a latent stall.

    Unchanged caveats

    This still is not the CI lane — 4x L40 rather than the 4x A10G runner, a DGXC node rather
    than the GitHub self-hosted runner, and the daily internal/isaac-lab image plus a locally
    built CI pytest-dep layer rather than the per-commit isaac-lab-ci image from
    docker/Dockerfile.base on pinned isaacsim@sha256:1bbd249c.

    The recommendation in the issue body is unchanged, and this series strengthens the case for
    taking it: a DO NOT MERGE stress PR dropping
    -e ISAACLAB_FABRIC_USE_GPU_INTEROP=0 from
    .github/actions/multi-gpu/multi_gpu_host_launcher.sh:107, dispatching Multi-GPU pytest
    several times on the real runner and image. If clean, remove the override, the env var, its
    docstring, and the two tests in source/isaaclab/test/app/test_env_var_launch.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

  • Isaac Lab
    Status
    Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions