Skip to content

[Bug Report] Deformable Object Tutorial Crashes on Multi-GPU with "illegal memory access" in headless mode #3117

Open
Open
[Bug Report] Deformable Object Tutorial Crashes on Multi-GPU with "illegal memory access" in headless mode#3117
Assignees
hujc7
Labels
bugSomething isn't working

Description

@Rahuboy
Rahuboy
opened on Aug 7, 2025

Describe the bug

Running the official deformable object tutorial (scripts/tutorials/01_assets/run_deformable_object.py) in headless mode on a multi-GPU system crashes (illegal memory access) during initialization of the physics scene. The bug appears to be specific to multiple GPUs as it does not occur when the simulation is constrained to a single GPU.

Steps to reproduce

  1. On a multi-GPU system with a fresh installation of Isaac Lab, run the unmodified deformable object tutorial in headless mode.
  2. python run_deformable_object.py --headless will reproduce the crash

Error

[Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/CudaKernels.cu: 2338
[Error] [omni.physx.tensors.plugin] Failed to fetch soft body nodalPositions
Traceback (most recent call last):
  File "/home/<USER>/codes/playground/isaaclab-demos/run_deformable_object.py", line 188, in <module>
    main()
  File "/home/<USER>/codes/playground/isaaclab-demos/run_deformable_object.py", line 179, in main
    sim.reset()
...
Exception: Failed to get soft body simulation mesh nodal positions from backend

System Info

Describe the characteristic of your environment:

  • Commit: 24a78b6
  • Isaac Sim Version: 4.5.0
  • OS: Ubuntu 22.04.5 LTS
  • GPU: 8x NVIDIA L40S
  • GPU Driver: 550.90.12

Additional context

The simulation succeeds under the following conditions:

  1. The environment is constrained to a single GPU (e.g., using CUDA_VISIBLE_DEVICES=0).
  2. No problems encountered with livestreaming.

Checklist

  • I have checked that there is no similar issue in the repo (required)
  • [] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

  • The unmodified run_deformable_object.py script runs to completion on a multi-GPU system without crashing.

Activity

  1. RandomOakForest commented on Aug 9, 2025

    @RandomOakForest
    RandomOakForest
    on Aug 9, 2025
    Collaborator

    Thank you for posting this. The team is looking into it.

  2. RandomOakForest
    added
    bugSomething isn't working
    on Aug 9, 2025

  3. RandomOakForest commented on Aug 12, 2025

    @RandomOakForest
    RandomOakForest
    on Aug 12, 2025
    Collaborator

    Following up, is this something you could try in Isaac Sim 5.0 using Isaac Lab 2.2.0? Could you also share a larger error log? as to see if there aren't any other previous errors or warnings that may have triggered this. Thanks!

  4. sardanian commented on Sep 9, 2025

    @sardanian
    sardanian
    on Sep 9, 2025

    I have this same issue and am running on latest versions of both isaac sim and isaac labs. I don't use headless mode though. I am just running it without any arguments.

    This is my log.
    You can see the error start at: ~27 seconds

    2025-09-09T22:20:10Z [2,042ms] [Warning] [gpu.foundation.plugin] PCIe link width current (8) and maximum (16) for device 0 don't match.
    2025-09-09T22:20:10Z [2,042ms] [Warning] [gpu.foundation.plugin] PCIe link width current (8) and maximum (16) for device 1 don't match.
    [2.565s] [ext: omni.kit.pipapi-0.0.0] startup
    [2.567s] [ext: omni.kit.pip_archive-0.0.0] startup
    [2.568s] [ext: omni.pip.compute-1.6.2] startup
    [2.568s] [ext: omni.materialx.libs-1.0.7] startup
    [2.577s] [ext: omni.pip.cloud-1.3.6] startup
    [2.580s] [ext: omni.isaac.core_archive-2.6.1] startup
    [2.580s] [ext: omni.usd.config-1.0.6] startup
    [2.587s] [ext: omni.gpucompute.plugins-0.0.0] startup
    [2.587s] [ext: omni.usd.libs-1.0.1] startup
    [2.666s] [ext: omni.mdl-56.0.3] startup
    [2.696s] [ext: omni.iray.libs-0.0.0] startup
    [2.701s] [ext: omni.mdl.neuraylib-0.2.12] startup
    [2.704s] [ext: omni.kit.usd.mdl-1.1.4] startup
    [2.797s] [ext: omni.kit.telemetry-0.5.2] startup
    [2.854s] [ext: omni.isaac.ml_archive-3.0.2] startup
    [2.855s] [ext: omni.kit.loop-isaac-1.3.7] startup
    [2.856s] [ext: omni.kit.test-2.0.1] startup
    [2.917s] [ext: omni.appwindow-1.1.10] startup
    [3.677s] [ext: omni.kit.renderer.core-1.1.0] startup
    [9.525s] [ext: omni.kit.renderer.capture-0.0.0] startup
    [9.528s] [ext: omni.kit.renderer.imgui-2.0.5] startup
    [9.711s] [ext: omni.ui-2.26.18] startup
    [9.721s] [ext: omni.kit.mainwindow-1.0.3] startup
    [9.723s] [ext: carb.audio-0.1.0] startup
    [9.729s] [ext: omni.uiaudio-1.0.0] startup
    [9.730s] [ext: omni.kit.uiapp-0.0.0] startup
    [9.730s] [ext: omni.usd.schema.metrics.assembler-107.3.1] startup
    [9.734s] [ext: omni.usd.schema.audio-0.0.0] startup
    [9.737s] [ext: omni.usd_resolver-1.0.0] startup
    [9.741s] [ext: omni.usd.core-1.5.3] startup
    [9.743s] [ext: omni.usd.schema.render_settings.rtx-0.0.0] startup
    [9.761s] [ext: omni.usd.schema.physx-107.3.18] startup
    [9.793s] [ext: omni.usd.schema.semantics-0.0.0] startup
    [9.796s] [ext: omni.usd.schema.omni_lens_distortion-0.0.0] startup
    [9.796s] [ext: omni.usd.schema.geospatial-0.0.0] startup
    [9.799s] [ext: omni.usd.schema.anim-0.0.0] startup
    [9.820s] [ext: omni.kit.window.popup_dialog-2.0.24] startup
    [9.825s] [ext: usdrt.scenegraph-7.6.1] startup
    [9.863s] [ext: omni.timeline-1.0.14] startup
    [9.865s] [ext: omni.resourcemonitor-107.0.1] startup
    [9.867s] [ext: omni.activity.core-1.0.3] startup
    [9.869s] [ext: omni.kit.widget.nucleus_connector-2.0.1] startup
    [9.872s] [ext: omni.hydra.usdrt_delegate-7.5.1] startup
    [9.878s] [ext: omni.kit.actions.core-1.0.0] startup
    [9.881s] [ext: omni.graph.exec-0.9.6] startup
    [9.881s] [ext: omni.kit.commands-1.4.10] startup
    [9.886s] [ext: omni.kit.usd_undo-0.1.8] startup
    [9.887s] [ext: omni.kit.exec.core-0.13.4] startup
    [9.889s] [ext: omni.kit.audiodeviceenum-1.0.2] startup
    [9.890s] [ext: omni.hydra.scene_delegate-0.3.4] startup
    [9.895s] [ext: omni.usd-1.13.10] startup
    2025-09-09T22:20:18Z [9,881ms] [Warning] [omni.usd.audio] failed to subscribe to stage events
    [9.921s] [ext: omni.kit.asset_converter-4.1.4] startup
    [9.932s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
    [9.937s] [ext: isaacsim.robot.schema-3.5.1] startup
    [9.945s] [ext: omni.usd.schema.omni_sensors-0.0.0] startup
    [9.946s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
    [9.953s] [ext: omni.kit.property.adapter.core-1.0.2] startup
    [9.957s] [ext: omni.kit.menu.core-1.1.2] startup
    [9.958s] [ext: omni.kit.property.adapter.fabric-1.0.3] startup
    [9.960s] [ext: omni.kit.usd.layers-2.2.10] startup
    [9.970s] [ext: omni.kit.clipboard-1.0.5] startup
    [9.971s] [ext: omni.kit.widget.context_menu-1.2.5] startup
    [9.972s] [ext: omni.kit.hotkeys.core-1.3.10] startup
    [9.978s] [ext: omni.kit.widget.path_field-2.0.11] startup
    [9.980s] [ext: omni.kit.context_menu-1.8.6] startup
    [9.982s] [ext: omni.index.libs-380600.5717.0] startup
    [9.982s] [ext: omni.kit.widget.browser_bar-2.0.10] startup
    [9.983s] [ext: omni.kit.widget.options_menu-1.1.6] startup
    [9.988s] [ext: omni.kit.helper.file_utils-0.1.9] startup
    [9.990s] [ext: omni.index-1.0.1] startup
    [9.991s] [ext: omni.hydra.rtx.shadercache.vulkan-1.0.0] startup
    [9.993s] [ext: omni.kit.widget.options_button-1.0.3] startup
    [9.994s] [ext: omni.kit.widget.filebrowser-2.12.2] startup
    [10.000s] [ext: omni.volume-0.5.2] startup
    [10.002s] [ext: omni.ujitso.client-0.0.0] startup
    [10.003s] [ext: omni.kit.notification_manager-1.0.10] startup
    [10.005s] [ext: omni.hydra.rtx-1.0.0] startup
    2025-09-09T22:20:18Z [9,995ms] [Warning] [omni.log] Source: omni.hydra was already registered.
    [10.038s] [ext: omni.ui.scene-1.11.4] startup
    [10.043s] [ext: omni.kit.widget.searchable_combobox-1.0.6] startup
    [10.044s] [ext: omni.kit.window.filepicker-2.13.3] startup
    [10.061s] [ext: omni.kit.widget.searchfield-1.1.8] startup
    [10.063s] [ext: omni.kit.widget.settings-1.2.3] startup
    [10.066s] [ext: omni.kit.menu.utils-2.0.3] startup
    [10.073s] [ext: omni.kit.window.file_exporter-1.0.33] startup
    [10.075s] [ext: omni.kit.widget.filter-1.1.4] startup
    [10.076s] [ext: omni.hydra.engine.stats-1.0.3] startup
    [10.078s] [ext: omni.kit.window.preferences-1.8.0] startup
    [10.083s] [ext: omni.kit.widget.stage-3.1.4] startup
    [10.099s] [ext: omni.kit.window.file_importer-1.1.18] startup
    [10.100s] [ext: omni.kit.raycast.query-1.1.0] startup
    [10.105s] [ext: omni.kit.viewport.registry-104.0.6] startup
    [10.106s] [ext: omni.kit.viewport.legacy_gizmos-1.0.18] startup
    [10.108s] [ext: omni.kit.material.library-2.0.7] startup
    [10.117s] [ext: omni.kit.property.adapter.usd-1.0.2] startup
    [10.118s] [ext: omni.kit.viewport.scene_camera_model-1.0.6] startup
    [10.123s] [ext: omni.kit.hydra_texture-1.4.5] startup
    [10.126s] [ext: omni.kit.window.drop_support-1.0.5] startup
    [10.127s] [ext: omni.kit.widget.viewport-107.1.3] startup
    [10.132s] [ext: omni.kit.primitive.mesh-1.0.17] startup
    [10.137s] [ext: omni.kit.viewport.window-107.0.12] startup
    [10.157s] [ext: omni.kit.manipulator.transform-107.0.0] startup
    [10.164s] [ext: omni.kit.stage_template.core-1.1.22] startup
    [10.165s] [ext: omni.kit.widget.toolbar-2.0.1] startup
    [10.174s] [ext: omni.kit.viewport.utility-1.1.2] startup
    [10.175s] [ext: omni.kit.window.content_browser_registry-0.0.6] startup
    [10.176s] [ext: omni.kit.widget.highlight_label-1.0.3] startup
    [10.176s] [ext: omni.kit.window.file-2.0.5] startup
    [10.181s] [ext: omni.kit.manipulator.tool.snap-1.5.13] startup
    [10.187s] [ext: omni.kit.manipulator.selector-1.1.3] startup
    [10.189s] [ext: omni.kit.window.property-1.12.1] startup
    [10.193s] [ext: omni.kit.window.content_browser-3.1.1] startup
    [10.206s] [ext: omni.kit.viewport.manipulator.transform-107.0.4] startup
    [10.209s] [ext: omni.kit.manipulator.viewport-107.0.1] startup
    [10.211s] [ext: omni.kit.property.usd-4.5.11] startup
    [10.225s] [ext: omni.kit.manipulator.selection-106.0.1] startup
    [10.227s] [ext: omni.fabric.commands-1.1.6] startup
    [10.230s] [ext: omni.kvdb-107.3.18] startup
    [10.232s] [ext: omni.kit.manipulator.prim.core-107.0.8] startup
    [10.242s] [ext: omni.convexdecomposition-107.3.18] startup
    [10.245s] [ext: omni.physx.foundation-107.3.18] startup
    [10.247s] [ext: omni.localcache-107.3.18] startup
    [10.248s] [ext: omni.kit.manipulator.prim.fabric-107.0.4] startup
    [10.250s] [ext: omni.kit.viewport.menubar.core-107.1.5] startup
    [10.273s] [ext: omni.kit.viewport.actions-107.0.2] startup
    [10.279s] [ext: omni.usdphysics-107.3.18] startup
    [10.282s] [ext: omni.kit.widget.prompt-1.0.7] startup
    [10.283s] [ext: omni.kit.manipulator.prim.usd-107.0.3] startup
    [10.284s] [ext: omni.kit.viewport.menubar.display-107.0.3] startup
    [10.286s] [ext: omni.physx.cooking-107.3.18] startup
    [10.289s] [ext: omni.inspect-1.0.2] startup
    [10.290s] [ext: omni.kit.widget.layers-1.8.6] startup
    [10.307s] [ext: omni.kit.manipulator.prim-107.0.0] startup
    [10.307s] [ext: omni.debugdraw-0.1.4] startup
    [10.311s] [ext: omni.physx-107.3.18] startup
    [10.320s] [ext: omni.physics-107.3.18] startup
    [10.323s] [ext: omni.graph.core-2.184.3] startup
    [10.326s] [ext: omni.usdphysics.ui-107.3.18] startup
    [10.353s] [ext: omni.physx.commands-107.3.18] startup
    [10.359s] [ext: omni.physics.stageupdate-107.3.18] startup
    [10.361s] [ext: isaacsim.core.deprecation_manager-0.2.7] startup
    [10.362s] [ext: omni.physx.ui-107.3.18] startup
    [10.395s] [ext: isaacsim.core.version-2.0.6] startup
    [10.396s] [ext: omni.physics.physx-107.3.18] startup
    [10.397s] [ext: isaacsim.storage.native-1.2.8] startup
    [10.399s] [ext: omni.usd.metrics.assembler-107.3.1] startup
    [10.405s] [ext: omni.kit.numpy.common-0.1.2] startup
    [10.408s] [ext: omni.physics.tensors-107.3.18] startup
    [10.413s] [ext: omni.warp.core-1.7.1] startup
    [10.803s] [ext: omni.usd.metrics.assembler.physics-107.3.18] startup
    [10.806s] [ext: omni.isaac.dynamic_control-2.0.7] startup
    2025-09-09T22:20:19Z [10,785ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.
    [10.813s] [ext: omni.physx.tensors-107.3.18] startup
    [10.842s] [ext: isaacsim.core.utils-3.4.5] startup
    [10.845s] [ext: isaacsim.core.simulation_manager-1.3.2] startup
    [11.290s] [ext: omni.kit.usd.collect-2.4.5] startup
    [11.294s] [ext: isaacsim.core.prims-0.5.1] startup
    [11.330s] [ext: omni.kit.usdz_export-1.0.9] startup
    [11.333s] [ext: omni.graph.tools-1.79.2] startup
    [11.371s] [ext: omni.graph.action_core-1.1.7] startup
    [11.374s] [ext: omni.graph-1.141.2] startup
    [11.440s] [ext: omni.kit.stage_templates-2.0.0] startup
    [11.443s] [ext: omni.graph.action_nodes-1.50.4] startup
    [11.447s] [ext: isaacsim.core.api-4.6.10] startup
    [11.533s] [ext: omni.kit.tool.asset_importer-4.3.2] startup
    [11.546s] [ext: omni.graph.action-1.130.0] startup
    [11.548s] [ext: omni.kit.selection-0.1.6] startup
    [11.551s] [ext: omni.ui_query-1.1.8] startup
    [11.553s] [ext: omni.kit.stagerecorder.core-107.0.3] startup
    [11.558s] [ext: omni.kit.widget.text_editor-1.1.1] startup
    [11.561s] [ext: omni.kit.ui_test-1.3.6] startup
    [11.565s] [ext: omni.kit.stagerecorder.ui-107.0.1] startup
    [11.571s] [ext: omni.kit.window.extensions-1.4.26] startup
    [11.584s] [ext: omni.graph.visualization.nodes-2.1.3] startup
    [11.600s] [ext: omni.kit.stagerecorder.bundle-105.0.2] startup
    [11.602s] [ext: isaacsim.gui.components-1.1.9] startup
    [11.620s] [ext: isaacsim.gui.menu-2.3.17] startup
    [11.911s] [ext: isaacsim.app.about-2.0.11] startup
    [11.918s] [ext: isaacsim.asset.importer.mjcf-2.5.8] startup
    [11.992s] [ext: semantics.schema.editor-2.0.1] startup
    2025-09-09T22:20:20Z [11,975ms] [Warning] [pxr.Semantics] pxr.Semantics is deprecated - please use Semantics instead
    [12.005s] [ext: semantics.schema.property-2.0.1] startup
    [12.008s] [ext: isaacsim.core.cloner-1.4.9] startup
    [12.013s] [ext: omni.graph.image.core-0.6.1] startup
    [12.015s] [ext: omni.kit.widget.stage_icons-1.0.8] startup
    [12.018s] [ext: omni.graph.image.nodes-1.3.1] startup
    [12.019s] [ext: omni.kit.window.stage-2.6.1] startup
    [12.027s] [ext: omni.kit.widget.graph-2.0.0] startup
    [12.040s] [ext: omni.kit.menu.create-2.0.1] startup
    [12.044s] [ext: omni.syntheticdata-0.6.13] startup
    [12.073s] [ext: isaacsim.util.debug_draw-3.1.0] startup
    [12.084s] [ext: omni.kit.graph.delegate.default-1.2.2] startup
    [12.092s] [ext: omni.kit.scripting-107.3.1] startup
    [12.123s] [ext: isaacsim.sensors.physx-2.2.27] startup
    [12.131s] [ext: omni.kit.graph.editor.core-1.5.3] startup
    [12.148s] [ext: omni.kit.graph.usd.commands-1.3.1] startup
    [12.151s] [ext: omni.kit.widget.material_preview-1.0.16] startup
    [12.157s] [ext: isaacsim.replicator.behavior-1.1.14] startup
    [12.159s] [ext: isaacsim.simulation_app-2.9.2] startup
    [12.161s] [ext: omni.anim.curve.core-1.3.1] startup
    [12.186s] [ext: omni.graph.scriptnode-1.50.0] startup
    [12.191s] [ext: omni.kit.window.material_graph-1.8.23] startup
    [12.259s] [ext: omni.sensors.nv.common-2.7.0-coreapi] startup
    [12.276s] [ext: omni.kit.viewport.menubar.settings-107.0.3] startup
    [12.286s] [ext: omni.graph.nodes-1.170.10] startup
    [12.299s] [ext: omni.graph.ui_nodes-1.50.5] startup
    [12.304s] [ext: omni.sensors.net-0.4.0-coreapi] startup
    [12.309s] [ext: omni.sensors.nv.materials-1.6.0-coreapi] startup
    [12.319s] [ext: omni.videoencoding-0.1.2] startup
    [12.324s] [ext: omni.sensors.nv.wpm-2.5.0-coreapi] startup
    [12.327s] [ext: omni.warp-1.7.1] startup
    [12.350s] [ext: omni.kit.manipulator.camera-106.0.4] startup
    [12.371s] [ext: omni.sensors.nv.ids-1.5.0-coreapi] startup
    [12.381s] [ext: omni.sensors.nv.lidar-2.7.0-coreapi] startup
    [12.396s] [ext: omni.sensors.nv.radar-2.8.0-coreapi] startup
    [12.416s] [ext: omni.replicator.core-1.12.16] startup
    2025-09-09T22:20:21Z [13,252ms] [Warning] [omni.replicator.core.scripts.extension] No material configuration file, adding configuration to material settings directly.
    [13.284s] [ext: omni.kit.property.audio-1.0.16] startup
    [13.293s] [ext: omni.kit.property.camera-1.0.10] startup
    [13.300s] [ext: omni.kit.property.light-1.0.12] startup
    [13.328s] [ext: omni.kit.property.geometry-2.0.4] startup
    [13.355s] [ext: isaacsim.core.nodes-3.2.16] startup
    [13.400s] [ext: isaacsim.gui.property-1.1.3] startup
    [13.451s] [ext: omni.kit.property.render-1.2.1] startup
    [13.583s] [ext: omni.hydra.scene_api-0.1.2] startup
    [13.711s] [ext: isaacsim.test.docstring-1.1.0] startup
    [13.781s] [ext: isaacsim.core.experimental.utils-0.2.0] startup
    [13.852s] [ext: omni.kit.widget.zoombar-1.0.6] startup
    [13.987s] [ext: omni.kit.property.transform-1.5.12] startup
    [14.288s] [ext: omni.kit.property.material-1.11.9] startup
    [14.985s] [ext: isaacsim.core.experimental.prims-0.6.2] startup
    [15.236s] [ext: omni.kit.browser.core-2.3.13] startup
    [15.273s] [ext: omni.kit.property.bundle-1.4.1] startup
    [15.291s] [ext: isaacsim.sensors.rtx-15.5.0] startup
    [15.325s] [ext: omni.kit.viewport.menubar.camera-107.0.3] startup
    [15.375s] [ext: isaacsim.robot.surface_gripper-3.2.6] startup
    [15.391s] [ext: omni.kit.browser.folder.core-1.10.9] startup
    [15.501s] [ext: isaacsim.sensors.physics-0.3.27] startup
    [15.516s] [ext: omni.kit.tool.collect-2.2.18] startup
    [15.568s] [ext: omni.kit.menu.stage-1.2.7] startup
    [15.670s] [ext: isaacsim.robot.manipulators-3.3.5] startup
    [15.924s] [ext: omni.kit.viewport.menubar.lighting-107.3.1] startup
    [16.359s] [ext: omni.kit.viewport.menubar.render-107.0.9] startup
    [16.648s] [ext: isaacsim.robot.policy.examples-4.1.11] startup
    [16.658s] [ext: isaacsim.examples.browser-0.1.12] startup
    [16.692s] [ext: isaacsim.asset.importer.urdf-2.4.19] startup
    [16.765s] [ext: omni.kit.window.console-1.1.4] startup
    [16.806s] [ext: omni.physx.demos-107.3.18] startup
    [17.115s] [ext: omni.kit.property.physx-107.3.18] startup
    [17.199s] [ext: omni.rtx.window.settings-0.6.19] startup
    [17.247s] [ext: omni.ocio-0.1.1] startup
    [17.252s] [ext: omni.replicator.replicator_yaml-2.0.11] startup
    [17.309s] [ext: omni.asset_validator.core-0.25.4] startup
    [17.428s] [ext: omni.physx.vehicle-107.3.18] startup
    [17.458s] [ext: omni.rtx.settings.core-0.6.5] startup
    [17.464s] [ext: omni.kit.window.script_editor-2.0.1] startup
    [17.470s] [ext: isaacsim.robot.wheeled_robots-4.0.23] startup
    [17.478s] [ext: omni.kit.window.toolbar-2.0.0] startup
    [17.487s] [ext: omni.physx.asset_validator-107.3.18] startup
    [17.509s] [ext: omni.physx.camera-107.3.18] startup
    [17.542s] [ext: omni.physx.cct-107.3.18] startup
    [17.569s] [ext: omni.physx.graph-107.3.18] startup
    [17.619s] [ext: omni.physx.supportui-107.3.18] startup
    [17.683s] [ext: omni.physx.telemetry-107.3.18] startup
    [17.688s] [ext: omni.kit.window.status_bar-0.1.8] startup
    [17.696s] [ext: omni.usd.metrics.assembler.ui-107.3.1] startup
    [17.717s] [ext: isaacsim.asset.browser-1.3.19] startup
    [17.726s] [ext: isaacsim.core.throttling-2.1.10] startup
    [17.730s] [ext: omni.kit.ui.actions-1.0.5] startup
    [17.734s] [ext: omni.physx.bundle-107.3.18] startup
    [17.736s] [ext: isaacsim.sensors.camera-1.3.1] startup
    [17.759s] [ext: isaaclab-0.45.7] startup
    [18.214s] [ext: isaaclab_assets-0.2.2] startup
    [18.648s] [ext: isaaclab_tasks-0.10.46] startup
    [21.200s] [ext: omni.kit.menu.common-2.0.0] startup
    [21.203s] [ext: isaaclab_rl-0.2.4] startup
    [21.204s] [ext: isaaclab_mimic-1.0.12] startup
    [21.205s] [ext: isaaclab.python-2.2.0] startup
    [21.208s] Simulation App Starting
    2025-09-09T22:20:29Z [21,389ms] [Warning] [usdrt.population.plugin] using high frequency span is disabled
    2025-09-09T22:20:29Z [21,420ms] [Warning] [omni.fabric.plugin] Warning: attribute viewportHandle not found for bucket id 9
    2025-09-09T22:20:29Z [21,584ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [carb::dictionary::ISerializer v1.1] (plugin: carb.dictionary.serializer-json.plugin), by client: carb.scenerenderer-rtx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
    [21.791s] app ready
    2025-09-09T22:20:32Z [23,828ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span is disabled
    2025-09-09T22:20:32Z [23,828ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span is disabled
    2025-09-09T22:20:32Z [23,829ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span with attrs is disabled
    2025-09-09T22:20:32Z [24,127ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span is disabled
    2025-09-09T22:20:32Z [24,127ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span is disabled
    [24.621s] Simulation App Startup Complete
    [24.706s] [ext: omni.physx.fabric-107.3.18] startup
    2025-09-09T22:20:36Z [27,733ms] [Error] [omni.physx.tensors.plugin] CUDA error: an illegal memory access was encountered: ../../../extensions/runtime/source/omni.physx.tensors/plugins/gpu/CudaKernels.cu: 2655
    2025-09-09T22:20:36Z [27,733ms] [Error] [omni.physx.tensors.plugin] Failed to fetch soft body nodalPositions
    2025-09-09T22:20:36Z [27,763ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span with attrs is disabled
    2025-09-09T22:20:36Z [27,763ms] [Warning] [usdrt.hydra.fabric_scene_delegate.plugin] using high frequency span with attrs is disabled
    2025-09-09T22:20:36Z [27,773ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
    2025-09-09T22:20:36Z [27,773ms] [Error] [carb.cudainterop.plugin] Failed to create CUDA stream.
    2025-09-09T22:20:36Z [27,773ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
    2025-09-09T22:20:36Z [27,773ms] [Error] [carb.cudainterop.plugin] Failed to create CUDA stream.
    2025-09-09T22:20:36Z [28,583ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:36Z [28,583ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:36Z [28,585ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:36Z [28,585ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:36Z [28,589ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:36Z [28,589ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:36Z [28,596ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:36Z [28,596ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [28,701ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [28,701ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [28,701ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [28,701ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [28,705ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [28,705ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [28,713ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [28,713ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [28,807ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [28,807ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [29,121ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [29,121ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:37Z [29,470ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:37Z [29,470ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:38Z [29,692ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:38Z [29,692ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:38Z [30,217ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:38Z [30,217ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:38Z [30,299ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:38Z [30,299ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:38Z [30,409ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:38Z [30,409ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:39Z [30,868ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:39Z [30,868ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:40Z [31,867ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:40Z [31,867ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:40Z [31,990ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
    2025-09-09T22:20:40Z [31,990ms] [Error] [carb.graphics-vulkan.plugin] vkCreateRayTracingPipelinesKHR failed.
    2025-09-09T22:20:40Z [31,991ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,991ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,992ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,992ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,992ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,993ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,993ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,993ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,994ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,994ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,994ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,995ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,995ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,995ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,996ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,996ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,996ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,997ms] [Error] [omni.kit.app.plugin] ASSERTION FAILED (continuing execution but crash may occur): ../../../source/plugins/rtx.psodb/PsoRaytracing.cpp(1494): void rtx::psodb::PsoRaytracing::buildPipeline()(): Assertion (collection.get() != nullptr) failed.
    2025-09-09T22:20:40Z [31,997ms] [Error] [carb.graphics-vulkan.plugin] No Collection/Library can be nullptr with eCollection as compileType.
    2025-09-09T22:20:40Z [31,997ms] [Error] [gpu.foundation.plugin] Failed to create the raytracing pipeline.

  5. RandomOakForest commented on Sep 15, 2025

    @RandomOakForest
    RandomOakForest
    on Sep 15, 2025
    Collaborator

    Following up on this, have you tried the recommended driver versions listed here? Can you confirm which driver you are using and if you still see this issue?

  6. AntoineRichard
    assigned
    maxkra15
    on Sep 3, 2026

  7. maxkra15 commented on Sep 22, 2026

    @maxkra15
    maxkra15
    on Sep 22, 2026
    Contributor

    Thanks @Rahuboy and @sardanian. Current 3.0 experiences pin the renderer to one GPU by default via #5933, which may remove the multi-GPU interaction seen here. That change did not modify the PhysX deformable path, however, and PhysX–Fabric GPU interop remains enabled by default. Could someone with two or more GPUs retest the current deformable tutorial on Isaac Lab 3.0 / Isaac Sim 6.1? If it still reproduces, please also try ISAACLAB_FABRIC_USE_GPU_INTEROP=0 and share the driver/GPU details plus the first CUDA error. Keeping this open until we have a current-stack result.

  8. maxkra15
    assigned
    hujc7
    and unassigned
    maxkra15
    on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions