[Bug Report] Isaaclab libcublas & libcudart bug in isaac-sim. Docker installation #6593

Describe the bug

Building and running Isaac Lab through the standard Docker workflow (docker/container.py start, unmodified docker/Dockerfile.base), ./isaaclab.sh --sim fails to load two extensions with libcublas.so.*[0-9] not found / libcudart.so.12: cannot open shared object file. The app still finishes loading ("app ready" / "Isaac Sim Full App is loaded"), but isaacsim.core.simulation_manager and isaacsim.ros2.bridge never actually start:

Steps to reproduce

git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab
python3 docker/container.py start
python3 docker/container.py enter base
./isaaclab.sh --sim

This gives the following error:

2026-07-17T18:50:04Z [1,907ms] [Error] [omni.ext._impl._internal] Failed to import python module isaacsim.core.simulation_manager from /isaac-sim/exts/isaacsim.core.simulation_manager. Error: libcublas.so.*[0-9] not found in the system path [...]
...
OSError: libcudart.so.12: cannot open shared object file: No such file or directory
...
2026-07-17T18:50:09Z [7,212ms] [Error] [omni.ext.plugin] [ext: isaacsim.ros2.bridge-4.12.4] Failed to startup python extension.
2026-07-17T18:50:09Z [7,356ms] [Error] [omni.graph.core._impl.extension] OGN node registration completed with errors: libcublas.so.*[0-9] not found in the system path [...]

Note: Tutorial "Train your first robot in Isaac Lab" scripts (e.g. ./isaaclab.sh -p scripts/skrl/train.py --task=Template-Cartpole-v0 and play.py) run without this error in the same container even though it's using isaac-sim. This might be because those scripts import torch at the top of the script, before Isaac Sim's extensions load. Once torch has successfully imported once, any later import torch from inside an extension just reuses it and the failure never surfaces. ./isaaclab.sh --sim launches the standalone Isaac Sim app directly with no user script, so isaacsim.core.simulation_manager is the first thing to import torch, and that's where it fails with above error.

After checking, I found out that cuda pip packages are not installed inside for this kit-interpreter (one that's running isaac-sim.) See the comparison of default python interpreter packages and one that's inside /isaac-sim below.

root@root:/# /isaac-sim/kit/python/bin/python3 -m pip list 2>/dev/null | grep -iE "torch|nvidia-cu"
nvidia-cublas             13.1.1.3
nvidia-cuda-cupti         13.0.85
nvidia-cuda-nvrtc         13.0.88
nvidia-cuda-runtime       13.0.96
nvidia-cudnn-cu13         9.20.0.48
nvidia-cufft              12.0.0.61
nvidia-cufile             1.15.1.6
nvidia-curand             10.4.0.35
nvidia-cusolver           12.0.4.66
nvidia-cusparse           12.6.3.3
nvidia-cusparselt-cu13    0.8.1
torch                     2.7.0+cu128
torchvision               0.22.0+cu128
root@root:/# python3 -m pip list 2>/dev/null | grep -iE "torch|nvidia-cu"
nvidia-cublas             13.1.1.3
nvidia-cublas-cu12        12.8.3.14
nvidia-cuda-cupti         13.0.85
nvidia-cuda-cupti-cu12    12.8.57
nvidia-cuda-nvrtc         13.0.88
nvidia-cuda-nvrtc-cu12    12.8.61
nvidia-cuda-runtime       13.0.96
nvidia-cuda-runtime-cu12  12.8.57
nvidia-cudnn-cu12         9.7.1.26
nvidia-cudnn-cu13         9.20.0.48
nvidia-cufft              12.0.0.61
nvidia-cufft-cu12         11.3.3.41
nvidia-cufile             1.15.1.6
nvidia-cufile-cu12        1.13.0.11
nvidia-curand             10.4.0.35
nvidia-curand-cu12        10.3.9.55
nvidia-cusolver           12.0.4.66
nvidia-cusolver-cu12      11.7.2.55
nvidia-cusparse           12.6.3.3
nvidia-cusparse-cu12      12.5.7.53
nvidia-cusparselt-cu12    0.6.3
nvidia-cusparselt-cu13    0.8.1
torch                     2.7.0+cu128
torchvision               0.22.0+cu128

System Info

Describe the characteristic of your environment:

• IsaacLab: main, Docker workflow, Dockerfile.base unmodified.
• Isaac Sim Version: 5.1.0-rc.19+release.26219.9c81211b.gl
• OS: Ubuntu 26.04
• GPU: . RTX 5090
• CUDA: 12.8
• GPU Driver: e.g. 580.159.03

Workaround

Installing the matching -cu12 packages into the kit-bundled python fixes it. This is the modification in the Dockerfile.base file:

# Fix CUDA 12 runtime lib mismatch for Isaac Sim's bundled Kit-python torch
# (see https://github.com/isaac-sim/IsaacLab/pull/4462 for a related upstream issue)
RUN /isaac-sim/kit/python/bin/python3 -m pip install \
    nvidia-cublas-cu12==12.8.3.14 \
    nvidia-cuda-cupti-cu12==12.8.57 \
    nvidia-cuda-nvrtc-cu12==12.8.61 \
    nvidia-cuda-runtime-cu12==12.8.57 \
    nvidia-cudnn-cu12==9.7.1.26 \
    nvidia-cufft-cu12==11.3.3.41 \
    nvidia-cufile-cu12==1.13.0.11 \
    nvidia-curand-cu12==10.3.9.55 \
    nvidia-cusolver-cu12==11.7.2.55 \
    nvidia-cusparse-cu12==12.5.7.53 \
    nvidia-cusparselt-cu12==0.6.3

right after

RUN --mount=type=cache,target=${DOCKER_USER_HOME}/.cache/pip \
    ${ISAACLAB_PATH}/isaaclab.sh --install

and before

# HACK: Remove install of quadprog dependency
RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip uninstall -y quadprog

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Not known now. Is this the proper fix, or is there another fix for this.