Repository navigation
[Bug Report] AppLauncher SIGTERM handler does not terminate Python workers #6530
Description
Describe the bug
AppLauncher currently registers _abort_signal_handle_callback as the
SIGTERM handler. The callback calls SimulationApp.close() and then returns.
Installing a Python signal handler replaces the operating system's default
SIGTERM termination behavior. Returning from the callback therefore allows
Python execution to continue instead of terminating the process.
During distributed training shutdown, this can leave a worker alive after the
torchrun parent and TCPStore have exited. The surviving worker's
ProcessGroupNCCL heartbeat thread continues accessing the closed TCPStore and
repeatedly prints:
[rank1] TCPStore.cpp:106 sendBytes failed ... Broken pipe
[rank1] ProcessGroupNCCL.cpp:1802 Failed to check the "should dump" flag
on TCPStore ... Broken pipe
Steps to reproduce
From the Isaac Lab repository root, start a two-rank RSL-RL training job:
CUDA_VISIBLE_DEVICES=1,2 ./isaaclab.sh -p \
-m torch.distributed.run \
--standalone \
--nnodes=1 \
--nproc_per_node=2 \
scripts/reinforcement_learning/rsl_rl/train.py \
--task Isaac-Cartpole \
--num_envs 2 \
--max_iterations 5000 \
--distributed \
--viz noneAfter both ranks enter training, send SIGTERM to the
torch.distributed.run parent process:
kill -TERM <torchrun-pid>Expected behavior:
- Both workers terminate.
- Normal Python cleanup paths run.
- No training processes remain.
- TCPStore
Broken pipemessages are not repeatedly emitted.
Actual behavior:
- The SIGTERM callback returns after closing
SimulationApp. - A worker may continue executing.
- Its ProcessGroupNCCL heartbeat thread continues accessing the closed
TCPStore. - The terminal repeatedly prints
Broken pipeerrors.
System Info
- Commit:
6f991d4becf764b151e0a1c775561ddd9c406f72 - Isaac Sim Version:
6.0.1.0 - OS: Ubuntu 24.04.4 LTS
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
- CUDA: 12.8
- GPU Driver: 610.43.02
Additional context
PR #5886 identified the missing SIGTERM termination behavior as an independent
latent bug. That PR was closed because it also bundled SIGHUP handling and a
force-exit mechanism. Its closing comment explicitly welcomed a smaller,
standalone fix.
PR #5933 addresses renderer/Fabric multi-GPU contention and does not change the
SIGTERM handler.
I searched the repository's open and closed issues and pull requests. There is
no equivalent issue and no matching fix that is currently open or already
merged.
Checklist
- I have checked that there is no similar issue in the repo (required)
- I have checked that the issue is not in running Isaac Sim itself and is related to the repo
Acceptance Criteria
- SIGTERM terminates the Python worker instead of returning to the interrupted execution path.
- Normal Python cleanup paths remain available.
- No worker processes remain after distributed shutdown.
- Distributed shutdown does not repeatedly emit TCPStore
Broken pipeerrors.
Activity
RandomOakForest commentedon Jul 22, 2026
Thank you for posting this and adding the PR. The team will review it.
Make AppLauncher teardown truthful and re-entrancy-safe; document mGP…
Make AppLauncher teardown truthful and re-entrancy-safe; document mGP…