[Mac OS] [RL] Run RL agents on the simulation device and raise without CUDA by AntoineRichard · Pull Request #8188 · isaac-sim/IsaacLab

AntoineRichard · 2026-09-30T13:55:42Z

Description

Hosts without CUDA, such as macOS, could not train with RSL-RL even with --device cpu. The agent configuration's device (cuda:0 by default) was used to build the policy, so the run failed with AssertionError: Torch not compiled with CUDA enabled.

This PR uses the device launch_simulation already resolves (since #8125), and sets it in each RL library's shared agent-config setup:

Clear error without CUDA. _resolve_device in isaaclab.app.sim_launcher raises RuntimeError: Device 'cuda:0' requires CUDA, which is not available; pass --device cpu. when the resolved device is a CUDA device and CUDA is unavailable. Before, the run failed later inside PyTorch. On macOS, [Mac OS] Default --device to cpu on macOS #8190 defaults --device to cpu.
The agent runs on the resolved device, set in one shared setup per library. Each library's config setup takes the device (args_cli.device, which launch_simulation resolves) as an explicit argument. Every workflow calls it after launch, before the agent is built or its config is saved. This is unconditional, not only for --distributed:
- RSL-RL: the existing update_rsl_rl_cfg(agent_cfg, args_cli) sets agent_cfg.device = args_cli.device, for train, play, and benchmark train. Export and benchmark play, whose CLIs don't define the RSL-RL training arguments, set the device directly.
- RL-Games: the new update_rl_games_cfg(agent_cfg, args_cli) mirrors update_rsl_rl_cfg: it sets device / device_name and the seed for train, play, export, and both benchmarks. RL-Games export gains --seed, matching RSL-RL export, so every RL-Games workflow can call it.
- Stable-Baselines3: the new update_sb3_cfg(agent_cfg, args_cli) mirrors update_rsl_rl_cfg: it sets the device and seed for train, play, and both benchmarks. Both PPO.load() calls pass device=agent_cfg["device"], so checkpoint loading follows the same setup.
skrl and TorchRL already used the simulation's device. The saved agent config records the device actually used, which addresses the bot findings.

Closes #8193.

Part of the experimental macOS (Apple silicon, CPU-only) support series.

Type of change

Bug fix (non-breaking change which fixes an issue)
Breaking change (existing functionality will not work without user modification)

Breaking: outside distributed runs, the RSL-RL, RL-Games and SB3 CLI workflows used to ignore --device for the agent:

Command	Before	After
`--device cpu`	Simulation on CPU, agent on `cuda:0`	Both on the CPU
`--device cuda:1`	Simulation on GPU 1, agent on GPU 0 (see #2148)	Both on GPU 1

The agent configuration's device is now overwritten by these workflows. The wrappers still support running the simulation and the agent on different devices for custom scripts; source/isaaclab_tasks/test/core/test_rl_device_separation.py covers that and is unaffected. isaaclab_rl gets a .major fragment.

Validation

On macOS arm64, CPU-only:

Run	Result
RSL-RL train, `--device cpu`	Trains
SB3 train, `--device cpu`	Trains (`Using cpu device`)
RL-Games train, `--device cpu` (512 envs)	Completes 2 epochs
RSL-RL train, `--device cuda:0`	Stops with `RuntimeError: Device 'cuda:0' requires CUDA, which is not available; pass --device cpu.`

source/isaaclab_rl/test/test_entrypoints.py, test_entrypoints_common.py, and source/isaaclab/test/app/test_launch_simulation_require_kit.py: 83 passed. The one failure is test_skrl_training_parser_leaves_algorithm_implicit, which needs skrl, not installed locally. test_run_summary_reports_the_launch_resolved_visualizers_and_device now launches on cpu: it only used cuda:0 as a label and still checks the device resolved after launch.

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with uv run isaaclab --format (all hooks pass except check-git-lfs-pointers, which needs git-lfs installed locally)
I have made corresponding changes to the documentation
My changes generate no new warnings
I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
I have added my name to the CONTRIBUTORS.md or my name already exists there

greptile-apps · 2026-09-30T14:12:19Z

Confidence Score: 4/5

[Medium risk] Adds device fallback logic to the reinforcement learning runner.

The PR appears safe to merge, though CPU training runs should record their effective agent device for reproducibility.

Findings

Saved agent device is inaccurate ▶

Summary

The PR makes RSL-RL runner construction fall back to the environment device when the configured CUDA device is unavailable, and adds a CPU construction test and changelog entry. The effective device is not reflected in the saved training configuration.

_{Reviews (1) · Last reviewed commit: "Warn when the RSL-RL runner falls back t..."}

greptile-apps · 2026-09-30T14:12:22Z

+            device,
+            env.device,
+        )
+        device = str(env.device)


Saved agent device is inaccurate
When CUDA is unavailable, this fallback can build the runner on the environment’s CPU device, but it does not update agent_cfg.device. Training later saves that unchanged configuration to params/agent.yaml, so a CPU run records cuda:0 as its agent device. This makes the saved configuration misleading when reproducing the run; please record the effective device.

isaaclab-review-bot

Isaac Lab Review Bot

The CPU-only fallback is correctly placed at the shared RSL-RL runner factory and has focused regression coverage, but the runner now receives conflicting device values: the explicit constructor argument uses the resolved environment device while the configuration dictionary still contains the unavailable configured CUDA device.

Design and architecture: Resolving the fallback in create_rsl_rl_runner appropriately covers training, playback, and export. However, the factory now passes two disagreeing device authorities into the runner: device contains the resolved fallback, while to_dict(agent_cfg) retains agent_cfg.device. Resolve the device once and ensure the dictionary passed to both runner types records the same value.
API: The public function signature, return types, supported runner classes, and unsupported-class ValueError remain unchanged. The new behavior is internal device resolution, but the configuration representation passed across the RSL-RL boundary should be kept consistent with the explicit device argument.
Compatibility and deprecation: Breaking changes: none identified. The fallback activates only when a configured CUDA device is unavailable, while CUDA-capable hosts retain the configured behavior. No public signature, default, configuration key, or export surface was removed or renamed.
Implementation: Both OnPolicyRunner and DistillationRunner use the resolved local device, and the warning identifies the configured and fallback devices. The remaining issue is that both calls still pass to_dict(agent_cfg) unchanged, leaving cuda:0 in the runner configuration when the explicit constructor device is CPU; update the dictionary before construction so runtime and stored configuration agree.
Style consistency: The module-level imports, logger placement, naming, warning construction, docstring update, and changelog fragment follow adjacent repository patterns. No separate style issue was identified.
Test quality: The new test is a justified owner-boundary regression: it constructs a real OnPolicyRunner, verifies CPU policy placement, and checks the warning on the no-CUDA path, which existing configuration-migration tests do not cover. It does not currently check that the configuration passed into the runner records the resolved device, so the conflicting-device state remains undetected.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

isaaclab-review-bot · 2026-09-30T14:18:20Z

+        device = str(env.device)
    if agent_cfg.class_name == "OnPolicyRunner":
-        return OnPolicyRunner(env, to_dict(agent_cfg), log_dir=log_dir, device=agent_cfg.device)
+        return OnPolicyRunner(env, to_dict(agent_cfg), log_dir=log_dir, device=device)


🔵 Suggestion · Design Architecture — Runner receives two conflicting devices

The fallback only changes the local variable, so to_dict(agent_cfg) passed as the runner's train config still records cuda:0 while the runner is constructed on cpu. The runner stores and serializes that config, so dumped and resumed agent configs keep the unavailable device. Resolve the device once and keep the config consistent, for example by updating the device entry in the dictionary handed to the runner.

StafaH · 2026-09-30T15:25:04Z

 ) -> OnPolicyRunner | DistillationRunner:
    """Instantiate the RSL-RL runner selected by ``agent_cfg.class_name``.

+    The runner uses ``agent_cfg.device``, or the environment's device when that is a CUDA device and


We can remove this docstring addition.

StafaH · 2026-09-30T15:27:44Z

        ValueError: If the configured runner class is not supported.
    """
+    device = agent_cfg.device
+    if device.startswith("cuda") and not torch.cuda.is_available():


This looks fine to me, but maybe the problem should be caught earlier that agent_cfg.device != env_cfg.device, and if either == cuda and cuda is not available then they've picked wrong?

yeah I have an issue to track this: #8193 it's a real problem that I would fix separately.

StafaH · 2026-09-30T15:56:57Z

@@ -0,0 +1,73 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).


The test can be dropped

ooctipus · 2026-09-30T21:56:42Z

Heads-up from #8125 (now merged), as @StafaH asked: launch_simulation now resolves the run's device once and writes it to env_cfg.sim.device. That covers --device, each rank's cuda:N in distributed runs, the config's own device when --device isn't given, and the runtime's refinement (e.g. XR picks the CPU). So the runner can take its device from env_cfg.sim.device inside the launch rather than from agent_cfg.device, and this fallback could build on that.

…l-runner-cpu-fallback

…without CUDA

…l-runner-cpu-fallback # Conflicts: # source/isaaclab/isaaclab/benchmark/entrypoints/backends/rsl_rl/benchmark_play_rsl_rl.py # source/isaaclab_rl/isaaclab_rl/entrypoints/backends/export_rsl_rl.py # source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_rsl_rl.py

ooctipus · 2026-10-05T05:12:40Z

    device = args.get("device")
    for launcher in launchers:
        device = launcher.device or device
+    if device is not None and device.startswith("cuda") and not torch.cuda.is_available():


id rather raise an error than fall back if device startwith("cuda")

ooctipus

Can we set the agent device as part of shared RL setup after launch? Right now each workflow has to remember to do it, including the benchmarks. This should happen before the agent is built and its config is saved. I've marked the copies below. I wouldn't move the RL-specific config handling into the simulation launcher, or add a one-line setter that still has to be called from every workflow.

ooctipus · 2026-10-05T05:53:58Z

            if args_cli.max_iterations is not None:
                agent_cfg.max_iterations = args_cli.max_iterations
+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg.device = env_cfg.sim.device


We already call update_rsl_rl_cfg() a few lines above. Can the resolved device be applied there with the other agent settings, and have the other RSL-RL workflows use the same setup?

ooctipus · 2026-10-05T05:53:58Z

            show_run_summary(screen, args_cli, env_cfg, library="rsl_rl", action="play")
            agent_cfg = cli_args.update_rsl_rl_cfg(agent_cfg, args_cli)
+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg.device = env_cfg.sim.device


Same as training: this comes straight after update_rsl_rl_cfg(). Can that call handle the resolved device too?

ooctipus · 2026-10-05T05:53:58Z

    if args_cli.experiment_name is not None:
        agent_cfg.experiment_name = args_cli.experiment_name
+    # the agent runs on the device launch_simulation resolved for the simulation
+    agent_cfg.device = env_cfg.sim.device


Can export go through the same RSL-RL config setup as train/play? Otherwise it's another place we have to keep this assignment in sync.

ooctipus · 2026-10-05T05:53:58Z

            env_cfg.seed = agent_cfg.seed

+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg.device = env_cfg.sim.device


This already uses update_rsl_rl_cfg() too. Can it pick up the device there so regular training and the benchmark share the same setup?

ooctipus · 2026-10-05T05:53:58Z

            env_cfg.seed = agent_cfg.seed

+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg.device = env_cfg.sim.device


Can benchmark play use the same RSL-RL config setup as regular play? The device should already be set by the time we get here.

ooctipus · 2026-10-05T05:53:58Z


            # Read rl_games device / clipping config.
+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg["params"]["config"]["device"] = agent_cfg["params"]["config"]["device_name"] = env_cfg.sim.device


This repeats the device setup from regular play. Can both paths get a config with the device already set?

ooctipus · 2026-10-05T05:53:58Z

                agent_cfg["n_timesteps"] = args_cli.max_iterations * agent_cfg["n_steps"] * env_cfg.scene.num_envs
            env_cfg.seed = agent_cfg["seed"]
+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg["device"] = env_cfg.sim.device


Can we set this during shared SB3 setup, before saving agent_cfg and constructing PPO? The training benchmark repeats the same assignment.

ooctipus · 2026-10-05T05:53:58Z

            agent_cfg["seed"] = args_cli.seed if args_cli.seed is not None else agent_cfg.get("seed", 0)
            env_cfg.seed = agent_cfg["seed"]
+            # the agent runs on the device launch_simulation resolved for the simulation
+            agent_cfg["device"] = env_cfg.sim.device


Same device assignment as regular SB3 training. Can both use the same config setup?

ooctipus · 2026-10-05T05:53:58Z


            print(f"Loading checkpoint from: {checkpoint_path}")
-            agent = PPO.load(checkpoint_path, env, print_system_info=True)
+            agent = PPO.load(checkpoint_path, env, device=env_cfg.sim.device, print_system_info=True)


Keep this explicit device argument when consolidating SB3 setup; changing agent_cfg alone won't affect PPO.load().

ooctipus · 2026-10-05T05:53:58Z


            # Load the trained policy.
-            agent = PPO.load(resume_path, env, print_system_info=True)
+            agent = PPO.load(resume_path, env, device=env_cfg.sim.device, print_system_info=True)


Same point as regular SB3 play: the shared setup needs to cover checkpoint loading too. This explicit device argument is needed, so we shouldn't just remove it.

…l-runner-cpu-fallback

…se without CUDA

# Description `--device` defaults to `cuda:0`. On macOS, which has no CUDA, every command without an explicit `--device cpu` fails at simulation start (`Torch not compiled with CUDA enabled` in `SimulationContext._init_usd_physics_scene`). The default is now `cpu` on macOS and stays `cuda:0` elsewhere. It checks `sys.platform` rather than `torch.cuda.is_available()` because the launcher defers importing torch until after Kit starts (see the comment in `_resolve_device_settings`). An explicit `--device` still wins, and `device_explicit` is untouched, so the XR device logic is unaffected. `launch_simulation` logs a warning when it applies this default (`No --device given; using 'cpu' because macOS has no CUDA.`), so the switch is visible. Passing `--device` explicitly suppresses it. Hosts that request a CUDA device without having CUDA are handled separately by #8188, which falls back to the CPU with its own warning. Part of the experimental macOS (Apple silicon, CPU-only) support series. ## Type of change - New feature (non-breaking change which adds functionality) ## Validation - On macOS arm64, rebased on `develop` with #8125: `isaaclab train --rl_library rsl_rl --task Isaac-Cartpole-Direct physics=newton_mjwarp --max_iterations 2` (no `--device`) logs the warning, reports `Device cpu`, and trains to completion. This also needs #8188. - With `--device cpu`, no warning is logged. ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` (all hooks pass except `check-git-lfs-pointers`, which needs git-lfs installed locally) - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump package versions — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there

AntoineRichard · 2026-10-09T13:06:59Z

run-ci

Fall back to the env device for RSL-RL runners without CUDA

827a989

github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Sep 30, 2026

AntoineRichard mentioned this pull request Sep 30, 2026

[Mac OS] Support macOS arm64 (CPU-only) in the uv environment #8191

Open

7 tasks done

AntoineRichard changed the title ~~Fall back to the env device for RSL-RL runners without CUDA~~ [Mac OS] Fall back to the env device for RSL-RL runners without CUDA Sep 30, 2026

AntoineRichard mentioned this pull request Sep 30, 2026

[Proposal] Unify how warnings are emitted across Isaac Lab #8192

Closed

1 of 5 tasks

Warn when the RSL-RL runner falls back to the env device

f2cf68f

AntoineRichard marked this pull request as ready for review September 30, 2026 14:09

AntoineRichard requested a review from a team September 30, 2026 14:09

AntoineRichard requested review from ClemensSchwarke, StafaH, kellyguo11 and ooctipus as code owners September 30, 2026 14:09

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

AntoineRichard mentioned this pull request Sep 30, 2026

[Proposal] RL agents should default to the environment's device #8193

Closed

1 of 5 tasks

isaaclab-review-bot Bot reviewed Sep 30, 2026

View reviewed changes

StafaH reviewed Sep 30, 2026

View reviewed changes

Drop the runner device test and docstring note

2284297

StafaH mentioned this pull request Sep 30, 2026

[Core] Decide visualizers and the device once in launch_simulation #8125

Merged

5 of 6 tasks

AntoineRichard added 2 commits October 1, 2026 17:47

Merge remote-tracking branch 'origin/develop' into antoiner/fix/rsl-r…

ce07f72

…l-runner-cpu-fallback

Run RL agents on the resolved simulation device and fall back to CPU …

566ac34

…without CUDA

AntoineRichard requested review from fatimaanes and pbarejko as code owners October 1, 2026 15:54

AntoineRichard changed the title ~~[Mac OS] Fall back to the env device for RSL-RL runners without CUDA~~ [Mac OS] Run RL agents on the resolved simulation device and fall back to CPU without CUDA Oct 1, 2026

AntoineRichard mentioned this pull request Oct 1, 2026

[Mac OS] Default --device to cpu on macOS #8190

Merged

6 of 7 tasks

ooctipus reviewed Oct 5, 2026

View reviewed changes

ooctipus approved these changes Oct 5, 2026

View reviewed changes

AntoineRichard added 2 commits October 5, 2026 09:20

Merge remote-tracking branch 'origin/develop' into antoiner/fix/rsl-r…

f82356a

…l-runner-cpu-fallback

Set the agent device in each RL library's shared config setup and rai…

0c81e67

…se without CUDA

AntoineRichard changed the title ~~[Mac OS] Run RL agents on the resolved simulation device and fall back to CPU without CUDA~~ [Mac OS] [RL] Run RL agents on the simulation device and raise without CUDA Oct 5, 2026

AntoineRichard added 4 commits October 5, 2026 09:56

Pass the agent device and seed to the shared RL config setup explicitly

062dd2a

Restore update_rsl_rl_cfg's args_cli signature and set the device there

9d208ef

Pass args_cli to update_sb3_cfg like update_rsl_rl_cfg

202545f

Add --seed to RL-Games export and pass args_cli to update_rl_games_cfg

9055fc6

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 9, 2026

maxkra15 self-requested a review October 9, 2026 14:01

maxkra15 approved these changes Oct 9, 2026

View reviewed changes

AntoineRichard merged commit 39300b0 into isaac-sim:develop Oct 9, 2026
58 checks passed

		@@ -0,0 +1,73 @@
		# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).

Conversation

AntoineRichard commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Validation

Checklist

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

Isaac Lab Review Bot

Uh oh!

isaaclab-review-bot Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ooctipus commented Sep 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ooctipus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AntoineRichard commented Oct 9, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AntoineRichard commented Sep 30, 2026 •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading