Add Policy Ghosts, recording and replay, an RTX toggle, and a Tracking Camera to the Newton GL visualizer

.gitignore

@@ -61,6 +61,7 @@ _build
 **/runs/*
 **/logs/*
 **/recordings/*
+**/recording/*
 
 # Local development tools and generated media.
 /tools/local/

docs/source/api/lab/isaaclab.visualizers.rst

@@ -21,6 +21,10 @@ The following classes are part of the public :mod:`isaaclab.visualizers` API.
    :nosignatures:
 
    BaseVisualizer
+   PolicyGhostCheckpoint
+   PolicyGhostController
+   PolicyGhostLayout
+   PolicyGhostSlot
    VisualizerCfg
    VisualizerBackendCfg
    WindowCfg
@@ -32,6 +36,22 @@ The following classes are part of the public :mod:`isaaclab.visualizers` API.
 .. autoclass:: BaseVisualizer
    :show-inheritance:
 
+.. autoclass:: PolicyGhostCheckpoint
+   :members:
+   :show-inheritance:
+
+.. autoclass:: PolicyGhostController
+   :members:
+   :show-inheritance:
+
+.. autoclass:: PolicyGhostLayout
+   :members:
+   :show-inheritance:
+
+.. autoclass:: PolicyGhostSlot
+   :members:
+   :show-inheritance:
+
 .. autoclass:: VisualizerCfg
    :show-inheritance:
 

docs/source/concepts/visualization.rst

@@ -776,6 +776,46 @@ without screen-sharing, running with ``--viz viser``:
 The share URL is logged on startup. Rerun has no equivalent config field, but exposes its own
 share button in the native UI.
 
+**Comparing checkpoints with Policy Ghosts**
+
+Newton GL can overlay several checkpoints of one task in a single viewport. The ``--checkpoint``
+drives the reference robots; ghosts of other checkpoints are tinted, translucent copies that start
+from the same terrain, spawn pose, and commands, so differences in behavior show up as the robots
+drift apart:
+
+.. code-block:: bash
+
+    uv run isaaclab play --rl_library rsl_rl --task Isaac-Velocity-Rough-AnymalD \
+        --num_envs 4 --checkpoint latest --viz newton_gl --policy_ghosts
+
+You can also start from a normal ``play`` run: open **Policy Ghosts** in the viewer's left panel and choose
+**Enable Policy Ghosts (restarts)**. The command restarts with ``--policy_ghosts`` added, because the simulation
+fixes its environment count at startup; **Disable Policy Ghosts (restarts)** reverses it. Policy Ghosts stay off
+unless you ask for them, since they simulate several times the environments.
+
+Open **Policy Ghosts** in the viewer's left panel and choose **Add Checkpoint...** to list the
+checkpoints of runs of this task that were written by training (those with a ``run.json``) under the
+experiment's log directory; pass ``--policy_ghost_root`` to search another directory. Each loaded checkpoint has a checkbox that
+shows or hides it. Under **Settings**, each ghost has opacity and tint sliders; a tint of 1.0 replaces all of
+the robot's native colors and textures with the ghost color, and lower values blend it with the native colors.
+**Remove** frees the slot. ``--policy_ghost_slots`` sets the number of slots, including the reference, up to eight;
+the default is eight.
+
+* Every slot simulates ``--num_envs`` environments, even an empty one. With a few environments per
+  slot the extra cost is small, since the step time is dominated by fixed overhead; it grows with the
+  total environment count (slots times ``--num_envs``).
+* Randomization that cannot be mirrored between slots is disabled: startup and interval events,
+  curriculum terms, and observation noise. Reset events and commands are copied from the reference.
+* Identical checkpoints do not stay exactly on top of each other; physics differences grow
+  to centimeters over a few seconds on rough terrain.
+* Policy Ghosts support RSL-RL and manager-based environments only.
+
+**Choosing how many environments are drawn**
+
+Every Newton GL viewer has a **Visible Envs** slider in its left panel that sets how many
+environments are drawn. It changes only what is rendered, not the simulation. With Policy Ghosts, it
+counts environments per slot.
+
 Resolution Rules
 ~~~~~~~~~~~~~~~~
 

source/isaaclab/changelog.d/policy-ghost-sync.added.rst

@@ -0,0 +1,3 @@
+* Added :class:`~isaaclab.visualizers.PolicyGhostController` and its slot and checkpoint types, and
+  :class:`~isaaclab.envs.utils.policy_ghost_sync.PolicyGhostSync`, which keep ghost slices of one manager-based
+  environment on the reference slice's terrain, spawn state, and commands.

source/isaaclab/changelog.d/rtx-viewer-camera-launch.added.rst

@@ -0,0 +1,3 @@
+* Added an OVRTX viewer camera that the launcher declares in the scene for Newton GL windows, so the window can switch
+  its main view to OVRTX rendering. The window size is fitted to the display so the camera and the window share an
+  aspect ratio.

source/isaaclab/changelog.d/tracking-camera-views.added.rst

@@ -0,0 +1,6 @@
+* Added ``env_index``, ``track_path``, ``follow_heading``, and ``heading_smoothing_time_constant`` to
+  :class:`~isaaclab.visualizers.PerspectiveCameraCfg`, and :class:`~isaaclab.visualizers.camera_follower.CameraFollower`,
+  so a perspective view can frame an environment or follow an asset with smoothed yaw. Only the Newton GL window
+  renders these views; other visualizers ignore them.
+* Added :meth:`~isaaclab.visualizers.camera_follower.CameraFollower.set_track_path`, ``asset_names``, and
+  ``body_names`` so a view can switch the asset or body it follows while running.

source/isaaclab/changelog.d/usd-spawn-kitless-visual-material.fixed.rst

@@ -0,0 +1,2 @@
+* Fixed :class:`~isaaclab.sim.UsdFileCfg` skipping its ``visual_material`` when Kit is not running, which left
+  visualization markers in kitless renderers such as OVRTX with the asset's own color instead of the configured one.

source/isaaclab/isaaclab/app/sim_launcher.py

@@ -8,6 +8,7 @@
 from __future__ import annotations
 
 import argparse
+import importlib.util
 import logging
 import os
 import sys
@@ -158,6 +159,39 @@ def visit(node):
     return tuple(configs)
 
 
+def _fit_window_to_display(size: tuple[int, int], margin: float = 0.92) -> tuple[int, int]:
+    """Shrink a window size, keeping its aspect ratio, so the window fits the display with room for decorations.
+
+    Args:
+        size: Requested window width and height [px].
+        margin: Fraction of the display the window may occupy.
+
+    Returns:
+        The requested size, or a proportionally smaller one; unchanged when the display size is unknown.
+    """
+    if sys.platform not in ("win32", "darwin") and not os.environ.get("DISPLAY"):
+        return size
+    try:
+        import pyglet  # noqa: PLC0415
+
+        screen = pyglet.display.get_display().get_default_screen()
+        scale = min(1.0, margin * screen.width / size[0], margin * screen.height / size[1])
+    except Exception:
+        return size
+    return (max(1, int(size[0] * scale)), max(1, int(size[1] * scale)))
+
+
+def _wants_rtx_viewer_camera(visualizer: VisualizerCfg, simulations: list[SimulationCfg]) -> bool:
+    """Return whether a Newton GL window should get an OVRTX viewer camera in its scene."""
+    backend = visualizer.backend
+    if not isinstance(backend, NewtonGLBackendCfg) or visualizer.headless or visualizer.view is not None:
+        return False
+    if backend.rtx_camera is not None:
+        return backend.rtx_camera
+    on_physx = any(isinstance(sim.physics, (PhysxCfg, PhysxAutoCfg, OvPhysxCfg)) for sim in simulations)
+    return importlib.util.find_spec("ovrtx") is not None and not on_physx
+
+
 def validate_launch(  # noqa: C901
     cfg, launcher_args: argparse.Namespace | dict | None = None
 ) -> tuple[tuple[Any, ...], tuple[str, ...]]:
@@ -310,11 +344,40 @@ def validate_launch(  # noqa: C901
 
     # Declare RTX perspective cameras before scene construction; runtime viewers only borrow their output.
     scene_cfg = next((node for node in configs if isinstance(node, InteractiveSceneCfg)), None)
+    # A Newton GL window can switch its main view to an OVRTX camera, which must already exist in the scene.
+    implicit_viewer_cameras: list[str] = []
+    for viewer_index, visualizer in enumerate(viewers):
+        if scene_cfg is None or not _wants_rtx_viewer_camera(visualizer, simulations):
+            continue
+        name = f"viewer_camera_{viewer_index}_rtx"
+        # The window must open at the camera's aspect ratio; an oversized window is clamped by the window manager.
+        visualizer.window.size = _fit_window_to_display(visualizer.window.size)
+        width, height = visualizer.window.size
+        rotation = quat_from_matrix(create_rotation_matrix_from_view(
+            torch.tensor([visualizer.eye]), torch.tensor([visualizer.lookat]), "Z", device="cpu"
+        ))[0].tolist()  # fmt: skip
+        implicit_viewer_cameras.append(name)
+        vars(scene_cfg)[name] = CameraCfg(
+            prim_path=f"/World/{name}", width=width, height=height, data_types=["rgba"],
+            spawn=PinholeCameraCfg(
+                focal_length=visualizer.focal_length, vertical_aperture=15.2908,
+                horizontal_aperture=15.2908 * width / height,
+            ),
+            offset=CameraCfg.OffsetCfg(pos=visualizer.eye, rot=tuple(rotation), convention="opengl"),
+            renderer_cfg=visualizer.backend.rtx_renderer_cfg, background_color=visualizer.background_color,
+            render_settings={"omni:rtx:quality": ("Int", 0), **visualizer.backend.rtx_render_settings},
+        )  # fmt: skip
+        visualizer.streaming_view = True  # The window lists scene cameras, including this one, as view sources.
     for viewer_index, visualizer in enumerate(viewers):
         if not isinstance(visualizer.backend, NewtonRTXBackendCfg) or scene_cfg is None:
             continue
+        # Task views belong to the Newton GL window; an RTX window gets an ordinary perspective camera instead.
+        declared = [
+            camera for camera in visualizer.cameras or []
+            if not (isinstance(camera, PerspectiveCameraCfg) and camera.is_task_view)
+        ]  # fmt: skip
         sources = (
-            [visualizer.view.source] if visualizer.view is not None else list(visualizer.cameras or [
+            [visualizer.view.source] if visualizer.view is not None else list(declared or [
                 PerspectiveCameraCfg(eye=visualizer.eye, lookat=visualizer.lookat, focal_length=visualizer.focal_length)
             ])
         )  # fmt: skip
@@ -385,6 +448,11 @@ def validate_launch(  # noqa: C901
             node.max_visible_envs = max_visible
 
     configs = _walk_runtime_configs(cfg, replacements)
+    if kit_sources and implicit_viewer_cameras:
+        # The OVRTX runtime cannot share a process with Kit, so a Kit run keeps the OpenGL view only.
+        for name in implicit_viewer_cameras:
+            vars(scene_cfg).pop(name, None)
+        configs = _walk_runtime_configs(cfg, replacements)
     if kit_sources:
         for node in configs:
             if isinstance(node, OvPhysxCfg):

source/isaaclab/isaaclab/envs/utils/policy_ghost_sync.py

@@ -0,0 +1,159 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Keeps ghost slices of one environment on the reference slice's terrain, spawn state and commands."""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING, Any
+
+import numpy as np
+import torch
+
+if TYPE_CHECKING:
+    from ..manager_based_rl_env import ManagerBasedRLEnv
+    from ..manager_based_rl_env_cfg import ManagerBasedRLEnvCfg
+
+logger = logging.getLogger(__name__)
+
+
+class PolicyGhostSync:
+    """Synchronizes the slices of a single environment so that several policies can be compared.
+
+    The environment is simulated with ``num_slices * slice_size`` worlds. Slice 0 is the reference; environment
+    ``i`` of every other slice mirrors environment ``i`` of the reference. Whenever a reference environment resets,
+    its ghost counterparts reset with it and receive the reference's spawn state. Velocity or pose commands are
+    copied from the reference after every command update. Ghost environments that terminate on their own wait
+    for the reference to reset.
+
+    Only :class:`~isaaclab.envs.ManagerBasedRLEnv` is supported. Call :meth:`configure_env_cfg` on the env config
+    before creating the environment, because per-environment randomization other than the reset events cannot
+    be mirrored.
+    """
+
+    def __init__(self, env: ManagerBasedRLEnv, slice_size: int, num_slices: int):
+        """Attach to an environment and place every slice on the reference slice's terrain.
+
+        Args:
+            env: Environment simulated with ``slice_size * num_slices`` worlds.
+            slice_size: Number of environments per slice.
+            num_slices: Number of slices, including the reference.
+
+        Raises:
+            TypeError: If the environment does not use managers.
+            ValueError: If the sizes do not match the environment.
+        """
+        for manager in ("command_manager", "event_manager", "scene"):
+            if not hasattr(env, manager):
+                raise TypeError("Policy Ghosts only support manager-based environments.")
+        if slice_size < 1 or num_slices < 2 or env.num_envs != slice_size * num_slices:
+            raise ValueError(f"Environment has {env.num_envs} envs, expected {num_slices} slices of {slice_size} envs.")
+        self._env = env
+        self._slice_size = slice_size
+        self._num_slices = num_slices
+        self._device = env.device
+        self._reference_ids = torch.arange(slice_size, device=self._device)
+        self._ghost_offsets = torch.arange(1, num_slices, device=self._device) * slice_size
+        self._sync_terrain()
+        self._reset_idx_original = env._reset_idx
+        self._compute_commands_original = env.command_manager.compute
+        env._reset_idx = self._reset_idx
+        env.command_manager.compute = self._compute_commands
+
+    @staticmethod
+    def configure_env_cfg(env_cfg: ManagerBasedRLEnvCfg) -> list[str]:
+        """Remove per-environment randomization that cannot be mirrored between slices.
+
+        Startup and interval events, curriculum terms, and observation corruption are disabled. Reset events
+        are kept because the reference's reset state is copied to the ghosts.
+
+        Args:
+            env_cfg: Environment config to modify in place.
+
+        Returns:
+            Names of the disabled terms.
+        """
+        removed = []
+        # A task may declare no events, curriculum, or observation groups at all.
+        events, curriculum, observations = (
+            getattr(env_cfg, name, None) for name in ("events", "curriculum", "observations")
+        )
+        for name, term in vars(events).items() if events is not None else ():
+            if term is not None and getattr(term, "mode", None) in ("startup", "interval"):
+                setattr(events, name, None)
+                removed.append(f"events.{name}")
+        for name, term in vars(curriculum).items() if curriculum is not None else ():
+            if term is not None:
+                setattr(curriculum, name, None)
+                removed.append(f"curriculum.{name}")
+        for group_name, group in vars(observations).items() if observations is not None else ():
+            if getattr(group, "enable_corruption", False):
+                group.enable_corruption = False
+                removed.append(f"observations.{group_name}.enable_corruption")
+        if removed:
+            logger.warning("Policy Ghosts disabled per-environment randomization: %s", ", ".join(removed))
+        return removed
+
+    def env_origin_offsets(self) -> np.ndarray:
+        """Return per-environment offsets [m], shape ``[num_envs, 3]``, that move each ghost onto the reference."""
+        origins = self._env.scene.env_origins
+        offsets = torch.zeros_like(origins)
+        for ghost_start in self._ghost_offsets.tolist():
+            offsets[ghost_start : ghost_start + self._slice_size] = (
+                origins[: self._slice_size] - origins[ghost_start : ghost_start + self._slice_size]
+            )
+        return offsets.cpu().numpy().astype(np.float32)
+
+    def close(self) -> None:
+        """Restore the environment's original reset and command update."""
+        self._env._reset_idx = self._reset_idx_original
+        self._env.command_manager.compute = self._compute_commands_original
+
+    def _sync_terrain(self) -> None:
+        """Place every ghost environment on the terrain tile of its reference environment."""
+        terrain = self._env.scene.terrain
+        if terrain is None or getattr(terrain, "terrain_origins", None) is None:
+            return
+        for name in ("terrain_levels", "terrain_types", "env_origins"):
+            values = getattr(terrain, name)
+            for ghost_start in self._ghost_offsets.tolist():
+                values[ghost_start : ghost_start + self._slice_size] = values[: self._slice_size]
+
+    def _reset_idx(self, env_ids: Any) -> None:
+        """Reset reference environments together with their ghosts, then copy the reference's spawn state."""
+        if isinstance(env_ids, slice):
+            env_ids = torch.arange(self._env.num_envs, device=self._device)[env_ids]
+        env_ids = torch.as_tensor(env_ids, device=self._device, dtype=torch.long)
+        reference = env_ids[env_ids < self._slice_size]
+        if reference.numel() == 0:
+            # ghost-only terminations wait for the reference to reset
+            return
+        ghosts = (reference[None, :] + self._ghost_offsets[:, None]).reshape(-1)
+        self._reset_idx_original(torch.cat([reference, ghosts]))
+        scene = self._env.scene
+        state = scene.get_state(is_relative=True)
+        scene.reset_to(_gather(state, reference.repeat(self._num_slices - 1)), env_ids=ghosts, is_relative=True)
+        self._copy_command_state(reference)
+
+    def _compute_commands(self, *args: Any, **kwargs: Any) -> None:
+        """Update commands, then copy the reference's commands to the ghosts."""
+        self._compute_commands_original(*args, **kwargs)
+        self._copy_command_state(self._reference_ids)
+
+    def _copy_command_state(self, reference: torch.Tensor) -> None:
+        """Copy every per-environment tensor of each command term from the reference to its ghosts."""
+        ghosts = (reference[None, :] + self._ghost_offsets[:, None]).reshape(-1)
+        source = reference.repeat(self._num_slices - 1)
+        manager = self._env.command_manager
+        for name in manager.active_terms:
+            for value in vars(manager.get_term(name)).values():
+                if isinstance(value, torch.Tensor) and value.ndim >= 1 and value.shape[0] == self._env.num_envs:
+                    value[ghosts] = value[source]
+
+
+def _gather(state: dict[str, Any], ids: torch.Tensor) -> dict[str, Any]:
+    """Index every tensor of a nested scene-state dictionary by environment id."""
+    return {key: _gather(value, ids) if isinstance(value, dict) else value[ids] for key, value in state.items()}