Skip to content

Add Policy Ghosts, recording and replay, an RTX toggle, and a Tracking Camera to the Newton GL visualizer - #8786

Draft
matthewtrepte wants to merge 53 commits into
isaac-sim:mainfrom
matthewtrepte:mtrepte/newton-gl-policy-ghosting
Draft

matthewtrepte wants to merge 53 commits into
isaac-sim:mainfrom
matthewtrepte:mtrepte/newton-gl-policy-ghosting

Conversation

@matthewtrepte

Copy link
Copy Markdown
Collaborator

Description

Reworks the Newton GL visualizer window and adds several features to it. All of it is in the Newton GL window and the code it launches; Kit, Viser, and Rerun are unchanged apart from ignoring the new task views.

  • Policy Ghosts. Compare several policy checkpoints of one task in a single window: extra environment slices tinted as ghosts over the reference, a checkpoint discovery window, and per-slot opacity and tint. Enabled with --policy_ghosts; a run without it offers an "Enable (restarts)" button.
  • RTX renderer toggle. The main view can switch between OpenGL and OVRTX. The OVRTX camera is declared at launch, renders only while selected, matches the window's aspect, and supports picking, markers, and Visible Envs. A Kit run drops the camera because the OVRTX runtime cannot share a process with Kit.
  • Recording section. Video with Start, Stop, and Save, a frame rate and an automatic length; .viser animation recording; and a policy checkpoint Save wired into the RSL-RL play and train entrypoints. Output goes to recording/ in the repository.
  • Animation playback (Replay mode). Open a .viser file in the running window. The simulation is hidden and paused, with play, step, loop, scrub, and speed, and a choice of Replay only, Overlay on live, or Side by side. The animation is drawn as separate shapes and never touches the Newton model.
  • Tracking Camera. The task's camera declaration appears in Camera Source and follows an asset or frames an environment, with editable environment, asset, body, eye, look-at, focal length, heading follow, and smoothing. It is a visualizer camera, not a scene sensor.
  • HUD layout. An always-visible header (task, run, physics, step, simulation rate and real-time factor, status badges), the sections Playback, Camera View, Scene, Policy Ghosts, Recording, Live Plots, Articulation Inspector, and Keyboard & Mouse, and tooltips on headers, markers, plots, and controls.
  • Fixes along the way. UsdFileCfg no longer skips its visual_material without Kit, which left OVRTX markers red instead of green and blue.

Relationship to #8620

The Tracking Camera brings over the follower, the PerspectiveCameraCfg fields (env_index, track_path, follow_heading, smoothing), and the task declarations from #8620, adapted to the current visualizer settings layout. It does not bring the Kit and Viser follow support, the viz:newton_gl:tracking video source, or the docs, because that PR conflicts with main in code I cannot test here. Expect overlapping hunks when it lands.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)

Verification

  • Roughly 135 focused tests pass in the visualizer, follower, launcher, replay, recording, and policy ghost files.
  • Run live on Newton MJWarp with a display: Ant and Anymal velocity (including the published Newton checkpoint), Anymal rough terrain, and Shadow Hand. I checked picking, markers, the RTX toggle, recording and replay, the Tracking Camera following the robot, and the HUD in OpenGL and RTX modes.
  • Not run: Kit, Viser, and Rerun windows, Newton RTX with task views, a Tracking Camera with heading follow, and the docs build.

Known limits

  • The Tracking Camera stops following when the Renderer is switched to RTX.
  • Policy Ghost tint and overlay do not apply in the RTX view; Visible Envs does.
  • The navigation tasks could not be exercised: they are PhysX-only and PhysX articulation initialization currently fails in my environment with a ProxyArray passed to a Warp kernel in isaaclab_physx, which this PR does not touch.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

…policy-ghosting

# Conflicts:
#	source/isaaclab_rl/isaaclab_rl/entrypoints/backends/play_rsl_rl.py
…icy-ghosting

# Conflicts:
#	source/isaaclab/isaaclab/visualizers/__init__.pyi
#	source/isaaclab_visualizers/isaaclab_visualizers/newton/newton_visualizer.py
#	source/isaaclab_visualizers/isaaclab_visualizers/viser/viser_visualizer.py
… Markers section, and label the camera combo Camera Source
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant