Repository navigation
Move the remaining keyboard scripts to Isaac Capture and deprecate the isaaclab.devices keyboard, gamepad and SpaceMouse - #8798
Open
rwiltz wants to merge 14 commits into
Conversation
BaseVisualizer.key_event_source exposes keys typed into a visualizer's own window, or None when it has none. A KeyEventSource reports each press and release as a W3C KeyboardEvent.code string, plus a focus loss when the window stops delivering keys (blur, the visualizer's UI taking the keyboard, or close). add_key_listener returns a KeyboardSubscription, an idempotent handle the caller closes. Events are forwarded as the window reports them, so a listener tracks the keys it holds. Subscribing, closing and callbacks all happen on the thread that steps the visualizer. Closing the source is terminal: every listener gets one final focus loss. BaseVisualizer.set_camera_keys_enabled lets a keyboard consumer turn off the visualizer's keyboard camera movement while it uses those keys; it does nothing by default.
KitVisualizer reports physical keys from the Kit app window and a focus loss on window blur. Kit has no Python query for UI keyboard ownership, so keys typed into its UI are reported too. Its viewport camera keys only act while the right mouse button is held, so it keeps the default set_camera_keys_enabled. NewtonGLVisualizer and NewtonRTXVisualizer report layout-mapped keys from their shared pyglet viewer window. They withhold presses typed into its ImGui widgets, checking ImGui's keyboard ownership after every frame. set_camera_keys_enabled(False) stops the window's WASD/QE and arrow-key camera movement, including for a window opened later. Each visualizer closes its keyboard source first in its teardown, while the window is still open, and finishes that teardown when the source fails to close. Headless runs and the Viser and Rerun visualizers have no keyboard source.
rwiltz
requested review from
AntoineRichard,
StafaH,
fatimaanes,
hougantc-nvda,
kellyguo11,
matthewtrepte,
ooctipus,
pbarejko and
peterd-NV
as code owners
October 23, 2026 15:20
github-actions
Bot
added
documentation
rwiltz
changed the title
This was referenced Oct 23, 2026
Contributor
 [Medium impact] This update appears safe to merge; no new actionable issue was found, and the numbered previous finding is fixed. SummaryThe PR moves keyboard, gamepad, and SpaceMouse scripts toward Isaac Capture while keeping legacy fallbacks.
Reviews (3) · Last reviewed commit: "Move add_key_callbacks into isaaclab.vis..." · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
rwiltz
force-pushed
the
rwiltz/teleop-scripts-isaac-capture
branch
from
October 23, 2026 15:36
d3ce34f to
16d7e5a
Compare
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Isaac Lab Review Bot
Needs attention: Design and architecture (1), Compatibility and deprecation (1), Implementation (3), Style consistency (2), Test quality (1). Other reviewed categories look good.
- PR description: The title undercovers the 71-file visualizer, input, and dependency diff; remove empty “Fixes #”, reconcile intentional deprecation warnings with the checklist, and list focused validation.
- Design and architecture: The generic add_key_callbacks helper depends only on visualizers but lives in optional isaaclab_teleop, forcing non-teleop hotkey consumers through the wrong package; move it to isaaclab.visualizers.
- Scope and over-engineering: No over-engineering issues.
- API: API changes look good.
- Compatibility and deprecation: Breaking changes: automatic Se3GamepadCfg routing reverses Y and yaw relative to legacy Se3Gamepad; preserve signs before enabling replacement, while constructor deprecations otherwise retain functional warnings.
- Implementation: IsaacTeleopDevice translates R into RESET but script_input promises add_callback("R"), so consolidated_demo loses reset under Capture; register the actual RESET callback or preserve R dispatch.
- Style consistency: The three added package deprecation directives omit Sphinx’s required version argument; documentation rendering can fail, so add the release version.
- Test quality: The device-selection fixture replaces routing with its own expected map, making builder-name assertions self-fulfilling; preserve production routing and assert observable forwarding.
Significant concerns. Posted 8 inline findings.
Automated review; human maintainers own approval decisions.
rwiltz
force-pushed
the
rwiltz/teleop-scripts-isaac-capture
branch
2 times, most recently
from
October 23, 2026 16:03
8b1a343 to
3aa3e49
Compare
Contributor
Author
Contributor
Author
Contributor
Author
|
run-ci |
isaaclab-bot
Bot
added
ci:run-docker
Kit and Newton visualizers now connect keyboard input when their window initializes. Keyboard input is optional, so a failure there is logged and key_event_source stays None instead of stopping the visualizer from starting.
Add builders for keyboard, gamepad and SpaceMouse SE(2)/SE(3) Isaac Capture pipelines. The keyboard is read inside the teleop session from whichever visualizer window has focus (no input group, no keys from other apps); while a pipeline reads it, the session turns off the visualizers' keyboard camera movement. The gamepad and SpaceMouse configs start the Isaac Capture gamepad and spacemouse plugins with the session, which read /dev/input/jsN and /dev/hidrawN. Like every Isaac Capture session they open an OpenXR session; without a headset they run on the standalone CloudXR profile. IsaacTeleopDevice now starts, pauses and resets teleoperation on the B, P and R keys typed into a visualizer window and resets on a SpaceMouse pipeline's right button, the bindings the teleop scripts made themselves through a Carb keyboard. Each key press is handled once, whatever the retargeting mode. device_selection translates an isaaclab.devices keyboard, gamepad or SpaceMouse config into the equivalent pipeline, so a script can serve --teleop_device and an environment's teleop_devices entries through Isaac Capture.
teleop_se3_agent.py and record_demos.py now serve keyboard, gamepad and SpaceMouse devices through Isaac Capture when the isaacteleop package is installed, and through the isaaclab.devices devices otherwise (e.g. on aarch64 or Windows). --teleop_device keeps its meaning: an entry of the environment's teleop_devices, or a built-in device. Environments need no changes. The scripts no longer create a Carb keyboard for the B, P and R control keys, which the device now handles.
…vices The keyboard, gamepad and SpaceMouse sources and the gamepad and spacemouse plugins first ship in Isaac Capture 1.7, published as isaaccapture (isaacteleop renamed; it still provides the isaacteleop import). The teleop extra now installs isaaccapture~=1.7.44a1 instead of isaacteleop~=1.4.145. uv.lock adds isaaccapture and cupy-cuda12x and moves the isaacteleop alias package to the same release. The docs point the release branch, web client and Haptikos plugin links at 1.7.x.
record_demos.py now applies the R callback's guard to an Isaac Capture reset: while a successful demo is still recording its post-success steps, the reset is refused with the same message as for the isaaclab.devices keyboard. Other resets, including the host reset after each episode, are handled as before. The session checks for unattached visualizer windows on every step, not only until the first one attaches, so a window that initializes later also gets keyboard input.
uv.lock is regenerated from main's lock with uv 0.13.5, which keeps lock revision 5 and viser's ipywidgets dependency; only isaaccapture, cupy-cuda12x and the isaacteleop alias package change. The B / P / R key codes are class constants at the top of IsaacTeleopDevice, and the device-selection test records the builders the routing table actually calls rather than names the test supplies.
replay_demos.py and annotate_demos.py read their hotkeys from the visualizer windows with add_key_callbacks, without a teleop device. pick_and_place.py and consolidated_demo.py open their keyboard or SpaceMouse with open_teleop_device: an Isaac Capture pipeline on the standalone CloudXR profile when isaacteleop is installed, and the isaaclab.devices device otherwise. Their key bindings are unchanged. consolidated_demo.py also reads the device command as the 7-element tensor the devices return, instead of unpacking it into a pose and a gripper flag, which raised.
Se2/Se3Keyboard, Se2/Se3Gamepad and Se2/Se3SpaceMouse now emit a DeprecationWarning pointing to the isaaclab_teleop builders that replace them. Their configs are not deprecated: environments keep declaring their devices with them, and the teleop scripts run those through the equivalent Isaac Capture pipeline when isaacteleop is installed.
pick_and_place.py and consolidated_demo.py read the device's control events, as teleop_se3_agent.py does: while an Isaac Capture device is paused (P, or after R), pick_and_place.py renders without stepping and consolidated_demo.py holds the robot still. The isaaclab.devices devices report no control events, so they are unaffected. add_key_callbacks tracks held keys across all visualizer windows, so a press several visualizers report (Kit visualizers share the app window) fires its callback once instead of once per visualizer.
isaaclab_teleop comes only with the teleop and mimic extras, so replay_demos.py and pick_and_place.py could no longer run on a base install. add_key_callbacks uses only the simulation's visualizers, so it now lives in isaaclab.visualizers next to KeyEventSource, and those scripts read their hotkeys from there. pick_and_place.py opens its keyboard through isaaclab_teleop when it is installed and through isaaclab.devices otherwise, and reads pause from the device's control events without importing isaaclab_teleop.
rwiltz
force-pushed
the
rwiltz/teleop-scripts-isaac-capture
branch
from
October 24, 2026 03:06
9f87fdf to
c3147df
Compare
ooctipus
added a commit
that referenced
this pull request
Oct 24, 2026
# Description Adds a keyboard input API to visualizers, so code can read the keys pressed in a visualizer's window, e.g. for keyboard teleoperation or script hotkeys, independent of Kit/carb. - `BaseVisualizer.key_event_source` returns a `KeyEventSource` for the visualizer's window, or `None` when it has none (the default; also before initialization, after close, and headless). It is created on first access and closed with the visualizer. - `KeyEventSource.add_key_listener(on_key, on_focus_lost)` returns a closable `KeyboardSubscription`. Presses and releases arrive as W3C `KeyboardEvent.code` strings (`"KeyW"`), forwarded as the window reports them; `on_focus_lost` runs on blur, when the visualizer's UI takes the keyboard, and on close. Subscribing, closing and callbacks all happen on the visualizer's thread. - `BaseVisualizer.set_camera_keys_enabled(enabled)` lets the caller turn off keyboard camera movement (Newton WASD/QE and arrow keys) while it uses those keys; a no-op by default. - Implemented for Kit (physical keys; Kit visualizers share the app window, so each reports the same keys) and for Newton GL and Newton RTX (one pyglet viewer window; US-layout key mapping, presses typed into the ImGui UI are withheld). Viser, Rerun and headless runs return `None`. - Each visualizer closes its keyboard source first in `close()`, while the window is still open; a failure there is logged and the rest of the teardown still runs. Opt-in: nothing is hooked until a listener subscribes, and existing visualizer behavior is unchanged. Used by #8797 (keyboard, gamepad and SpaceMouse teleop through Isaac Capture) and #8798 (remaining scripts and deprecation of the `isaaclab.devices` keyboard, gamepad and SpaceMouse devices). Fixes # (issue) ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Release backport - [ ] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` ## Checklist Docker and GPU tests run on demand. Push the commits you want tested, then comment `run-ci` on the pull request. - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump package versions — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: ooctipus <zhengyuz@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stacked on #8797, which is stacked on #8158; review those first. This PR's own changes are the last two commits
(compare).
Finishes moving keyboard, gamepad and SpaceMouse input off the
isaaclab.devicesdevices:isaaclab_teleop.script_inputadds two helpers for scripts.add_key_callbackscalls a script's hotkey callbacks for keys pressed in a visualizer window, with no teleop session.open_teleop_deviceopens a keyboard, gamepad or SpaceMouse through Isaac Capture (standalone CloudXR profile, no headset) when
isaacteleopis installed, and throughisaaclab.devicesotherwise.replay_demos.pyandannotate_demos.pyread their hotkeys withadd_key_callbacks;pick_and_place.pyandconsolidated_demo.pyopen their device withopen_teleop_device. Key bindings areunchanged.
consolidated_demo.pyalso reads the device command as the 7-element tensor the devices return; it unpacked it into two values, which raised.Se2/Se3Keyboard,Se2/Se3GamepadandSe2/Se3SpaceMouseare deprecated: their constructors emit aDeprecationWarningpointing to theisaaclab_teleopbuilders. Their configs are not deprecated;environments keep declaring devices with them.
Fixes # (issue)
Type of change
Checklist
Docker and GPU tests run on demand. Push the commits you want tested, then
comment
run-cion the pull request.pre-commitchecks withuv run isaaclab --formatsource/<pkg>/changelog.d/for every touched package (do not editCHANGELOG.rstor bump package versions — CI handles that)CONTRIBUTORS.mdor my name already exists there