Move the remaining keyboard scripts to Isaac Capture and deprecate the isaaclab.devices keyboard, gamepad and SpaceMouse

docs/source/api/lab/isaaclab.visualizers.rst

@@ -40,6 +40,8 @@ The following classes are part of the public :mod:`isaaclab.visualizers` API.
 .. autoclass:: KeyboardSubscription
    :members: closed, close
 
+.. autofunction:: add_key_callbacks
+
 .. autoclass:: VisualizerCfg
    :show-inheritance:
 

docs/source/api/lab_teleop/isaaclab_teleop.rst

@@ -30,6 +30,12 @@ isaaclab_teleop
     create_isaac_teleop_device
     create_haptic_feedback_driver
     remove_camera_configs
+    se2_gamepad_teleop_cfg
+    se2_keyboard_teleop_cfg
+    se2_spacemouse_teleop_cfg
+    se3_gamepad_teleop_cfg
+    se3_keyboard_teleop_cfg
+    se3_spacemouse_teleop_cfg
 
 Configuration
 -------------
@@ -64,6 +70,27 @@ Device
 
 .. autofunction:: create_isaac_teleop_device
 
+Keyboard
+--------
+
+.. autofunction:: se2_keyboard_teleop_cfg
+
+.. autofunction:: se3_keyboard_teleop_cfg
+
+Gamepad
+-------
+
+.. autofunction:: se2_gamepad_teleop_cfg
+
+.. autofunction:: se3_gamepad_teleop_cfg
+
+SpaceMouse
+----------
+
+.. autofunction:: se2_spacemouse_teleop_cfg
+
+.. autofunction:: se3_spacemouse_teleop_cfg
+
 Haptic Feedback
 ---------------
 

docs/source/features/imitation-learning/teleop_imitation.rst

@@ -88,7 +88,8 @@ Teleoperation
 
 Isaac Lab supports teleoperation of robots through a variety of input devices including keyboard, SpaceMouse, and XR headsets.
 
-Run the following command to spin up the environment and teleoperate the robot using the keyboard. Familiarize yourself
+Run the following command to spin up the environment and teleoperate the robot using the keyboard: click the
+visualizer window and type (see :ref:`isaac-teleop-keyboard`). Familiarize yourself
 with the controls and perform the stacking task. The order of the stacked cubes should be blue (bottom), red (middle), green (top).
 Once you feel sufficiently comfortable with the controls, you may shutdown
 the environment by quitting the script with Ctrl+C.
@@ -145,12 +146,15 @@ the key bindings are:
       Move arm along z-axis: Push or pull the SpaceMouse
       Rotate arm: Twist the SpaceMouse
 
+.. _isaac-teleop-spacemouse-permissions:
+
 .. tip::
 
-   If the SpaceMouse is not detected, you most likely need additional user permissions. The ``hidapi``
-   wheel installed by Isaac Lab bundles a backend that talks to the device over ``libusb``, so it needs
-   read and write access to the USB node under ``/dev/bus/usb`` -- granting access to ``/dev/hidraw*``
-   alone is **not** sufficient, and without USB access the device is enumerated without a product name.
+   If the SpaceMouse is not detected, you most likely need additional user permissions. Isaac Capture
+   reads the SpaceMouse from its HID node (``/dev/hidraw*``). Without the ``isaacteleop`` package, the
+   :class:`~isaaclab.devices.Se3SpaceMouse` reads it through the ``hidapi`` wheel's ``libusb``
+   backend instead, which needs read and write access to the USB node under ``/dev/bus/usb``; without
+   it the device is enumerated without a product name. The rule below grants both.
 
    Grant the permission by installing a udev rule for the 3Dconnexion vendor id:
 

docs/source/features/isaac_teleop.rst

@@ -303,7 +303,8 @@ interchangeable, and :func:`~isaaclab_teleop.poll_control_events` stays authorit
    A clientless CloudXR runtime advertises no HMD, so the standalone OpenXR session needs a device
    profile that reports a system without a connected client. The scripts default ``--cloudxr_env``
    to :data:`~isaaclab_teleop.CLOUDXR_STANDALONE_ENV` when ``--xr`` is omitted (and to ``cloudxrjs``
-   when it is passed); see :ref:`isaac-teleop-cloudxr-profiles`.
+   when it is passed); see :ref:`isaac-teleop-cloudxr-profiles`. Keyboard, gamepad and SpaceMouse
+   sessions use the same standalone profile (see :ref:`isaac-teleop-keyboard`).
 
 .. _isaac-teleop-so101-leader-example:
 
@@ -343,9 +344,9 @@ Prerequisites
      sudo apt-get update
      sudo apt-get install -y build-essential cmake libx11-dev clang-format-14 ccache patchelf
 
-  Then clone, check out the release branch matching the ``isaacteleop`` version Isaac Lab is
-  pinned to (``isaacteleop~=1.4.145`` in the ``teleop`` extra of the root ``pyproject.toml``, so the
-  plugin's wire format matches the ``isaacteleop`` package Isaac Lab installs), configure, build,
+  Then clone, check out the release branch matching the ``isaaccapture`` version Isaac Lab is
+  pinned to (``isaaccapture~=1.7.44a1`` in the ``teleop`` extra of the root ``pyproject.toml``, so the
+  plugin's wire format matches the ``isaaccapture`` package Isaac Lab installs), configure, build,
   and install. To target a specific Python version, pass
   ``-DISAAC_TELEOP_PYTHON_VERSION=3.12`` (or ``3.11``, ``3.13``) on the configure line; each
   version needs its own build directory if building multiple at once:
@@ -354,7 +355,7 @@ Prerequisites
 
      git clone https://github.com/NVIDIA/IsaacCapture.git
      cd IsaacCapture
-     git checkout release/1.4.x
+     git checkout release/1.7.x
 
      cmake -B build                       # configure (default: Python 3.11)
      cmake --build build --parallel       # build
@@ -611,7 +612,7 @@ Source Nodes
 Available Retargeters
 ~~~~~~~~~~~~~~~~~~~~~
 
-Retargeters are provided by the ``isaacteleop`` package from the
+Retargeters are provided by the ``isaaccapture`` package from the
 `Isaac Capture <https://github.com/NVIDIA/IsaacCapture>`_ repository. The retargeters listed below
 are those used by the built-in Isaac Lab environments. Isaac Capture may offer additional
 retargeters not listed here -- refer to the
@@ -748,13 +749,58 @@ These environments use the Isaac Capture XR pipeline with motion controllers or
    ideal for tasks that need a gripper or simple hand mapping. **Hand tracking** captures 26
    wrist and finger joints per hand, required for dexterous retargeting to complex robot hands.
 
-Keyboard and SpaceMouse Environments
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _isaac-teleop-keyboard:
+
+Keyboard, SpaceMouse, and Gamepad Environments
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. note::
 
-   Keyboard and SpaceMouse teleoperation uses the legacy native Isaac Lab teleop stack
-   (``isaaclab.devices``), not Isaac Capture. These environments do not require an XR headset.
+   Keyboard, SpaceMouse, and Gamepad teleoperation run through Isaac Capture like every other
+   input device, driven by a keyboard, spacemouse, or gamepad
+   :class:`~isaaclab_teleop.IsaacTeleopDevice` pipeline instead of an XR headset -- see
+   :func:`~isaaclab_teleop.se3_keyboard_teleop_cfg`, :func:`~isaaclab_teleop.se3_spacemouse_teleop_cfg`,
+   and :func:`~isaaclab_teleop.se3_gamepad_teleop_cfg`. These environments do not require an XR
+   headset. An environment's ``teleop_devices`` entries (:class:`~isaaclab.devices.Se3KeyboardCfg`,
+   ...) run through the equivalent pipeline too. Without the ``isaaccapture`` package, the scripts
+   fall back to the :mod:`isaaclab.devices` keyboard, gamepad and SpaceMouse devices.
+
+**Keyboard input and window focus.** The keyboard runs inside the Isaac Lab process and reads the
+keys typed into whichever visualizer window has focus, like any desktop app: click the window and
+type. It needs no ``input`` group membership, ignores keys typed into other apps, and releases
+held keys when the window loses focus. Like every Isaac Capture session, a keyboard session opens an
+OpenXR session: without ``--xr`` the scripts launch a CloudXR runtime with the standalone profile
+(see :ref:`isaac-teleop-standalone`), so no headset is needed.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 75
+
+   * - Visualizer
+     - Keyboard input
+   * - ``--viz kit``
+     - Keys typed into the Kit window, including keys typed into Kit's own UI (e.g. a text field).
+   * - ``--viz newton_gl`` / ``--viz newton_rtx``
+     - Keys typed into the Newton viewer window, except keys typed into its UI. While a keyboard
+       pipeline runs, the viewer's WASD/QE and arrow-key camera movement is turned off; the mouse
+       still moves the camera.
+   * - ``--viz viser`` / ``--viz rerun`` / ``--viz none``
+     - None.
+
+**SpaceMouse and Gamepad.** Both are read by an Isaac Capture plugin that their configs start
+with the session, so they also run headset-free from a single command. The ``gamepad`` plugin
+reads the first connected joystick (``/dev/input/jsN``), which udev lets the logged-in user read
+without the ``input`` group. The ``spacemouse`` plugin reads the SpaceMouse's HID device
+(``/dev/hidrawN``), which needs a udev rule (see
+:ref:`SpaceMouse permissions <isaac-teleop-spacemouse-permissions>`). Their buttons are sampled
+once per frame, so a very quick tap (pressed and released within one frame, more likely when the
+simulation runs slowly) can be missed; press again if the gripper or reset does not respond.
+
+The ``B`` / ``P`` / ``R`` start/pause/reset keys work with every pipeline, including XR, SpaceMouse
+and Gamepad ones: :class:`~isaaclab_teleop.IsaacTeleopDevice` reads them straight from the focused
+visualizer window, so each press fires once whatever the retargeting mode. Script hotkeys, such as the
+``N`` / ``B`` / ``S`` keys of ``annotate_demos.py`` and ``replay_demos.py``, are read from the visualizer
+windows the same way (see :func:`~isaaclab.visualizers.add_key_callbacks`).
 
 The device button layouts below apply to all environments in this section. Per-environment
 differences (gripper enabled/disabled, sensitivity) are noted in the environment table that
@@ -790,9 +836,15 @@ follows.
    * - Gripper toggle
      - ``K``
      - Open / close gripper or suction (disabled in Reach envs).
+   * - Start/resume teleoperation
+     - ``B``
+     - Resume after a pause.
+   * - Pause teleoperation
+     - ``P``
+     - Pause without resetting.
    * - Reset
-     - ``L``
-     - Clear accumulated delta pose and gripper state.
+     - ``R``
+     - Reset the environment.
 
 **SpaceMouse**
 
@@ -814,7 +866,7 @@ follows.
      - Open / close gripper or suction (disabled in Reach envs).
    * - Reset
      - Right button
-     - Clear accumulated delta pose and gripper state.
+     - Reset the environment.
 
 **Gamepad** (Reach environments only)
 
@@ -838,7 +890,7 @@ follows.
      - Right stick (left/right)
      - Rotate about Z axis.
    * - Gripper toggle
-     - X button
+     - West button (X on Xbox, Square on PlayStation)
      - Open / close gripper (disabled in Reach envs).
 
 .. list-table::
@@ -1171,7 +1223,8 @@ times spike. This favors lower latency over CloudXR's pose-wait smoothing.
 :data:`~isaaclab_teleop.CLOUDXR_STANDALONE_ENV` is the default when running **without** ``--xr``
 (see :ref:`isaac-teleop-standalone`). It pins an emulated ``quest3`` device profile so a CloudXR
 runtime with no client still advertises an OpenXR system, working around
-``XR_ERROR_FORM_FACTOR_UNAVAILABLE``.
+``XR_ERROR_FORM_FACTOR_UNAVAILABLE``. Keyboard, gamepad and SpaceMouse sessions without ``--xr``
+use it too.
 
 Override at launch time
 ~~~~~~~~~~~~~~~~~~~~~~~
@@ -1683,9 +1736,9 @@ the registered task ID, but not the command-line selector values:
        --dataset_file ./datasets/dataset.hdf5 \
        physics=isaacsim_physx presets=diffik
 
-Some environments use the legacy ``teleop_devices`` configuration instead of ``isaac_teleop``
-(e.g. the Galbot RmpFlow relative-mode tasks). For these, pass ``--teleop_device`` to select
-the input device:
+Some environments declare their input devices in ``teleop_devices`` instead of ``isaac_teleop``
+(e.g. the Galbot RmpFlow relative-mode tasks). For these, pass ``--teleop_device`` to select the
+input device; keyboard and spacemouse run through Isaac Capture when ``isaacteleop`` is installed:
 
 .. tab-set::
 
@@ -1701,11 +1754,11 @@ the input device:
 The workflow is:
 
 #. Configure your environment with ``IsaacTeleopCfg`` (see :ref:`isaac-teleop-env-config`)
-   or ``teleop_devices`` for legacy devices (keyboard, spacemouse).
+   or ``teleop_devices`` for keyboard and spacemouse.
 #. Run ``record_demos.py`` with the task name and any ``physics=``, ``renderer=``, or ``presets=``
    selectors required by the task.
 #. For XR tasks: start AR, connect your XR device, and teleoperate.
-   For legacy tasks: use the configured input device directly.
+   For keyboard/spacemouse: use the input device directly.
 #. Demonstrations are recorded to HDF5 files.
 #. Use the recorded data with Isaac Lab Mimic or other imitation learning frameworks.
 

docs/source/how-to/cloudxr_teleoperation.rst

@@ -382,12 +382,12 @@ choose the tab that matches your hardware.
          start automatically.
 
       #. Open the browser on your headset and navigate to the hosted CloudXR.js client:
-         `<https://nvidia.github.io/IsaacCapture/client/release-1.4.x/>`_.
+         `<https://nvidia.github.io/IsaacCapture/client/release-1.7.x/>`_.
 
          .. note::
 
-            The web client URL is versioned. The ``release-1.4.x`` path corresponds to the
-            ``isaacteleop`` version pinned by Isaac Lab (``isaacteleop~=1.4.145`` in the ``teleop``
+            The web client URL is versioned. The ``release-1.7.x`` path corresponds to the
+            ``isaaccapture`` version pinned by Isaac Lab (``isaaccapture~=1.7.44a1`` in the ``teleop``
             extra of the root ``pyproject.toml``). When Isaac Lab updates its teleoperation
             dependency, update this link to the matching client release.
 
@@ -594,16 +594,16 @@ Prerequisites
      ``install/plugins/manus/manus_hand_plugin`` does not exist in your Isaac Capture checkout, this
      step has not been completed.
 
-  Clone the repository and check out the release branch matching the ``isaacteleop`` version Isaac
-  Lab is pinned to (``isaacteleop~=1.4.145`` in the ``teleop`` extra of the root
-  ``pyproject.toml``), so the plugin's wire format matches the ``isaacteleop`` package Isaac Lab
+  Clone the repository and check out the release branch matching the ``isaaccapture`` version Isaac
+  Lab is pinned to (``isaaccapture~=1.7.44a1`` in the ``teleop`` extra of the root
+  ``pyproject.toml``), so the plugin's wire format matches the ``isaaccapture`` package Isaac Lab
   installs:
 
   .. code-block:: bash
 
      git clone https://github.com/NVIDIA/IsaacCapture.git
      cd IsaacCapture
-     git checkout release/1.4.x
+     git checkout release/1.7.x
 
   .. note::
 
@@ -765,7 +765,7 @@ time.
 Haptikos Exoskeletons with Quest
 --------------------------------
 
-The `Haptikos plugin <https://github.com/NVIDIA/IsaacCapture/tree/release/1.4.x/src/plugins/haptikos>`_
+The `Haptikos plugin <https://github.com/NVIDIA/IsaacCapture/tree/release/1.7.x/src/plugins/haptikos>`_
 combines controller wrist poses with exoskeleton finger tracking from the Haptikos Core App and
 pushes hand joints into the OpenXR runtime. Isaac Lab receives them through Isaac Capture's
 standard hand-tracking input, so no Haptikos-specific Isaac Lab device is needed. The plugin
@@ -776,18 +776,18 @@ Build the plugin
 ^^^^^^^^^^^^^^^^
 
 The Haptikos plugin is built from Isaac Capture source; it is not included in Isaac Lab's
-``teleop`` extra. Check out the release branch matching Isaac Lab's ``isaacteleop`` pin (currently
-``1.4.x``), obtain the `Haptikos Robotics API <https://github.com/Haptikostech/HaptikosAPI>`_,
+``teleop`` extra. Check out the release branch matching Isaac Lab's ``isaaccapture`` pin (currently
+``1.7.x``), obtain the `Haptikos Robotics API <https://github.com/Haptikostech/HaptikosAPI>`_,
 and copy its ``HaptikosCpp_API_Shared`` directory into ``src/plugins/haptikos``. The C++ API is
 required to build the tracking plugin, even if you do not use haptic feedback. Follow the
-`plugin's setup instructions <https://github.com/NVIDIA/IsaacCapture/tree/release/1.4.x/src/plugins/haptikos>`_
+`plugin's setup instructions <https://github.com/NVIDIA/IsaacCapture/tree/release/1.7.x/src/plugins/haptikos>`_
 for Haptikos account and licensing requirements.
 
 .. code-block:: bash
 
    git clone https://github.com/NVIDIA/IsaacCapture.git
    cd IsaacCapture
-   git checkout release/1.4.x
+   git checkout release/1.7.x
    # Copy HaptikosCpp_API_Shared to src/plugins/haptikos before building.
    cmake -S . -B build -DENABLE_CLANG_FORMAT_CHECK=OFF
    cmake --build build --target haptikos_hands_plugin --parallel 4

examples/demos/pick_and_place.py

@@ -8,6 +8,7 @@
 from __future__ import annotations
 
 import argparse
+import contextlib
 from collections.abc import Sequence
 
 import torch
@@ -45,7 +46,7 @@
 from isaaclab.sim import SimulationCfg
 from isaaclab.utils import configclass, index_fill_, instantiate, replace
 from isaaclab.utils.math import sample_uniform
-from isaaclab.visualizers import VisualizerCfg
+from isaaclab.visualizers import VisualizerCfg, add_key_callbacks
 
 from isaaclab_assets.robots.pick_and_place import PICK_AND_PLACE_CFG
 
@@ -326,30 +327,46 @@ def main() -> None:
         pick_and_place.reset()
         actions = torch.zeros((pick_and_place.num_envs, 4), device=pick_and_place.device)
         teleop = None
+        teleop_device = contextlib.ExitStack()
         if pick_and_place.sim.has_gui:
             teleop_cfg = Se3KeyboardCfg(pos_sensitivity=10.0, sim_device=pick_and_place.device)
-            teleop = instantiate(teleop_cfg)
-            teleop.add_callback("N", lambda: pick_and_place.auto_aim(cube=True))
-            teleop.add_callback("M", lambda: pick_and_place.auto_aim(cube=False))
+            try:
+                from isaaclab_teleop.script_input import open_teleop_device
+            except ImportError:  # without the teleop or mimic extra
+                teleop = instantiate(teleop_cfg)
+            else:
+                teleop = teleop_device.enter_context(open_teleop_device(teleop_cfg))
+            add_key_callbacks(
+                {"N": lambda: pick_and_place.auto_aim(cube=True), "M": lambda: pick_and_place.auto_aim(cube=False)}
+            )
             print(teleop)
             print("Pick up the purple cube and drop it on the red sphere, in ALL environments at once.")
             print("\tW/S and A/D move the gantries, Q/E latch them UP/DOWN, K toggles the grippers.")
             print("\tN/M make the grippers track the cube/target position.")
         step_count = 0
+        teleop_active = True
         try:
             while pick_and_place.sim.is_running() and (args_cli.max_steps < 0 or step_count < args_cli.max_steps):
-                if teleop is not None:
-                    cmd = teleop.advance()
+                cmd = teleop.advance() if teleop is not None else None
+                # P / B pause and resume an Isaac Capture device; paused, the scene renders without stepping
+                events = getattr(teleop, "last_control_events", None)
+                if events is not None and events.is_active is not None:
+                    teleop_active = events.is_active
+                if not teleop_active:
+                    pick_and_place.sim.render()
+                    continue
+                if cmd is not None:
                     actions[:, :2] = cmd[:2]
                     # Latch the gantry height effort; the z joint moves up for negative effort
                     if cmd[2] != 0:
                         actions[:, 2] = -200.0 if cmd[2] > 0 else 100.0
-                    # Se3Keyboard reports +1 for open, the surface gripper uses -1 for open
+                    # the keyboard reports +1 for open, the surface gripper uses -1 for open
                     actions[:, 3] = -cmd[6]
                 with torch.inference_mode():
                     pick_and_place.step(actions)
                 step_count += 1
         finally:
+            teleop_device.close()
             pick_and_place.close()