Add keyboard input from the focused visualizer window

docs/source/api/lab/isaaclab.visualizers.rst

@@ -21,6 +21,8 @@ The following classes are part of the public :mod:`isaaclab.visualizers` API.
    :nosignatures:
 
    BaseVisualizer
+   KeyEventSource
+   KeyboardSubscription
    VisualizerCfg
    VisualizerBackendCfg
    WindowCfg
@@ -32,6 +34,12 @@ The following classes are part of the public :mod:`isaaclab.visualizers` API.
 .. autoclass:: BaseVisualizer
    :show-inheritance:
 
+.. autoclass:: KeyEventSource
+   :members: closed, add_key_listener, close
+
+.. autoclass:: KeyboardSubscription
+   :members: closed, close
+
 .. autoclass:: VisualizerCfg
    :show-inheritance:
 

docs/source/concepts/visualization.rst

@@ -706,6 +706,63 @@ Rerun cannot.
 
 See :doc:`/source/how-to/record_video` for the full guide and tutorial.
 
+Keyboard Input
+~~~~~~~~~~~~~~
+
+A visualizer with a local window reports the keys typed into it through
+:attr:`~isaaclab.visualizers.BaseVisualizer.key_event_source`, for example to drive a robot from
+the keyboard. Each key press and release arrives as a W3C ``KeyboardEvent.code`` string such as
+``"KeyW"``:
+
+- :class:`~isaaclab_visualizers.kit.KitVisualizer` reports physical key positions, independent of
+  the keyboard layout. Keys typed into Kit's own UI, such as a text field, are reported too.
+- :class:`~isaaclab_visualizers.newton.NewtonGLVisualizer` and
+  :class:`~isaaclab_visualizers.newton.NewtonRTXVisualizer` report layout-mapped keys: a code names
+  the key that types the same character on a US layout. Presses typed into the window's UI are
+  withheld. Presses and releases pair up reliably on a US layout; on other layouts, changing Shift
+  while holding a digit or punctuation key can leave it held until the window loses focus.
+
+The Viser and Rerun visualizers, and any visualizer running headless, have no keyboard source and
+return ``None``. A headless Viser server still takes input from connected browsers, but not through
+``key_event_source``.
+
+Each Newton window has its own source. Kit visualizers share the Kit application window, so each of
+them reports the same keys; a consumer that listens to every visualizer must accept a key from more
+than one source.
+
+The visualizer connects native keyboard and focus events when its window initializes and
+disconnects them before the window closes. A subscription only adds or removes a listener;
+it does not initialize window resources. The source owns no held-key state or camera policy.
+
+Close the subscription when you are done with it. A consumer that uses the camera's motion keys
+also turns off keyboard camera movement with
+:meth:`~isaaclab.visualizers.BaseVisualizer.set_camera_keys_enabled`; a Newton window then stops
+moving its camera with WASD/QE and the arrow keys, and Kit's viewport, whose camera keys only act
+while the right mouse button is held, is unaffected:
+
+.. code-block:: python
+
+   visualizer = sim.visualizers[0]
+   source = visualizer.key_event_source
+   if source is not None:
+       held = set()
+
+       def on_key(code: str, pressed: bool) -> None:
+           (held.add if pressed else held.discard)(code)
+
+       visualizer.set_camera_keys_enabled(False)
+       try:
+           with source.add_key_listener(on_key, held.clear):
+               ...  # use held while stepping the simulation
+       finally:
+           visualizer.set_camera_keys_enabled(True)
+
+Subscribe, unsubscribe and close on the thread that steps the visualizer; callbacks run there too,
+in event order. Events arrive as the window reports them: key repeat can arrive as further presses,
+and a release can arrive without its press, so a listener tracks the keys it holds.
+``on_focus_lost`` runs when the window stops delivering keys, because it lost focus, its UI took
+the keyboard, or the visualizer closed; treat every held key as released.
+
 
 .. _visualization-configuration:
 
@@ -945,6 +1002,12 @@ Limitations
      - ✓
      - ✓
      - ✓
+   * - Keyboard input, ``key_event_source``
+     - ✓
+     - ✓
+     - ✗
+     - ✗
+     - ✓
 
 **Lighting differences across visualizers**
 

source/_isaaclab_testing/test/rendering/test_visualizers_kit.py

@@ -141,6 +141,9 @@ def test_visualizer_images_and_motion(scene, mode, physics, request, caplog, cap
         assert runtime.sim.is_playing()
         assert omni.timeline.get_timeline_interface().is_playing()
         kit, gl = runtime.sim.visualizers
+        if (scene, mode, physics) == ("cartpole", "viewport", "physx"):
+            assert kit.key_event_source is not None, "Window input is missing."
+            assert gl.key_event_source is None, "The headless viewer must not connect window input."
         if scene == "cartpole":
             robot = runtime.scene["robot"]
             robot.write_joint_position_to_sim_index(position=robot.data.default_joint_pos.torch)

source/isaaclab/changelog.d/minimal-window-keyboard.changed.rst

@@ -0,0 +1,4 @@
+* Simplified keyboard sources to listener registration and event forwarding; native window hooks
+  were initialized and closed by visualizers instead of changing with listener counts.
+* Centralized the optional keyboard source and teardown in the base visualizer and removed
+  redundant subscription state.

source/isaaclab/changelog.d/rwiltz-visualizer-key-events.added.rst

@@ -0,0 +1,8 @@
+* Added :attr:`~isaaclab.visualizers.BaseVisualizer.key_event_source` and
+  :class:`~isaaclab.visualizers.KeyEventSource`, keyboard input from a focused visualizer window.
+  :meth:`~isaaclab.visualizers.KeyEventSource.add_key_listener` returns a
+  :class:`~isaaclab.visualizers.KeyboardSubscription` that receives each key press and release as a
+  W3C ``KeyboardEvent.code`` string, and a focus loss when the window stops delivering keys.
+  Visualizers without a local window that can report keys return ``None``, the default.
+* Added :meth:`~isaaclab.visualizers.BaseVisualizer.set_camera_keys_enabled`, which turns a
+  visualizer's keyboard camera movement off while a keyboard consumer uses those keys.

source/isaaclab/isaaclab/visualizers/__init__.pyi

@@ -7,6 +7,8 @@ __all__ = [
     "BaseVisualizer",
     "ImageView",
     "ImageViewCfg",
+    "KeyEventSource",
+    "KeyboardSubscription",
     "PerspectiveCameraCfg",
     "SceneCameraCfg",
     "VisualizerCfg",
@@ -16,4 +18,5 @@ __all__ = [
 
 from .base_visualizer import BaseVisualizer
 from .image_view import ImageView
+from .key_event_source import KeyboardSubscription, KeyEventSource
 from .visualizer_cfg import WindowCfg, ImageViewCfg, PerspectiveCameraCfg, SceneCameraCfg, VisualizerCfg, VisualizerBackendCfg

source/isaaclab/isaaclab/visualizers/base_visualizer.py

@@ -22,6 +22,7 @@
 from ..utils import validate
 from ..utils.backend_utils import FactoryBase
 from .image_view import ImageView
+from .key_event_source import KeyEventSource
 from .visualizer_cfg import (
     ImageViewCfg,
     PerspectiveCameraCfg,
@@ -71,6 +72,7 @@ def __init__(self, cfg: VisualizerCfg):
         self._sim_time = 0.0
         self.current_image_view: ImageView | None = None
         self._image_views: list[ImageView] = []
+        self._key_event_source: KeyEventSource | None = None
 
     @property
     def visual_material_writer(self) -> Callable[[tuple[VisualMaterialBatch, ...]], Any] | None:
@@ -231,6 +233,29 @@ def physics_backend(self) -> str | None:
         """
         return FactoryBase._get_backend(sim=self._sim) if self._sim is not None else None
 
+    @property
+    def key_event_source(self) -> KeyEventSource | None:
+        """Keyboard input from this visualizer's local window, or ``None`` when there is none.
+
+        Consumers subscribe to the keys pressed and released while the window has focus, e.g. for
+        keyboard teleoperation or script hotkeys. The source connects when the visualizer initializes
+        its window, and the visualizer closes it before releasing that window. ``None``
+        before initialization, after close, and for backends or modes with no local window that can
+        report keys (the default).
+        """
+        return self._key_event_source
+
+    def set_camera_keys_enabled(self, enabled: bool) -> None:
+        """Enable or disable moving this visualizer's camera with the keyboard.
+
+        A keyboard consumer such as keyboard teleoperation disables it, so the motion keys it uses
+        (e.g. WASD/QE and the arrow keys) do not also move the camera. Mouse camera control is
+        unaffected. Visualizers whose camera takes no keyboard input ignore it (the default).
+
+        Args:
+            enabled: Whether keyboard camera movement is enabled.
+        """
+
     def supports_markers(self) -> bool:
         """Check if visualizer supports VisualizationMarkers.
 
@@ -422,3 +447,12 @@ def pause(self) -> None:
     def stop(self) -> None:
         """Handle simulation stop. No-op by default."""
         pass
+
+    def _close_key_event_source(self) -> None:
+        """Close window input before native teardown; cleanup failures must not retain the window."""
+        source, self._key_event_source = self._key_event_source, None
+        if source is not None:
+            try:
+                source.close()
+            except Exception:
+                logger.exception("[%s] Keyboard input cleanup failed", type(self).__name__)

source/isaaclab/isaaclab/visualizers/key_event_source.py

@@ -0,0 +1,128 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Keyboard input from a focused visualizer window, shared by the visualizer backends."""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable
+
+logger = logging.getLogger(__name__)
+
+KeyCallback = Callable[[str, bool], None]
+"""Called with a W3C ``KeyboardEvent.code`` string (e.g. ``"KeyW"``) and ``True`` on press, ``False`` on release."""
+
+FocusLostCallback = Callable[[], None]
+"""Called when the window stops delivering keys: treat every held key as released."""
+
+
+class KeyboardSubscription:
+    """A listener subscribed with :meth:`KeyEventSource.add_key_listener`.
+
+    Close it (or use it as a context manager) to unsubscribe. Closing is idempotent and delivers no
+    final focus loss: the listener chose to leave.
+    """
+
+    def __init__(self, source: KeyEventSource, on_key: KeyCallback, on_focus_lost: FocusLostCallback) -> None:
+        self._source: KeyEventSource | None = source
+        self._on_key = on_key
+        self._on_focus_lost = on_focus_lost
+
+    @property
+    def closed(self) -> bool:
+        """Whether the subscription no longer receives callbacks."""
+        return self._source is None
+
+    def close(self) -> None:
+        """Unsubscribe."""
+        source, self._source = self._source, None
+        if source is not None and not source.closed:
+            source._subscriptions.remove(self)
+
+    def __enter__(self) -> KeyboardSubscription:
+        return self
+
+    def __exit__(self, *exc) -> None:
+        self.close()
+
+
+class KeyEventSource:
+    """Keyboard input from a focused visualizer window.
+
+    Obtained from :attr:`~isaaclab.visualizers.BaseVisualizer.key_event_source`. For every backend:
+
+    - Subscribing, closing and every callback happen on the thread that steps the visualizer.
+    - ``on_key`` receives the window's presses and releases as W3C ``KeyboardEvent.code`` strings.
+      Key repeat can arrive as further presses, and a release can arrive without its press (e.g. a
+      key pressed in the visualizer's UI), so a listener tracks the keys it holds.
+    - ``on_focus_lost`` runs when the window stops delivering keys: it lost focus, the visualizer's
+      UI took the keyboard, or the source closed. The listener treats every held key as released.
+    - A listener that raises is logged and does not affect other listeners.
+    - Closing the source delivers a final focus loss to every listener; afterwards
+      :meth:`add_key_listener` raises :class:`RuntimeError`.
+
+    The visualizer connects native events when its window initializes and closes the source before
+    the window closes. Backend bridges report events through :meth:`_emit_key` and
+    :meth:`_emit_focus_lost`; adding or removing a listener never changes native window resources.
+    """
+
+    def __init__(self) -> None:
+        self._subscriptions: list[KeyboardSubscription] = []
+        self._closed = False
+
+    @property
+    def closed(self) -> bool:
+        """Whether the source was closed."""
+        return self._closed
+
+    def add_key_listener(self, on_key: KeyCallback, on_focus_lost: FocusLostCallback) -> KeyboardSubscription:
+        """Subscribe to key presses and releases from the initialized window.
+
+        Args:
+            on_key: Called with the key's W3C ``KeyboardEvent.code`` and whether it was pressed.
+            on_focus_lost: Called when the window stops delivering keys.
+
+        Returns:
+            The subscription. Close it to unsubscribe.
+
+        Raises:
+            RuntimeError: If the source is closed.
+        """
+        if self._closed:
+            raise RuntimeError(f"{type(self).__name__} is closed")
+        subscription = KeyboardSubscription(self, on_key, on_focus_lost)
+        self._subscriptions.append(subscription)
+        return subscription
+
+    def close(self) -> None:
+        """Close every subscription and deliver a final focus loss to its listener."""
+        if self._closed:
+            return
+        self._closed = True
+        subscriptions, self._subscriptions = self._subscriptions, []
+        for subscription in subscriptions:
+            if not subscription.closed:
+                subscription.close()
+                _call(subscription._on_focus_lost)
+
+    def _emit_key(self, code: str, pressed: bool) -> None:
+        """Report a key press or release to every listener."""
+        for subscription in list(self._subscriptions):
+            if not subscription.closed:
+                _call(subscription._on_key, code, pressed)
+
+    def _emit_focus_lost(self) -> None:
+        """Tell every listener that the window stopped delivering keys."""
+        for subscription in list(self._subscriptions):
+            if not subscription.closed:
+                _call(subscription._on_focus_lost)
+
+
+def _call(callback: Callable, *args) -> None:
+    try:
+        callback(*args)
+    except Exception:
+        logger.exception("Keyboard listener %r failed", callback)