RealHand plugin: dexterous hands, robot arms, and data gloves

CMakeLists.txt

@@ -63,6 +63,13 @@ option(BUILD_PLUGIN_OAK_CAMERA "Build OAK camera plugin (requires vcpkg for Dept
 option(BUILD_PLUGIN_NOITOM_MOCAP "Build Noitom mocap plugin (downloads MocapApi SDK)" OFF)
 option(BUILD_PLUGIN_OGLO "Build OGLO tactile glove plugin (BLE, Linux only; fetches SimpleBLE + nlohmann/json)" OFF)
 option(BUILD_PLUGIN_WUJI_GLOVE "Build Wuji glove plugin (requires the wuji_sdk C SDK)" OFF)
+if(UNIX)
+    set(_realhand_ffg_glove_default ON)
+else()
+    set(_realhand_ffg_glove_default OFF)
+endif()
+option(BUILD_PLUGIN_REALHAND_FFG_GLOVE "Build RealHand FFG Glove USB serial plugin (POSIX only)" ${_realhand_ffg_glove_default})
+unset(_realhand_ffg_glove_default)
 option(BUILD_EXAMPLES "Build examples" ON)
 option(BUILD_EXAMPLE_TELEOP_ROS2 "Build only the teleop_ros2 ROS 2 reference integration (e.g. for Docker)" OFF)
 option(BUILD_TESTING "Build unit tests" ON)

docs/source/_data/devices.yaml

@@ -223,6 +223,25 @@ devices:
         - label: haptikos.tech
           url: https://haptikos.tech/
 
+  - id: realhand-ffg-glove
+    name: RealHand FFG Glove
+    url: https://github.com/RealHand-Robotics/FFG_realhand_pure_python
+    group: peripheral
+    modes: Finger joint sensing over USB
+    since: main
+    details:
+      setup:
+        - label: RealHand FFG Glove docs
+          doc: /device/realhand_ffg_glove
+        - label: RealHand FFG Glove Plugin
+          url: https://github.com/NVIDIA/IsaacCapture/tree/main/src/plugins/realhand_ffg_glove
+      requirements:
+        - Linux workstation
+        - USB serial access through the dialout group
+      acquire:
+        - label: RealHand FFG Glove SDK source
+          url: https://github.com/RealHand-Robotics/FFG_realhand_pure_python
+
   - id: oglo-tactile-glove
     name: OGLO Tactile Glove
     url: https://www.opengraphlabs.com/

docs/source/device/realhand_ffg_glove.rst

@@ -0,0 +1,119 @@
+.. SPDX-FileCopyrightText: Copyright (c) 2026 RealHand. All rights reserved.
+.. SPDX-License-Identifier: Apache-2.0
+
+RealHand FFG Glove
+==================
+
+The Linux RealHand FFG Glove plugin reads one or two gloves over USB serial and publishes each
+glove as a standard Isaac Teleop ``JointStateOutput`` collection. It discovers serial ports and
+baud rates, queries the firmware-reported side, and reconnects after a disconnect. Either side can
+operate alone; a missing glove does not control the opposite hand.
+
+.. contents:: On this page
+   :local:
+   :depth: 2
+
+Data flow
+---------
+
+.. code-block:: text
+
+   RealHand FFG Glove USB ─► realhand_ffg_glove plugin ─► JointStateSource ─► RealHandFFGGloveRetargeter
+                         side discovery       21 sensors       L6 / O6 / L20 joints
+
+The serial protocol follows the Apache-2.0
+`RealHand FFG Glove pure-Python SDK
+<https://github.com/RealHand-Robotics/FFG_realhand_pure_python>`_. The plugin implements the protocol
+directly in C++, so the Python SDK is not redistributed or required at runtime.
+
+Build
+-----
+
+The plugin is enabled by default on POSIX systems. Configure and build Isaac Teleop normally:
+
+.. code-block:: console
+
+   $ cmake -S . -B build
+   $ cmake --build build --target realhand_ffg_glove_plugin
+   $ cmake --install build --component realhand_ffg_glove
+
+Set ``-DBUILD_PLUGIN_REALHAND_FFG_GLOVE=OFF`` to exclude it from a POSIX build. The serial backend
+is not built by default on Windows.
+
+USB permissions
+---------------
+
+The current user must be able to open the glove's serial device. On Ubuntu, add the user to the
+``dialout`` group, then log out and back in:
+
+.. code-block:: console
+
+   $ sudo usermod -aG dialout "$USER"
+
+Do not run Isaac Teleop as root. Check permissions with ``ls -l /dev/ttyUSB*`` and membership with
+``id -nG``.
+
+Run
+---
+
+Start the CloudXR runtime, source its OpenXR environment, and launch the installed plugin:
+
+.. code-block:: console
+
+   $ python -m isaaccapture.cloudxr.service start
+   $ source ~/.cloudxr/run/cloudxr.env
+   $ ./install/plugins/realhand_ffg_glove/realhand_ffg_glove_plugin
+
+By default the plugin scans ``/dev/ttyUSB*``, ``/dev/ttyACM*``, ``/dev/ttyXRUSB*``, and
+``/dev/ttyOBC*`` using the supported baud rates. Restrict discovery when diagnosing a connection:
+
+.. code-block:: console
+
+   $ ./install/plugins/realhand_ffg_glove/realhand_ffg_glove_plugin \
+       --ports=/dev/ttyUSB0,/dev/ttyUSB1 \
+       --baudrates=2000000,460800
+
+The left and right collections are ``realhand_ffg_glove_left`` and
+``realhand_ffg_glove_right``. Each publishes ``sensor_0`` through ``sensor_20``.
+
+Retargeting and calibration
+---------------------------
+
+RealHand FFG Glove samples are glove measurements, not robot joint commands. Use one
+``RealHandFFGGloveRetargeter`` per connected side and select ``l6``, ``o6``, or ``l20``. Calibration
+must be captured while wearing the glove in its final fit. Hold each pose steadily and avoid
+pressing the fingertips together hard enough to deform the glove.
+
+For L6 and O6, capture these poses:
+
+* **Open palm:** wrist neutral, palm flat, all five digits naturally straight and separated.
+* **Full fist:** all four fingers fully flexed; thumb folded naturally across the index and middle
+  fingers rather than forced into the palm.
+* **Thumb-only curl:** four fingers stay open; move the thumb through opposition and flexion toward
+  the palm without closing another finger.
+* **Thumb to index:** touch the two fingertip pads lightly while the other fingers stay relaxed.
+* **Thumb to middle:** touch the two fingertip pads lightly while the other fingers stay relaxed.
+
+For L20, additionally capture thumb-to-ring and thumb-to-pinky poses. These samples describe the
+thumb's calibrated response surface. At runtime, thumb outputs use only thumb sensors, and every
+other digit uses only its own sensors; there is no contact snapping or cross-finger pose trigger.
+Samples that continue beyond a calibrated endpoint saturate at that endpoint instead of reopening
+the corresponding mechanical-hand joints.
+
+Capture a calibration from the standard DeviceIO stream:
+
+.. code-block:: console
+
+   $ python examples/teleop/python/realhand_ffg_glove_calibration.py \
+       --hand-model l20 --side both \
+       --plugin-path install/plugins \
+       --output /path/to/calibration_l20.yml
+
+Use ``--side left`` or ``--side right`` when one glove is connected. A later run for the other
+side updates the same YAML and preserves the existing side. Calibration files contain
+user- and glove-specific measurements. Keep them outside the source tree and pass the resulting
+path explicitly with ``--realhand-ffg-glove-calibration`` when running an FFG example.
+
+.. seealso::
+
+   :doc:`/references/retargeting/realhand` describes the P7 and RealHand output pipelines.

docs/source/index.rst

@@ -66,6 +66,7 @@ Table of Contents
    device/body_tracking
    device/haptic_feedback
    device/manus
+   device/realhand_ffg_glove
    device/oak
    device/oglo
    device/wuji_glove

docs/source/references/retargeting/index.rst

@@ -21,6 +21,14 @@ Source Nodes
 Available Retargeters
 ---------------------
 
+.. dropdown:: P7 and RealHand L6 / O6 / L20
+
+   ``P7ControllerPoseRetargeter`` and ``P7HandPoseRetargeter`` map an absolute OpenXR pose into
+   the calibrated P7 workspace. ``RealHandHandTrackingRetargeter``,
+   ``ControllerTriggerRealHandRetargeter``, and ``RealHandFFGGloveRetargeter`` produce joint commands
+   for the L6, O6, and L20 hands. See :doc:`realhand` for the full input-mode matrix and pipeline
+   example.
+
 .. dropdown:: Se3AbsRetargeter / Se3RelRetargeter
 
    Maps hand or controller tracking to end-effector pose. ``Se3AbsRetargeter`` outputs a 7D
@@ -373,3 +381,4 @@ See the :doc:`Contributing Guide <../../getting_started/contributing>` for detai
    so101
    joint_space
    wuji
+   realhand

docs/source/references/retargeting/realhand.rst

@@ -0,0 +1,128 @@
+.. SPDX-FileCopyrightText: Copyright (c) 2026 RealHand. All rights reserved.
+.. SPDX-License-Identifier: Apache-2.0
+
+Retargeters: P7 and RealHand
+============================
+
+The RealHand integration separates device acquisition, robot-independent retargeting, and
+simulation control along Isaac Teleop's existing boundaries.
+
+Components
+----------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 30 40
+
+   * - Component
+     - Input
+     - Output
+   * - ``P7ControllerPoseRetargeter``
+     - One OpenXR controller grip pose
+     - Absolute seven-value P7 end-effector pose
+   * - ``P7HandPoseRetargeter``
+     - One OpenXR hand wrist pose
+     - Absolute seven-value P7 end-effector pose
+   * - ``ControllerTriggerRealHandRetargeter``
+     - One controller trigger / squeeze value
+     - L6, O6, or L20 hand joints
+   * - ``RealHandHandTrackingRetargeter``
+     - One 26-joint OpenXR hand
+     - L6, O6, or L20 hand joints
+   * - ``RealHandFFGGloveRetargeter``
+     - One RealHand FFG Glove 21-sensor ``JointStateSource``
+     - L6, O6, or L20 hand joints
+
+P7 arm control
+--------------
+
+The P7 nodes perform only the calibrated absolute workspace transform, pose limiting, and temporal
+filtering. They deliberately do not solve robot joint angles. In Isaac Lab, feed their two
+``ee_pose`` outputs to ``PinkInverseKinematicsActionCfg`` so collision settings, joint limits, and
+the simulated robot state remain owned by the environment.
+
+RealHand control
+----------------
+
+``get_realhand_profile()`` provides the active joint limits and the combined bimanual action order
+for each hand model. Controller mode maps the analog trigger linearly from open to the model's
+closed posture. Hand-tracking mode derives each digit from OpenXR joint geometry. RealHand FFG
+Glove mode maps a calibrated glove independently per side and per digit.
+
+Run the combined example
+------------------------
+
+Install the lightweight retargeting dependency and build the package first:
+
+.. code-block:: console
+
+   $ pip install 'isaaccapture[retargeters-lite]'
+   $ cmake --build build --target python_package
+
+The example supports all three hand models and all three input modes:
+
+.. code-block:: console
+
+   $ python examples/teleop/python/p7_realhand_bimanual_example.py \
+       --mode handtracking --hand-model l20
+
+   $ python examples/teleop/python/p7_realhand_bimanual_example.py \
+       --mode controller --hand-model l6
+
+   $ python examples/teleop/python/p7_realhand_bimanual_example.py \
+       --mode ffg --hand-model o6 \
+       --plugin-path install/plugins \
+       --realhand-ffg-glove-calibration /path/to/calibration_o6.yml
+
+The action layout is always ``left_ee_pose + right_ee_pose + profile.action_joint_names``. Its
+width is 36 for L6, 26 for O6, and 46 for L20. Use ``--dry-run`` to validate graph construction
+without starting OpenXR.
+
+Robot assets
+------------
+
+The P7 + L6, O6, and L20 URDF assemblies and meshes are published separately to keep binary robot
+assets out of the Isaac Teleop source repository. Download all three assemblies from the repository
+root with:
+
+.. code-block:: console
+
+   $ python3 examples/teleop/python/scripts/fetch_realhand_assets.py
+
+To download only one hand model or to use a cache outside the source tree:
+
+.. code-block:: console
+
+   $ python3 examples/teleop/python/scripts/fetch_realhand_assets.py \
+       --hand-model l20 \
+       --output-dir ~/.cache/isaaccapture/realhand
+
+The script uses the immutable commit behind RealHand Teleop v0.2.0 and verifies the downloaded
+files before use. Personal glove calibrations, source CAD files, controller mounts, scenes, and
+demonstrations are not downloaded. Resolve the selected assembly in an Isaac Lab environment with
+``get_realhand_profile("l20").resolve_urdf(asset_root)``.
+
+Complete Isaac Lab reference package
+------------------------------------
+
+A complete reference package containing the P7 robot environments, RealHand L6, O6, and L20
+assets, and runnable controller, hand-tracking, and RealHand FFG Glove examples is available from
+the `RealHand Teleop v0.2.0 release
+<https://huggingface.co/realhandinc/realhand-teleop/tree/v0.2.0>`_.
+
+This package is optional and is not required to build or use the Isaac Teleop plugins and
+retargeters in this repository.
+
+Coordinate calibration
+----------------------
+
+``P7WorkspacePoseConfig`` exposes controller or wrist center, robot workspace center, per-axis
+scale and limits, fixed orientation and local position offsets, and filtering limits. The example
+contains the measured neutral P7 configurations used by the reference setup. A different robot
+mount, OpenXR anchor, or controller-on-glove bracket should override those values in the consuming
+Isaac Lab environment rather than changing the device plugin.
+
+.. seealso::
+
+   :doc:`/device/realhand_ffg_glove` covers RealHand FFG Glove discovery, permissions, and
+   calibration poses.

examples/teleop/python/assets/realhand/.gitignore

@@ -0,0 +1,6 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 RealHand. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+*
+!.gitignore
+!README.md

examples/teleop/python/assets/realhand/README.md

@@ -0,0 +1,30 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 RealHand. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# P7 and RealHand Robot Assets
+
+This directory receives the P7 + L6, O6, and L20 URDF assemblies and their meshes. The binary
+assets are published separately in the public
+[RealHand Teleop repository](https://huggingface.co/realhandinc/realhand-teleop) and are not stored
+in Isaac Teleop.
+
+From the Isaac Teleop repository root, download all three assemblies with:
+
+```bash
+python3 examples/teleop/python/scripts/fetch_realhand_assets.py
+```
+
+Download one assembly or choose another destination with:
+
+```bash
+python3 examples/teleop/python/scripts/fetch_realhand_assets.py \
+  --hand-model l20 \
+  --output-dir ~/.cache/isaaccapture/realhand
+```
+
+The fetcher pins the immutable commit behind RealHand Teleop v0.2.0, validates Hugging Face LFS
+objects and the main URDF, and checks that every mesh referenced by the URDF exists. It downloads
+robot assets and license notices only; personal glove calibrations, STEP files, controller mounts,
+scenes, and demonstrations are excluded.