Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 38 additions & 7 deletions docs/source/features/isaac_teleop.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1162,9 +1162,10 @@ can be referenced via constants:
- ``0``
- ``0``

All three profiles set ``NV_CXR_ENABLE_PUSH_DEVICES=0``, which is correct for headset optical hand
tracking (the most common setup). For external push-device peripherals such as Manus gloves, set
this to ``1`` in a custom profile (see below).
The ``cloudxrjs`` profile sets ``NV_CXR_ENABLE_PUSH_DEVICES=1`` to admit its WebXR
input path. The AVP and standalone profiles set it to ``0`` for headset optical
hand tracking. Custom profiles for external push-device peripherals such as
Manus gloves must enable this setting.
They also set ``NV_ENABLE_POSE_WAIT=0`` so CloudXR does not throttle the application when frame
times spike. This favors lower latency over CloudXR's pose-wait smoothing.

Expand Down Expand Up @@ -1204,8 +1205,9 @@ Copy a shipped profile and edit it:
cp $(uv run --extra teleop,isaacsim python -c \
"from isaaclab_teleop import CLOUDXR_JS_ENV; print(CLOUDXR_JS_ENV)") ~/my-cloudxr.env

Edit ``~/my-cloudxr.env`` to change any values (e.g. ``NV_CXR_ENABLE_PUSH_DEVICES=1`` for
Manus gloves), then pass it via ``--cloudxr_env ~/my-cloudxr.env``.
Edit ``~/my-cloudxr.env`` to change any values, then pass it via
``--cloudxr_env ~/my-cloudxr.env``. Keep ``NV_CXR_ENABLE_PUSH_DEVICES=0`` for headset optical hands;
enable it only for external push-device peripherals such as Manus gloves.

Disable auto-launch
~~~~~~~~~~~~~~~~~~~
Expand Down Expand Up @@ -1266,6 +1268,33 @@ through this variable -- later runs no longer prompt. The variable applies to ev
that launches the runtime, including the process-scoped launcher in
``teleop_replay_agent.py``.

.. _isaac-teleop-ovxr-mcap:

Replay MCAP with a Kitless OVXR Viewer
--------------------------------------

The MCAP replay entrypoint can present recorded input through an independent
OVXR viewer. Follow the :ref:`kitless XR guide <kitless-xr>` to install Isaac
Capture with its OVXR extra and select the compatible rendering dependencies.

.. code-block:: bash

uv run --extra ovxr python \
scripts/environments/teleoperation/teleop_replay_agent.py \
--task IsaacContrib-PickPlace-GR1T2-Abs --physics ovphysx \
--device cpu --num_envs 1 --ovxr \
--replay_file /absolute/path/to/session.mcap

Use ``--physics newton_mjwarp`` for Newton. MCAP supplies tracker and control
input; the connected client supplies the view. A recorded STOP, task success
or failure, or the replay duration limit ends the run. The separate
``isaaclab teleop replay`` command replays HDF5 demonstrations through Kit.

Add ``--ovxr_pip_camera robot_pov_cam --ovxr_pip_head_locked`` for a camera
panel, or ``--ovxr_desktop_mirror`` for the optional desktop window. See the
kitless XR guide for build prerequisites, camera selection and display requirements.


.. _isaac-teleop-xr-anchor:

Configure the XR Anchor
Expand Down Expand Up @@ -1518,8 +1547,10 @@ Kit Scene UI presentation
PiP presentation uses Kit Scene UI and ``SpatialSource`` placement. Kit imports are deferred until
an enabled feed is requested. If the Scene UI modules cannot be imported, the scripts log a warning
and continue without PiP; task-owned cameras and recording observations remain unchanged. This keeps
the camera selection configuration usable when a future kitless entry point no longer provides
Scene UI. Render-product tuning is also best-effort: backends without a compatible render product
the camera selection configuration usable without Scene UI. The
:ref:`kitless XR guide <kitless-xr>` uses separate OpenXR quad
layers and does not import Scene UI; those layers are visible in the headset/client, not in the
passive eye-image desktop mirror. Render-product tuning is also best-effort: backends without a compatible render product
keep the Camera-buffer path, and schema-authoring failures warn while PiP continues. Other
configuration, camera-buffer, and panel-initialization errors still fail during startup.

Expand Down
14 changes: 7 additions & 7 deletions docs/source/how-to/cloudxr_teleoperation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,10 @@ to capture demonstrations, and ``replay`` to play a dataset back. :ref:`The next

.. note::

``teleop`` cannot be combined with ``ov`` or ``ovphysx`` in a single ``uv run``: the
bundled Isaac Sim pins ``packaging==26.0`` while those runtimes require ``<24``. Install
the OV runtimes separately when you need them.
For XR teleoperation without Isaac Sim, follow the
:ref:`kitless XR guide <kitless-xr>`. It explains how to select a source-built
Isaac Capture wheel and its optional OVXR dependencies. The Kit setup below
uses the published ``teleop`` extra.

Complete these steps first:

Expand Down Expand Up @@ -695,10 +696,9 @@ Launch a teleoperation session paired with a hand-tracking task, as shown in

.. important::

Manus gloves and other external push-device peripherals require
``NV_CXR_ENABLE_PUSH_DEVICES=1``. The shipped ``.env`` profiles set this to ``0``
(optimised for headset optical hand tracking). Create a custom ``.env`` file with the value set
to ``1`` and pass it via ``--cloudxr_env`` instead of the plain command above:
The shipped ``cloudxrjs`` profile sets ``NV_CXR_ENABLE_PUSH_DEVICES=0`` for headset optical
hands. Manus gloves and other external push-device peripherals instead require a custom
``.env`` file with push devices enabled, passed via ``--cloudxr_env``:

.. tab-set::

Expand Down
11 changes: 11 additions & 0 deletions docs/source/how-to/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -500,6 +500,12 @@ How-to Guides

:guide-tag:`PhysX` :guide-tag:`Isaac Sim`

.. container:: guide-entry

:doc:`Source-built Capture with optional OVXR </source/how-to/kitless_xr>`

Select public OVXR providers and replay MCAP with a source-built Capture wheel.

.. container:: guide-entry

:doc:`Setting up Haply teleoperation </source/how-to/haply_teleoperation>`
Expand Down Expand Up @@ -546,3 +552,8 @@ How-to Guides

This collection is a work in progress. If a question is not answered here, open an issue on the
`Isaac Lab GitHub repository <https://github.com/isaac-sim/IsaacLab>`_.

.. toctree::
:hidden:

kitless_xr
10 changes: 10 additions & 0 deletions docs/source/how-to/kitless_xr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<!--
Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
All rights reserved.
SPDX-License-Identifier: BSD-3-Clause
-->

(kitless-xr)=

```{include} ../../../source/isaaclab_teleop/docs/OVXR_RUNTIME.md
```
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -410,7 +410,8 @@ environments = [
# ``importers`` extras pin older versions of these packages, so the overrides prevent downgrades.
# Direct torch-stack pins also apply to published wheels; overrides reconcile Isaac Sim's older pins.
override-dependencies = [
"numpy>=2",
# Numba requires <2.5; cmeel-boost used by teleop requires >=2.3,<2.4.
"numpy>=2.3,<2.4",
"mujoco~=3.15.0",
"mujoco-warp~=3.15.0",
"newton[sim] @ git+https://github.com/newton-physics/newton.git@71eeba16a717d4dcf0b9dd7056a7b6f863054dc3",
Expand Down
Loading
Loading