[pull] develop from isaac-sim:develop

.github/test-subsets/postmerge-rendering.toml

@@ -20,6 +20,8 @@ rendering-correctness-kitless-legacy = [
   "source/isaaclab_tasks/test/core/test_rendering_franka_cloth_kitless.py::test_rendering_franka_cloth_kitless[legacy-newton-ovrtx_renderer-static]",
   "source/isaaclab_tasks/test/core/test_rendering_franka_soft_kitless.py::test_rendering_franka_soft_kitless[legacy-newton-ovrtx_renderer-static]",
   "source/isaaclab_tasks/test/core/test_rendering_franka_cable_kitless.py::test_rendering_franka_cable_kitless[legacy-newton-ovrtx_renderer-static]",
+  "source/isaaclab_tasks/test/core/test_rendering_cartpole_kitless.py::test_rendering_cartpole_kitless[legacy-newton-ovrtx-async]",
+  "source/isaaclab_tasks/test/core/test_rendering_cartpole_kitless.py::test_rendering_cartpole_kitless[legacy-ovphysx-ovrtx-async]",
 ]
 
 rendering-correctness-kitless-ovstage = [

docs/source/concepts/ovphysx.rst

@@ -63,12 +63,43 @@ validated at the time of writing. The following pieces are available on
   `PR #5678 <https://github.com/isaac-sim/IsaacLab/pull/5678>`_.
 * :class:`~isaaclab.assets.DeformableObject` — experimental volume- and
   surface-deformable support on CUDA simulation devices.
+* Fast-path cloning of heterogeneous rigid-body and articulation geometry
+  variants with matching native tensor layouts.
 
 Additional OvPhysX work remains in flight. IMU, Frame Transformer, Joint Wrench,
 PVA, Ray Caster, and rendering support are not documented as supported here
 until their implementations land on ``develop`` and pass the backend smoke
 tests.
 
+Heterogeneous cloning
+---------------------
+
+OvPhysX 0.6.3 is required for heterogeneous runtime cloning. Tensor bindings use
+numeric environment order so indexed resets, actions and observations address
+the correct variant. Variants may differ in geometry but must preserve body,
+joint and tendon layout. Isaac Lab checks prototype rotation axes and tendon layouts;
+OVPhysX checks body and joint metadata when binding articulations.
+
+The clone context compiles the plan's prototypes and world assignments once.
+The physics manager attaches one exported stage, replays the native copies, and
+warms the runtime before assets and sensors bind. Hard reset reuses the same
+declarations; there is no separate consumable queue. Binding paths come from the
+plan's world layout and environment template, not from native clone order.
+
+Authored sources all receive native environment ID zero. Every world containing
+a source or USD-only physics therefore imports all its declared assets as originals. Other worlds
+clone their assets from a complete original world of the same composition,
+using one native environment ID per destination world. This keeps a robot,
+table and object able to contact one another without destination placeholders.
+Only declared asset subtrees are copied; unrelated authored assets are retained.
+
+Keep :attr:`~isaaclab.scene.InteractiveSceneCfg.filter_collisions` enabled to
+isolate original worlds with USD collision groups. GPU copies additionally use
+native environment-ID filtering. CPU has no native environment-ID filtering
+in OvPhysX 0.6.3, so it imports all declared USD copies and uses collision groups.
+Large overlapping original layouts can still exhaust broadphase pair capacity;
+use spatially separated environments.
+
 Deformable limitations
 ----------------------
 
@@ -80,7 +111,8 @@ padded state that would produce incorrect reductions.
 Deformable scenes also require full-stage materialization. Startup cost therefore
 grows with the number of authored environments. Use this path for small validation
 scenes; training-scale workloads with thousands of environments are not currently
-supported.
+supported. Deformable bodies are not supported by the heterogeneous fast-path
+cloner.
 
 Installation
 ------------
@@ -92,7 +124,9 @@ The ``ovphysx`` extra requires OvPhysX 0.6.3. Install it from the repository roo
     uv sync --inexact --extra ovphysx
 
 The ``--inexact`` flag preserves packages installed through other extras.
-Use ``--extra ov`` to install both public OvPhysX and OVRTX runtimes. The legacy
+Use ``--extra ov`` to install both public OvPhysX and OVRTX runtimes. The combined
+extra pairs OVRTX 0.5.0.377615 with OVStage 0.2; OVRTX 0.4.1 is not compatible
+with this runtime combination. The legacy
 Isaac Lab installer also supports ``./isaaclab.sh -i 'ov[ovphysx]'`` and
 ``./isaaclab.sh -i 'ov[all]'``.
 

docs/source/concepts/renderers.rst

@@ -438,8 +438,8 @@ rendering, then read each camera's output. An empty sequence performs no renderi
 
 :meth:`~isaaclab.renderers.BaseRenderer.render` continues to accept a single render-data object.
 The default ``render_batch()`` implementation calls ``render()`` for each entry, so existing
-custom renderers and single-camera callers need no changes. OVRTX overrides ``render_batch()``
-to submit the requested camera products in one native renderer step.
+custom renderers and single-camera callers need no changes. OVRTX steps all registered render
+products together but publishes only the requested cameras' observations.
 
 With eager sensor updates (``scene.cfg.lazy_sensor_update=False``), ``scene.update()`` advances
 sensor clocks in scene order and collects batch-capable sensors. After the loop, the camera
@@ -508,6 +508,55 @@ Or install the public ``ovrtx`` package directly from PyPI:
 
    The :class:`~isaaclab.renderers.BaseRenderer` class is under active development and may change without notice.
 
+.. _renderers-async-data-flow:
+
+Asynchronous Rendering
+----------------------
+
+:attr:`~isaaclab_ov.renderers.OVRTXRendererCfg.async_rendering` returns a camera's previous capture
+while rendering its next image. This allows rendering to overlap simulation and Python work.
+The first capture, including after reset, waits for a fresh image. The ovstage path stays synchronous.
+
+Latency is one capture, not necessarily one physics or control step. State observations remain
+current, so enabling this option changes the policy's observation timing; training may need to
+account for that delay. Physics and policy computation may also use the GPU.
+
+For a camera capturing once per environment step, steady-state operation is:
+
+.. code-block:: text
+
+   Synchronous:   physics k -> submit image k -> wait/read image k -> policy
+   Asynchronous:  physics k -> submit image k -> wait/read image k-1 -> policy
+                                   |                                 |
+                                   +---- image k renders ------------+--> physics k+1
+
+The first image is returned immediately after priming and repeated once on the next capture.
+
+Each camera owns its pending observations. Reading camera A never changes camera B's pixels or
+metadata, even when they share a native render submission. Reset discards that camera's pending
+observations. A partial environment reset re-primes the whole tiled camera product, without
+invalidating other cameras.
+
+Live fields such as ``camera.data.pos_w`` and ``camera.data.intrinsic_matrices`` remain current.
+Use the capture metadata when pairing delayed pixels with a pose or calibration, for example
+when deprojecting depth:
+
+.. code-block:: python
+
+   data = camera.data
+   depth = data.output["distance_to_image_plane"]
+   capture = data.info["distance_to_image_plane"]["capture"]
+   position, orientation = capture["pos_w"], capture["quat_w_world"]
+   intrinsics, frame = capture["intrinsic_matrices"], capture["frame"]
+
+Capture metadata remains unchanged on later captures. Direct renderer callers use
+:meth:`~isaaclab.renderers.BaseRenderer.prepare_capture` before submitting a capture and
+:meth:`~isaaclab.renderers.BaseRenderer.read_output` to publish it.
+
+SDP fills renderer-owned transform and geometry buffers, including deformable meshes, particles,
+and cables. Two buffers alternate per write and are reused only after OVRTX consumes them.
+Physics can therefore update its live arrays while the previous capture renders.
+
 See Also
 --------
 

docs/source/developer-tools/benchmarking/run_benchmarks.rst

@@ -66,6 +66,24 @@ From a source installation, run:
        --output_path ./benchmark_results \
        physics=isaacsim_physx
 
+OVRTX camera tasks can compare synchronous against asynchronous (pipelined) rendering through a
+Hydra override on the camera's renderer configuration:
+
+.. code-block:: bash
+
+   ./isaaclab.sh benchmark runtime \
+       --task Isaac-Cartpole-Camera-Direct \
+       --num_envs 256 \
+       --warmup_steps 30 \
+       --num_steps 200 \
+       --benchmark_formatter schema,omniperf \
+       --output_path ./results/ovrtx_async \
+       physics=newton_mjwarp renderer=ovrtx presets=rgb \
+       env.scene.tiled_camera.renderer_cfg.async_rendering=True
+
+Only the OVRTX renderer implements asynchronous rendering.
+See :ref:`renderers-async-data-flow` for how states, observations, and frames line up in each mode.
+
 Read the result
 ~~~~~~~~~~~~~~~
 

skills/user/select-backends/SKILL.md

@@ -60,6 +60,7 @@ Keep this skill synchronized with `docs/source/concepts/physics_backends.rst`, `
 - [Physics backends](../../../docs/source/concepts/physics_backends.rst)
 - [Backend architecture](../../../docs/source/concepts/backend_architecture.rst)
 - [Backends and presets](../../../docs/source/concepts/backends_and_presets.rst)
+- [Renderers, including asynchronous rendering](../../../docs/source/concepts/renderers.rst)
 - [Schema cfgs](../../../docs/source/concepts/schema_cfgs.rst)
 - [Installation](../../../docs/source/setup/installation/index.rst)
 - [Task examples](../../../source/isaaclab_tasks/isaaclab_tasks)

skills/user/use-sensors-actuators/SKILL.md

@@ -59,6 +59,8 @@ renderers, update the docs or source examples first.
 - [Evaluations](evaluations.md)
 - [Examples](examples.md)
 - [Add sensors on robot tutorial](../../../docs/source/how-to/add_sensors_on_robot.rst)
+- [Camera docs](../../../docs/source/concepts/sensors/camera.rst)
+- [Renderers, including asynchronous rendering](../../../docs/source/concepts/renderers.rst)
 - [Contact sensor docs](../../../docs/source/concepts/sensors/contact_sensor.rst)
 - [Joint wrench sensor docs](../../../docs/source/concepts/sensors/joint_wrench_sensor.rst)
 - [Actuators docs](../../../docs/source/concepts/actuators.rst)

source/isaaclab/changelog.d/ovrtx-async-capture-metadata.minor.rst

@@ -0,0 +1,5 @@
+Added
+^^^^^
+
+* Attached capture poses, calibration, and frame indices to delayed OVRTX images through
+  ``CameraData.info[output_name]["capture"]`` while preserving the live camera fields.

source/isaaclab/changelog.d/pv-ovrtx-async-frame-announcements.minor.rst

@@ -0,0 +1,7 @@
+Added
+^^^^^
+
+* Added :meth:`~isaaclab.renderers.BaseRenderer.prepare_capture` and
+  :meth:`~isaaclab.renderers.BaseRenderer.reset` hooks for delayed camera observations.
+  Asynchronous renderers can publish matching image metadata through ``CameraData.info`` and
+  discard pending observations on reset without changing live camera fields.

source/isaaclab/isaaclab/renderers/base_renderer.py

@@ -116,6 +116,19 @@ def set_outputs(self, render_data: Any, output_data: dict[str, ProxyArray]) -> N
         """
         pass
 
+    def prepare_capture(self, render_data: Any, camera_data: CameraData, frame: ProxyArray) -> None:
+        """Snapshot metadata for the next capture when its image will be delivered asynchronously.
+
+        Synchronous renderers need no snapshot. Delayed captures publish their pose, calibration,
+        and frame indices through ``camera_data.info[output_name]["capture"]`` without changing
+        the live camera fields.
+
+        Args:
+            render_data: Renderer-owned camera resources.
+            camera_data: Current camera pose and calibration.
+            frame: Current per-environment capture indices, shape (N,), dtype ``wp.int64``.
+        """
+
     @abstractmethod
     def update_transforms(self) -> None:
         """Update scene transforms before rendering.
@@ -176,15 +189,15 @@ def update_camera_intrinsics(self, render_data: Any, intrinsics: wp.array, param
 
     @abstractmethod
     def render(self, render_data: Any) -> None:
-        """Perform rendering and write to output buffers.
+        """Submit a capture; :meth:`read_output` publishes its camera's available observation.
 
         Args:
             render_data: The render data object from :meth:`create_render_data`.
         """
         pass
 
     def render_batch(self, render_data: Sequence[Any]) -> None:
-        """Render a collection of cameras into their bound output buffers.
+        """Submit captures for a collection of cameras.
 
         All camera poses and shared scene state must be prepared before calling this method.
         An empty sequence is a no-op. Each object must belong to this renderer and appear once.
@@ -201,6 +214,9 @@ def render_batch(self, render_data: Sequence[Any]) -> None:
     def read_output(self, render_data: Any, camera_data: CameraData) -> None:
         """Read rendered outputs from the renderer into the camera data container.
 
+        Asynchronous renderers may return the previous capture with its matching metadata in
+        ``camera_data.info``. Publishing one camera must not change another camera's observations.
+
         Args:
             render_data: The render data object from :meth:`create_render_data`.
             camera_data: The :class:`~isaaclab.sensors.camera.camera_data.CameraData`
@@ -217,6 +233,19 @@ def cleanup(self, render_data: Any) -> None:
         """
         pass
 
+    def reset(self, render_data: Any, env_ids: Sequence[int] | None = None) -> None:
+        """Reset the renderer-owned state of a camera when its environments reset.
+
+        A renderer implementation drops whatever per-camera state must not survive a reset.
+        Examples are pending asynchronous observations and accumulated temporal render history.
+        The default does nothing.
+
+        Args:
+            render_data: The render data object from :meth:`create_render_data`.
+            env_ids: Environments being reset, or ``None`` for all. An implementation may reset
+                more than the given environments when its state is not separable per environment.
+        """
+
     def close(self) -> None:
         """Release resources owned by the renderer itself rather than by a render data.
 

source/isaaclab/isaaclab/sensors/camera/camera.py

@@ -606,6 +606,11 @@ def set_world_poses_from_view(
     def reset(self, env_ids: Sequence[int] | slice | None = None, env_mask: wp.array | None = None):
         if not self._is_initialized:
             raise RuntimeError("Camera could not be initialized. Check the renderer and simulation logs for details.")
+        # The hook takes plain id sequences, so a slice resolves to its range. A mask-only call
+        # passes None, which means all environments: the hook permits resetting more than asked.
+        if isinstance(env_ids, slice):
+            env_ids = range(*env_ids.indices(self._num_envs))
+        self._renderer.reset(self._render_data, env_ids)
         # reset the timestamps
         super().reset(env_ids, env_mask)
         # reset the data
@@ -704,6 +709,7 @@ def _prepare_camera(self, env_mask: wp.array) -> None:
             self._update_poses(env_mask=env_mask, frame_op=1)
         else:
             self._update_camera_state(env_mask=env_mask, frame_op=1)
+        self._renderer.prepare_capture(self._render_data, self._data, self._frame)
 
     def _update_buffers_impl(self, env_mask: wp.array):
         if not self._env_mask_has_any(env_mask):

source/isaaclab/isaaclab/sensors/camera/camera_data.py

@@ -43,8 +43,10 @@ def __init__(self):
         """The retrieved sensor info with sensor types as key.
 
         This contains extra information provided by the sensor such as semantic segmentation label mapping, prim paths.
-        For semantic-based data, this corresponds to the ``"info"`` key in the output of the sensor. For other sensor
-        types, the info is empty.
+        For semantic-based data, this corresponds to the ``"info"`` key in the output of the sensor.
+        Delayed OVRTX images also include ``"capture"``: snapshots of ``pos_w``, ``quat_w_world``,
+        ``intrinsic_matrices``, and per-environment ``frame`` indices matching the returned pixels.
+        Live pose and calibration fields are unchanged. Other outputs may have empty info.
         """
 
     ##

source/isaaclab/test/renderers/test_simulation_render_context.py

@@ -576,9 +576,13 @@ def test_camera_batch_respects_period_partial_reset_and_updated_pose(camera_batc
     assert batches == [[("fast", 0.0), ("slow", 0.0)], [("fast", 0.0)]]
     np.testing.assert_array_equal(slow.frame.warp.numpy(), [1, 1])
     np.testing.assert_allclose(slow._timestamp_last_update.numpy(), [0.1, 0.1])
+    assert renderer.prepare_capture.call_args_list == [
+        call(camera._render_data, camera._data, camera._frame) for camera in (fast, slow, fast)
+    ]
 
     slow.pose = 3.0
     slow.reset(env_mask=wp.array([False, True], dtype=wp.bool, device="cpu"))
+    renderer.reset.assert_called_once_with(slow._render_data, None)
     InteractiveScene.update(scene, 0.1)
     assert batches[-1] == [("fast", 0.0), ("slow", 3.0)]
     assert slow.data.info["pose"] == 3.0

source/isaaclab_ov/changelog.d/maximiliank-heterogeneous-cloning-ovphysx.minor.rst

@@ -0,0 +1,16 @@
+Added
+^^^^^
+
+* Added plan-driven OvPhysX cloning for heterogeneous rigid-body and articulation geometry variants. GPU cloning imported complete original worlds and assigned one native environment ID per copied world; CPU imported full USD copies with collision grouping.
+
+Fixed
+^^^^^
+
+* Preserved numeric environment order in tensor bindings, including non-cloned multi-instance assets, so indexed state reads and writes address the intended instances.
+* Paired each cloned contact sensor with its own environment's filters and shared filters, and rejected missing filters that would silently remove sensor rows.
+* Preserved all active source variants and independently authored assets during physics export, including custom environment templates and scaled asset roots.
+* Preserved contacts with USD-only physics in each world composition and rejected incompatible active rotation-axis and tendon layouts before combining articulation variants.
+* Removed destination placeholders and empty exported environment roots that caused quadratic native binding lookup costs.
+* Registered Newton USD schemas before OVStage parsing. This affected all authored physics assets, including mimic-joint constraints, not only cloned assets.
+* Deferred scene-data tensor bindings until visualization requests them, avoiding unnecessary headless-training startup cost.
+* Matched the native scene's declared step rate to ``SimulationCfg.dt`` so automatic contact offsets used the actual simulation timestep instead of the 60 Hz default.

source/isaaclab_ov/changelog.d/ovrtx-async-operation-ownership.rst

@@ -0,0 +1,5 @@
+Fixed
+^^^^^
+
+* Unified retention of asynchronous OVRTX camera, transform, and geometry buffers until native writes completed,
+  isolated camera captures across resets, and avoided extracting priming images twice.

source/isaaclab_ov/changelog.d/pv-ovrtx-async-slot-buffers.minor.rst

@@ -0,0 +1,9 @@
+Added
+^^^^^
+
+* Added an opt-in asynchronous OVRTX render path controlled by
+  :attr:`~isaaclab_ov.renderers.OVRTXRendererCfg.async_rendering`.
+  ``True`` returns each camera's previous capture while rendering the next image. The first
+  capture and the first capture after reset wait for a fresh image. Matching capture poses,
+  calibration, and frame indices are available in ``camera.data.info[output_name]["capture"]``;
+  the live camera fields remain current.

source/isaaclab_ov/isaaclab_ov/_clone.py

@@ -5,9 +5,25 @@
 
 """Internal clone payload definitions shared by the OvPhysX cloner and manager."""
 
+import re
 from collections.abc import Sequence
+from fnmatch import fnmatchcase
 
 CloneTransform = tuple[float, float, float, float, float, float, float]
+# Source, destinations, destination poses, destination world IDs, source world ID.
+CloneRecipe = tuple[str, list[str], list[CloneTransform], list[int] | None, int | None]
+
+
+def ordered_clone_paths(paths: list[str], patterns: list[str]) -> list[str]:
+    """Order binding rows numerically within each requested body pattern."""
+    if paths == patterns:
+        return paths
+
+    def order_key(path: str) -> tuple:
+        group = next((i for i, pattern in enumerate(patterns) if fnmatchcase(path, pattern)), len(patterns))
+        return group, tuple(int(part) if part.isdigit() else part for part in re.split(r"(\d+)", path))
+
+    return sorted(paths, key=order_key)
 
 
 def clone_transforms_from_positions(positions: Sequence[Sequence[float]]) -> list[CloneTransform]: