[Backport release/3.0.0] [5D] Bind Newton consumers to shared native resources

docs/source/api/lab_newton/isaaclab_newton.physics.rst

@@ -10,6 +10,7 @@
     NewtonManager
     NewtonCfg
     NewtonBackendCfg
+    NewtonBuilderCfg
     NewtonSoftContactCfg
     NewtonCollisionPipelineCfg
     NewtonFeatherstoneManager
@@ -59,6 +60,13 @@ Physics Configuration
   :show-inheritance:
   :exclude-members: __init__
 
+.. autoclass:: NewtonBuilderCfg
+  :members:
+  :show-inheritance:
+  :exclude-members: __init__
+
+.. autofunction:: create_newton_builder
+
 .. autoclass:: NewtonSoftContactCfg
   :members:
   :show-inheritance:

docs/source/concepts/backend_architecture.rst

@@ -70,21 +70,24 @@ and release resources with ``close()``.
 
 The manager exposes :class:`~isaaclab.physics.PhysicsEvent` callbacks for
 cross-backend lifecycle work. ``MODEL_INIT`` occurs during scene construction,
-``PHYSICS_READY`` after physics initialization, and ``STOP`` during shutdown.
+``PHYSICS_READY`` after physics initialization, and ``STOP`` before native resources are replaced
+or shut down.
 The concrete ``close()`` implementation dispatches the ``STOP`` event.
 
 ``SimulationContext`` owns native resources and renderer instances in one registry.
 ``get_or_create_backend(backend_cfg)``
 reuses one resource for equal configurations of the same concrete type; a cache miss
 constructs ``instantiate(backend_cfg)``.
-:class:`~isaaclab.sim.BackendCfg` describes resource construction inputs, and
+:class:`~isaaclab.sim.BackendCfg` describes resource settings and identity, and
 :class:`~isaaclab.renderers.RendererCfg` extends it for renderer instances.
 ``PhysicsCfg`` selects a physics manager. Finalize configurations before
 registration and treat them, including nested values, as read-only afterward.
 Use a new configuration for different settings. ``close_backend(backend)`` closes
 the exact registered object after all consumers have released their bindings;
-it does not compare or hash configurations. Resources implement ``close()``;
-failed release retains the entry for retry. After physics shutdown invalidates camera
+it does not compare or hash configurations. Resources declared through ``BackendCfg`` must
+implement ``close()``. Plain construction cfgs share Python-owned data without a teardown
+operation; removing the registry entry releases its reference. A failed close retains
+the entry for retry. After physics shutdown invalidates camera
 render data, simulation teardown closes material writers, renderer instances, visualizers,
 and remaining native resources, in that order, before closing the stage.
 
@@ -96,6 +99,29 @@ Exposing native handles does not replace SDP transport.
 Clone contexts are registered separately as ``sim.clone_contexts[Context] = Context(...)``
 before plan dispatch. They apply the plan but do not own native runtime resources.
 
+Newton has two resources with different lifetimes, not two interchangeable backends:
+
+* ``ModelBuilder`` holds mutable construction data. Cloning populates it and sensors declare
+  requirements before finalization. It remains available for hard reset. ``NewtonBuilderCfg``
+  is a plain construction cfg, not a ``BackendCfg``; the builder needs no native ``close()``.
+* ``NewtonBackend`` owns the finalized model and native buffers. Physics and render consumers
+  borrow those handles. Closing it releases runtime allocations without closing the builder.
+
+Both resources use the same registry:
+
+.. code-block:: python
+
+    builder_cfg = NewtonBuilderCfg(physics_cfg=sim.cfg.physics)
+    builder = sim.get_or_create_backend(builder_cfg)
+    # Clone/import populates this builder before model allocation.
+    model_cfg = NewtonBackendCfg(physics_cfg=sim.cfg.physics, device=sim.device)
+    backend = sim.get_or_create_backend(model_cfg)
+
+Both configurations use the selected physics cfg; non-Newton physics selects a render-only
+representation. ``SimulationContext`` has no backend-specific cfg fields, and consumers do not
+access clone contexts. Consumers request body transforms and visual points directly through SDP.
+Queries share the native resource's BVHs but keep each consumer's captured work separate.
+
 Portable asset and sensor interfaces
 ------------------------------------
 

docs/source/developer-tools/extending_newton_solvers.rst

@@ -91,23 +91,21 @@ class invokes them.
      - Stores the simulation context, reads gravity from the simulation
        configuration, and creates the scene data backend.
      - none
-   * - :meth:`~isaaclab_newton.physics.NewtonManager.create_builder` and
-       :meth:`~isaaclab_newton.physics.NewtonManager.set_builder`
-     - The cloner imports declared prototypes, composes worlds, and supplies the completed builder.
-     - ``_register_builder_attributes()`` (not called by ``set_builder()``)
+   * - ``sim.get_or_create_backend(NewtonBuilderCfg(physics_cfg=sim.cfg.physics))``
+     - Acquires the shared builder through the active manager's ``create_builder()`` factory.
+       The cloner imports declared prototypes and composes worlds into it.
+     - ``_register_builder_attributes()``
    * - :meth:`~isaaclab_newton.physics.NewtonManager.start_simulation`
      - Finalizes the model, then allocates states and reset masks.
-     - ``_register_builder_attributes()``,
-       ``_prepare_builder_for_finalize()``
+     - ``_prepare_builder_for_finalize()``
    * - :meth:`~isaaclab_newton.physics.NewtonManager.initialize_solver`
      - Builds the solver, checks that it was assigned, and allocates contacts.
      - ``_build_solver()``, ``_initialize_contacts()``
    * - :meth:`~isaaclab_newton.physics.NewtonManager.reset`
      - A hard reset re-runs ``start_simulation()`` and ``initialize_solver()``
        against a re-finalized model; a soft reset reuses the existing model,
        solver, contacts, and graph.
-     - ``_register_builder_attributes()``,
-       ``_prepare_builder_for_finalize()``, ``_build_solver()``,
+     - ``_prepare_builder_for_finalize()``, ``_build_solver()``,
        ``_initialize_contacts()`` (hard reset only)
    * - :meth:`~isaaclab_newton.physics.NewtonManager.step`
      - Runs one actuator pass plus ``num_substeps`` solver substeps, then

docs/source/developer-tools/scene_data_providers.rst

@@ -141,9 +141,11 @@ path.
 PhysX owns its native Fabric refresh and publishes the resulting matrices through SDP without
 fetching packed poses. ``isaaclab_physx.renderers.fabric.FabricBackend`` owns the shared native stage
 and hierarchy handles. Its identity is the stage and device, not the SDP source or attribute type.
-``SimulationContext`` declares ``fabric_cfg`` when Kit is available. After physics initializes, Kit,
-Isaac RTX, and explicit Fabric synchronization obtain the same resource through
-``get_or_create_backend(sim.fabric_cfg)``. Transform bindings are state on that resource, not a
+After physics initializes, Kit, Isaac RTX, and explicit Fabric synchronization obtain
+the same resource through
+``sim.get_or_create_backend(FabricBackendCfg(stage=sim.stage, device=sim.device))``.
+The simulation registry does not need to know which backends exist.
+Transform bindings are state on that resource, not a
 separate backend. Consumers pass the simulation's SDP to ``update_transforms(provider)``; for foreign
 physics it converts directly into Fabric local matrices, then propagates the GPU hierarchy.
 Core ``RenderContext`` owns no Fabric bindings.

docs/source/migration/migrating_to_isaaclab_3-0.rst

@@ -1613,9 +1613,10 @@ For configuration and data access examples, see the :ref:`overview_sensors_joint
    import torch
    from isaaclab.scene import InteractiveSceneCfg
    from isaaclab.sensors import JointWrenchSensorCfg
+   from isaaclab.utils import replace
 
    class MySceneCfg(InteractiveSceneCfg):
-       robot = ROBOT_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
+       robot = replace(ROBOT_CFG, prim_path="{ENV_REGEX_NS}/Robot")
        joint_wrench = JointWrenchSensorCfg(prim_path="{ENV_REGEX_NS}/Robot")
 
    sensor = env.scene.sensors["joint_wrench"]
@@ -1643,9 +1644,10 @@ depend on the joint-wrench sensor instead:
    from isaaclab.managers import ObservationTermCfg as ObsTerm
    from isaaclab.scene import InteractiveSceneCfg
    from isaaclab.sensors import JointWrenchSensorCfg
+   from isaaclab.utils import replace
 
    class MySceneCfg(InteractiveSceneCfg):
-       robot = ROBOT_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot")
+       robot = replace(ROBOT_CFG, prim_path="{ENV_REGEX_NS}/Robot")
        joint_wrench = JointWrenchSensorCfg(prim_path="{ENV_REGEX_NS}/Robot")
 
    feet_body_forces = ObsTerm(
@@ -2640,8 +2642,9 @@ tracking is handled directly by :class:`~isaaclab_visualizers.kit.KitVisualizer`
 The ``tiled_cam_*`` configuration fields on visualizer configs (e.g. ``tiled_cam_view``,
 ``tiled_cam_num``, ``tiled_cam_prim_path``) have been removed and replaced by the unified
 ``streaming_*`` API available on all four visualizer backends.  A one-release deprecation
-shim forwards each removed field to its ``streaming_*`` equivalent and emits
+shim forwards the old fields except renderer selection to their ``streaming_*`` equivalents and emits
 :class:`DeprecationWarning`; the shim will be removed in the next major release.
+Renderer selection now takes a configuration directly, without a nickname compatibility path.
 
 .. list-table:: Field rename reference
    :header-rows: 1
@@ -2663,8 +2666,15 @@ shim forwards each removed field to its ``streaming_*`` equivalent and emits
      - ``streaming_cam_eye``
      -
    * - ``tiled_cam_renderer``
-     - ``streaming_cam_renderer``
-     - Accepts ``"newton_warp"``, ``"ovrtx"``, ``"isaac_rtx"``, or ``None``
+     - ``streaming_cam_renderer_cfg``
+     - Renderer configuration, not a nickname
+
+Replace ``streaming_cam_renderer="ovrtx"`` with ``streaming_cam_renderer_cfg=OVRTXRendererCfg()``
+(imported from ``isaaclab_ov.renderers``). Likewise, pass ``NewtonWarpRendererCfg()`` or
+``IsaacRtxRendererCfg()`` for Newton Warp or Isaac RTX. Custom configurations use their existing
+``class_type`` class or resolvable string. Kit defaults to Isaac RTX; the other visualizers default
+to Newton Warp. An unavailable explicitly selected renderer raises its construction error instead
+of silently selecting another renderer.
 
 .. code-block:: python
 

examples/demos/newton_viewer_block_and_tackle.py

@@ -24,7 +24,7 @@
 import newton
 import newton.utils
 import warp as wp
-from isaaclab_newton.physics import NewtonCfg, NewtonManager, NewtonShapeCfg, VBDSolverCfg
+from isaaclab_newton.physics import NewtonBuilderCfg, NewtonCfg, NewtonManager, NewtonShapeCfg, VBDSolverCfg
 
 import isaaclab.sim as sim_utils
 from isaaclab.utils import configclass
@@ -315,11 +315,10 @@ def main() -> None:
         sim_cfg = sim_utils.SimulationCfg(dt=1.0 / 60.0, device=args_cli.device, physics=resolved_physics_cfg)
         sim = sim_utils.SimulationContext(sim_cfg)
         sim.set_camera_view(eye=(1.35, -2.1, 1.25), target=(0.50, 0.0, 0.72))
-        builder = NewtonManager.create_builder()
+        builder = sim.get_or_create_backend(NewtonBuilderCfg(physics_cfg=sim.cfg.physics))
         builder.rigid_gap = CABLE_GAP
         cable_body_start, wrapped_xforms = _build_system(builder)
         builder.color(balance_colors=False)
-        NewtonManager.set_builder(builder)
         sim.reset()
         _initialize_wrapped_cable(cable_body_start, wrapped_xforms)
         print("[INFO]: Setup complete. Right-drag the yellow cable handle downward to lift the red load.", flush=True)

source/isaaclab/changelog.d/newton-consumer-ownership.major.rst

@@ -0,0 +1,25 @@
+Changed
+^^^^^^^
+
+* Extended ``SimulationContext.get_or_create_backend(cfg)`` to own Python resources as well as native
+  backends. Resources declared through ``BackendCfg`` implemented ``close()``; plain construction cfgs
+  shared Python-owned data released with the registry reference.
+  Resource construction used ``instantiate(cfg)`` without backend-specific context fields.
+* Reused SDP transform mappings for matching source and destination layouts, allowing independent
+  consumers to share converted buffers without retaining SDP bindings on a native backend. Matching
+  unique native and consumer paths needed no index-map allocation. Binding rejected duplicate named
+  paths and destinations without native publications, including when the publication was empty.
+* Added ``RayCasterCfg.use_cuda_graph`` for Newton query execution independently of physics capture.
+  Set it to ``False`` to execute Newton ray casts eagerly.
+* **Breaking:** Replaced ``VisualizerCfg.streaming_cam_renderer`` and its deprecated
+  ``tiled_cam_renderer`` alias with ``streaming_cam_renderer_cfg``. Pass a renderer configuration,
+  such as ``NewtonWarpRendererCfg()`` or ``OVRTXRendererCfg()``, instead of a renderer nickname.
+  Its ``class_type`` accepted either an implementation class or a resolvable string.
+
+Fixed
+^^^^^
+
+* Released rendering bindings on physics stop so a hard reset reinitialized consumers against
+  the replacement native resource.
+* Preserved each visualizer's default streaming renderer when applying camera settings from
+  another visualizer configuration.

source/isaaclab/isaaclab/renderers/render_context.py

@@ -10,7 +10,7 @@
 import logging
 import warnings
 from collections.abc import Sequence
-from typing import TYPE_CHECKING, Any
+from typing import Any
 
 import torch
 import warp as wp
@@ -19,9 +19,6 @@
 from .base_renderer import BaseRenderer, VisualMaterialBatch
 from .renderer_cfg import RendererCfg
 
-if TYPE_CHECKING:
-    from ..sim import BackendCfg
-
 logger = logging.getLogger(__name__)
 
 
@@ -81,7 +78,7 @@ class RenderContext:
         "_consumers_finalized",
     )
 
-    def __init__(self, backend_registry: list[tuple[BackendCfg, Any]]) -> None:
+    def __init__(self, backend_registry: list[tuple[Any, Any]]) -> None:
         self.clone_contexts: set[type | str] = set()
         """Scene representations declared by camera renderers and visualizers before cloning."""
         self._backend_registry = backend_registry

source/isaaclab/isaaclab/scene_data/scene_data_backend.py

@@ -178,5 +178,5 @@ def transform_count(self) -> int:
 
     @property
     def transform_paths(self) -> list[str]:
-        """Return the paths for each transform."""
+        """Return unique native body paths in publication order, fixed for each initialized layout."""
         raise NotImplementedError

source/isaaclab/isaaclab/scene_data/scene_data_provider.py

@@ -66,6 +66,7 @@ def __init__(self, backend: SceneDataBackend):
         self._num_envs_cache: int | None = None
         self._interactive_scene: Any | None = None
         self._cache: dict[Any, TimestampedBuffer] = {}
+        self._transform_mappings: dict[tuple, wp.array | None] = {}
 
     def get_transforms(
         self,
@@ -276,26 +277,36 @@ def create_mapping(self, paths: list[str | None]) -> wp.array(dtype=wp.int32) |
         appear in ``paths`` (or maps to ``None``) receive an index of ``-1`` and are
         skipped during conversion.
 
+        Named paths must be unique in both layouts. Binding validates this once per layout pair;
+        independent consumers reuse the mapping and share SDP conversions.
+
         Args:
             paths: Desired output ordering expressed as prim paths. Use ``None`` for
                 slots that should not receive any transform.
 
         Returns:
             A Warp int32 array of length :attr:`transform_count` containing the
-            remapped indices, or ``None`` if the sim backend provides no transform
-            paths or if no mapping is needed.
+            remapped indices, or ``None`` for identical layouts.
+
+        Raises:
+            ValueError: A source or named destination path is repeated.
+            KeyError: A named destination has no native publication.
         """
-        if input_paths := self.backend.transform_paths:
-            # The map keeps resolution linear in the number of paths. For duplicate
-            # paths the first occurrence wins, matching ``list.index``.
-            path_to_out: dict[str | None, int] = {}
-            for out_idx, out_path in enumerate(paths):
-                if out_path not in path_to_out:
-                    path_to_out[out_path] = out_idx
-            mapping = [path_to_out.get(path, -1) for path in input_paths]
-            if len(paths) != len(input_paths) or not np.array_equal(mapping, np.arange(len(input_paths))):
-                return wp.array(mapping, dtype=wp.int32, device=_publication_device(self.backend.transforms))
-        return None
+        input_paths, paths = tuple(self.backend.transform_paths), tuple(paths)
+        key = input_paths, paths
+        if key not in self._transform_mappings:
+            sources = {path: index for index, path in enumerate(input_paths)}
+            destinations = {path: index for index, path in enumerate(paths) if path is not None}
+            if len(sources) != len(input_paths) or len(destinations) != len(paths) - paths.count(None):
+                raise ValueError("Transform paths must be unique; only destination None slots may repeat.")
+            result = None
+            if input_paths != paths:
+                mapping = np.full(len(input_paths), -1, dtype=np.int32)
+                for path, index in destinations.items():
+                    mapping[sources[path]] = index
+                result = wp.array(mapping, dtype=wp.int32, device=_publication_device(self.backend.transforms))
+            self._transform_mappings[key] = result
+        return self._transform_mappings[key]
 
     def get_geometry_points(
         self, *, output: wp.array | SceneDataFormat.FabricPoints | None = None, offsets: dict[str, int] | None = None

source/isaaclab/isaaclab/sensors/ray_caster/ray_caster_cfg.py

@@ -91,6 +91,12 @@ class OffsetCfg:
     rays outside the carrier's geometry in that case.
     """
 
+    use_cuda_graph: bool = True
+    """Capture Newton BVH queries independently of the physics solver's graph setting.
+
+    CPU queries and the static Warp-mesh implementations run eagerly.
+    """
+
     drift_range: tuple[float, float] = (0.0, 0.0)
     """The range of drift (in meters) to add to the ray starting positions (xyz) in world frame. Defaults to (0.0, 0.0).
 

source/isaaclab/isaaclab/sim/simulation_cfg.py

@@ -11,8 +11,9 @@
 
 from __future__ import annotations
 
+from collections.abc import Callable
 from dataclasses import MISSING
-from typing import Literal
+from typing import Any, Literal
 
 from ..physics import PhysicsCfg
 from ..utils import configclass
@@ -22,13 +23,13 @@
 
 @configclass
 class BackendCfg:
-    """Construction inputs and value identity for a simulation-owned resource.
+    """Declarative settings and value identity for a simulation-owned resource.
 
     Finalize all fields before registration and treat them as read-only afterward.
     """
 
-    class_type: type = MISSING
-    """Resource class constructed as ``instantiate(cfg)``; must implement ``close()``."""
+    class_type: Callable[[BackendCfg], Any] = MISSING
+    """Constructor called through ``instantiate(cfg)``; the returned resource must implement ``close()``."""
 
 
 @configclass