Move PPISP processing from renderers into camera pipelines

docs/source/api/lab/isaaclab.sensors.camera.rst

@@ -13,7 +13,12 @@ The following classes are part of the public :mod:`isaaclab.sensors.camera` API.
 .. autosummary::
    :nosignatures:
 
+   CameraExposureCfg
    CameraISPMode
 
 .. autoclass:: CameraISPMode
    :show-inheritance:
+
+.. autoclass:: CameraExposureCfg
+   :members:
+   :exclude-members: __init__

docs/source/api/lab_ppisp/isaaclab_ppisp.rst

@@ -14,10 +14,33 @@ The following classes are part of the public :mod:`isaaclab_ppisp` API.
    :nosignatures:
 
    PpispCfg
+   PpispProcessorCfg
+   PpispProcessor
+   PpispControllerWeights
+   PpispDiscoveryMode
    PpispPipeline
 
 .. autoclass:: PpispCfg
    :show-inheritance:
 
+.. autoclass:: PpispProcessorCfg
+   :members:
+   :inherited-members:
+   :exclude-members: __init__
+   :show-inheritance:
+
+.. autoclass:: PpispProcessor
+   :members:
+   :inherited-members:
+   :special-members: __call__
+   :show-inheritance:
+
+.. autoclass:: PpispControllerWeights
+   :members:
+
+.. autoclass:: PpispDiscoveryMode
+   :members:
+   :show-inheritance:
+
 .. autoclass:: PpispPipeline
    :show-inheritance:

docs/source/concepts/sensors/camera.rst

@@ -153,8 +153,9 @@ updating it adds frame-query overhead.
 Output types
 ------------
 
-The active renderer validates ``data_types`` and allocates the channel count and data type declared
-by its :class:`~isaaclab.renderers.RenderBufferSpec`.
+For raw outputs, the active renderer validates the requested names and allocates the channel count
+and data type declared by its :class:`~isaaclab.renderers.RenderBufferSpec`. A camera pipeline may
+produce additional output names; its processors allocate those buffers.
 
 .. list-table:: Common output contracts
    :header-rows: 1
@@ -168,7 +169,7 @@ by its :class:`~isaaclab.renderers.RenderBufferSpec`.
      - Low-dynamic-range color
    * - ``rgb_hdr``
      - 3, ``float32``
-     - Scene-linear high-dynamic-range color
+     - High-dynamic-range RGB using the active camera settings
    * - ``albedo``
      - 4, ``uint8``
      - Material base color
@@ -198,6 +199,12 @@ by its :class:`~isaaclab.renderers.RenderBufferSpec`.
 non-colorized segmentation uses one ``int32`` ID channel. Label and prim-path mappings are stored in
 ``camera_data.info[output_name]``.
 
+``rgb_hdr`` uses the camera's active exposure settings. PPISP requires scene-linear HDR;
+set ``CameraCfg.exposure=CameraExposureCfg()`` explicitly to disable RTX automatic exposure and
+use neutral exposure. Leaving ``exposure=None`` preserves authored camera settings. Selecting an
+output never changes exposure. PPISP requests ``rgb_hdr`` privately; include it in ``data_types``
+only to publish the raw input as well.
+
 .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/camera-renderer-isaac-rtx.webp
    :align: center
    :figwidth: 100%
@@ -263,6 +270,12 @@ current support matrix is:
      - Yes
      - No
 
+.. note::
+
+   Explicit exposure settings affect every output from the camera prim, including other sensors
+   sharing it. Use separate camera prims when different exposures are required. Newton's HDR
+   output is already scene-linear and does not consume RTX exposure attributes.
+
 Querying an unsupported output fails during camera initialization. Renderer configuration controls
 semantic filters, segmentation colorization, and depth clipping where those options are
 backend-specific.
@@ -305,47 +318,72 @@ background. Set a normalized RGB tuple to use a solid color for pixels that miss
 The setting is per camera. Cameras with renderer-default and solid backgrounds can coexist in one
 scene.
 
-Post-render image signal processing
------------------------------------
+Post-render image processing
+----------------------------
 
-:attr:`~sensors.CameraCfg.isp_cfg` optionally applies an image signal processing (ISP) pass to the
-renderer's scene-linear HDR output. The shipped implementation is PPISP (Physically Plausible Image
-Signal Processing), which applies responsivity, exposure, vignetting, color correction, and a camera
-response function before writing ``rgb`` or ``rgba``.
+:attr:`~sensors.CameraCfg.pipeline` applies
+:class:`~isaaclab.utils.tensor_processors.DictTensorProcessor` instances after rendering.
+Each processor consumes named ``ProxyArray`` tensors and returns persistent named outputs;
+its :class:`~isaaclab.utils.tensor_processors.DictTensorProcessorCfg` declares those keys.
+The camera requests required renderer inputs privately and publishes the names in ``data_types``.
 
-The field accepts:
-
-* ``None`` to disable post-render ISP.
-* :class:`~isaaclab_ppisp.PpispCfg` for explicit coefficients or coefficients imported from a USD
-  camera.
-* :class:`~sensors.CameraISPMode` to discover ``ppisp:*`` attributes on a camera prim.
+For example, PPISP (Physically Plausible Image Signal Processing) converts scene-linear HDR
+to uint8 color through exposure, vignetting, color correction, and a camera response function:
 
 .. code-block:: python
 
-   from isaaclab.sensors.camera import CameraCfg, CameraISPMode
+   from isaaclab.sensors.camera import CameraExposureCfg
    from isaaclab.utils import replace
-   from isaaclab_ppisp import PpispCfg
+   from isaaclab_ppisp import PpispCfg, PpispProcessorCfg
+
+   processed_camera = replace(
+       front_camera,
+       data_types=["rgb"],
+       exposure=CameraExposureCfg(),
+       pipeline=[PpispProcessorCfg(params={"cfg": PpispCfg(inputs={"exposureOffset": 1.5})})],
+   )
 
-   explicit_isp = replace(front_camera, data_types=["rgb"], isp_cfg=PpispCfg(inputs={"exposureOffset": 1.5}))
+PPISP requires float32 scene-linear input with shape ``(N, H, W, 3)``. Configure neutral camera
+exposure before simulation initialization, as above; PPISP's ``exposureOffset`` is measured in stops.
+Include ``rgb_hdr`` in ``data_types`` to publish its raw input alongside the processed output.
+For RGBA, use ``outputs=["rgba"]`` and ``params={"cfg": ppisp_cfg, "channels": 4}``.
+A processed RGBA buffer can also provide a zero-copy RGB view; RGB alone cannot satisfy an RGBA request.
 
-   discovered_isp = replace(front_camera, data_types=["rgb"], isp_cfg=CameraISPMode.AUTO_CAMERA)
+Published outputs must retain the camera's batch size, height, width, and NHWC layout so calibration
+remains valid. Intermediate buffers may change shape. Processors borrow their inputs read-only,
+and each camera owns its pipeline workspace. Outputs reuse storage across captures; copy values
+that must survive another update. Explicitly shared immutable controller weights are supported
+through :class:`~isaaclab_ppisp.PpispControllerWeights`.
 
-``AUTO_CAMERA`` checks the first matched camera prim. ``AUTO_ANY`` falls back to the first PPISP
-camera anywhere on the stage. Discovery happens once during camera construction.
+The pipeline runs once per new capture, including batched captures. If an asynchronous renderer
+republishes an old capture, its processed result and capture metadata are retained. Camera reset
+forwards selected environment indices to processor reset callbacks; teardown closes processors.
+Existing image observation terms, such as ``mdp.image_rgb``, read these processed camera outputs.
 
-PPISP is composed by Isaac RTX, OVRTX, and Newton Warp. It requires ``rgb`` or ``rgba`` output. A
-static configuration is shared by all cloned views in one camera batch; controller weights may
-predict per-view exposure and color parameters, while the remaining coefficients stay shared. ISP
-configuration and discovered USD attributes are fixed for the camera lifetime.
+Migrating renderer-owned ISP configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. important::
+``CameraCfg.isp_cfg`` now raises a migration error for non-None values. Replace it with
+``CameraCfg.pipeline`` and explicit neutral exposure as above. ``CameraISPMode`` remains
+importable for legacy configuration values; new code resolves USD attributes before constructing
+the processor:
 
-   With Isaac RTX and OVRTX, enabling ``isp_cfg`` makes PPISP the ISP authority. The renderer
-   disables RTX auto-exposure, authors neutral ``exposure:*`` values, and applies the
-   ``OmniRtxCameraAutoExposureAPI_1`` and ``OmniRtxCameraExposureAPI_1`` schemas on every matched
-   camera prim so RTX does not process the image a second time. Do not combine ``isp_cfg`` with
-   separately authored RTX exposure or tonemapping settings. When ``isp_cfg`` is ``None``, the
-   renderer leaves authored camera exposure unchanged.
+.. code-block:: python
+
+   from isaaclab_ppisp import PpispDiscoveryMode, resolve_and_normalize
+
+   ppisp_cfg = resolve_and_normalize(PpispDiscoveryMode.AUTO_CAMERA, stage, "/World/Camera")
+   discovered_camera = replace(
+       front_camera,
+       data_types=["rgb"],
+       exposure=CameraExposureCfg(),
+       pipeline=[PpispProcessorCfg(params={"cfg": ppisp_cfg})] if ppisp_cfg is not None else [],
+   )
+
+``AUTO_CAMERA`` checks the supplied camera; ``AUTO_ANY`` can fall back to another PPISP camera
+on the stage. Alternatively, use ``ppisp_cfg_from_usd_camera(camera_prim)``.
+The processor performs no camera or stage discovery. Static coefficients apply to the whole batch;
+controller weights may predict per-view exposure and color parameters.
 
 Run the ``ppisp-camera`` example for a complete PPISP workflow. It supports
 ``--renderer newton_renderer`` (Newton Warp, the default) and ``--renderer isaac_rtx`` (Isaac RTX).
@@ -416,3 +454,8 @@ A runnable camera example is available as ``camera``:
 
 For saving output to disk, see :doc:`/source/how-to/save_camera_output`. For renderer selection
 and customization, see :doc:`/source/how-to/configure_rendering`.
+
+The observation processor API is ``isaaclab.utils.tensor_processors``. Custom stateful processors
+inherit ``TensorProcessor`` and use ``TensorProcessorCfg``; attach configurations through
+``ObservationTermCfg.tensor_processors``. These replace ``ModifierBase``, ``ModifierCfg``, and
+``ObservationTermCfg.modifiers`` respectively.

examples/sensors/ppisp_camera.py

@@ -120,12 +120,13 @@
 import matplotlib.pyplot as plt
 import numpy as np
 import torch
-from isaaclab_ppisp.cfg import PpispCfg, ppisp_cfg_from_usd_camera
+from isaaclab_ppisp import PpispCfg, PpispProcessorCfg, ppisp_cfg_from_usd_camera
 
 import isaaclab.sim as sim_utils
 from isaaclab.assets import AssetBaseCfg, RigidObjectCfg
 from isaaclab.scene import InteractiveSceneCfg
 from isaaclab.sensors import CameraCfg
+from isaaclab.sensors.camera import CameraExposureCfg
 from isaaclab.sim.spawners.materials import UsdPhysicsRigidBodyMaterialCfg
 from isaaclab.utils import configclass, instantiate
 
@@ -374,7 +375,8 @@ def make_camera(camera_prim_path: str, *, ppisp_cfg: PpispCfg | None, width: int
         width=width,
         data_types=["rgb"],
         spawn=None,
-        isp_cfg=ppisp_cfg,
+        exposure=CameraExposureCfg() if ppisp_cfg is not None else None,
+        pipeline=[PpispProcessorCfg(params={"cfg": ppisp_cfg})] if ppisp_cfg is not None else [],
         renderer_cfg=make_renderer_cfg(),
     )
     return instantiate(camera_cfg)

source/isaaclab/changelog.d/feat-render-buffer-pipe.changed.rst

@@ -0,0 +1,3 @@
+* **Breaking:** Replaced renderer-owned PPISP with camera processors consuming ``rgb_hdr``.
+  Replace ``CameraCfg.isp_cfg`` with ``PpispProcessorCfg`` in ``CameraCfg.pipeline``.
+  Set ``CameraCfg.exposure=CameraExposureCfg()`` explicitly for neutral RTX HDR before PPISP.

source/isaaclab/changelog.d/feat-render-buffer-pipe.fixed.rst

@@ -0,0 +1,2 @@
+* Avoided unnecessary CUDA synchronization during masked camera resets when image pipeline
+  operations had no reset callbacks.

source/isaaclab/isaaclab/renderers/output_contract.py

@@ -26,6 +26,7 @@ class RenderBufferKind(StrEnum):
     RGB = "rgb"
     RGBA = "rgba"
     RGB_HDR = "rgb_hdr"
+    """HDR RGB using the camera's active rendering settings."""
     ALBEDO = "albedo"
     DEPTH = "depth"
     DISTANCE_TO_IMAGE_PLANE = "distance_to_image_plane"

source/isaaclab/isaaclab/renderers/render_context.py

@@ -9,8 +9,9 @@
 
 import logging
 import warnings
+import weakref
 from collections.abc import Sequence
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
 import torch
 import warp as wp
@@ -19,6 +20,9 @@
 from .base_renderer import BaseRenderer, VisualMaterialBatch
 from .renderer_cfg import RendererCfg
 
+if TYPE_CHECKING:
+    from ..sensors.camera.camera import Camera
+
 logger = logging.getLogger(__name__)
 
 
@@ -65,6 +69,7 @@ class RenderContext:
     __slots__ = (
         "clone_contexts",
         "_backend_registry",
+        "_cameras",
         "_physics_initialized",
         "_prepared_renderer_ids",
         "_prepared_num_envs",
@@ -82,6 +87,7 @@ def __init__(self, backend_registry: list[tuple[Any, Any]]) -> None:
         self.clone_contexts: set[type | str] = set()
         """Scene representations declared by camera renderers and visualizers before cloning."""
         self._backend_registry = backend_registry
+        self._cameras: weakref.WeakValueDictionary[int, Camera] = weakref.WeakValueDictionary()
         self._physics_initialized: bool = False  # Set to True after the first PHYSICS_READY callback fires.
         self._prepared_renderer_ids: set[int] = set()
         self._prepared_num_envs: int | None = None
@@ -133,6 +139,19 @@ def ensure_initialize(self) -> None:
         for _cfg, renderer in self._renderer_entries:
             renderer.initialize()
 
+    def register_camera(self, camera: Camera) -> None:
+        """Include a camera in stage preparation without extending its lifetime."""
+        self._cameras[id(camera)] = camera
+
+    def unregister_camera(self, camera: Camera) -> None:
+        """Stop preparing a camera whose sensor callbacks have been cleared."""
+        self._cameras.pop(id(camera), None)
+
+    def prepare_cameras(self) -> None:
+        """Prepare every live camera before any sensor can export the shared renderer stage."""
+        for camera in tuple(self._cameras.values()):
+            camera._prepare_renderer()
+
     def register_visual_material(self, material: Any) -> None:
         """Register one initialized material asset for flat channel composition."""
         if any(registered is material for registered in self._visual_materials):
@@ -372,6 +391,8 @@ def close(self) -> None:
 
         Stage preparation remains valid for the renderer's lifetime across physics resets.
 
+        Live camera registrations survive a stop so their stage preparation runs again on reset.
+
         Raises:
             RuntimeError: If a material writer failed to close.
         """

source/isaaclab/isaaclab/sensors/camera/__init__.pyi

@@ -6,6 +6,7 @@
 __all__ = [
     "Camera",
     "CameraCfg",
+    "CameraExposureCfg",
     "CameraData",
     "CameraISPMode",
     "RenderBufferKind",
@@ -20,6 +21,7 @@ __all__ = [
 
 from .camera import Camera
 from .camera_cfg import CameraCfg
+from .camera_exposure_cfg import CameraExposureCfg
 from .camera_data import CameraData, RenderBufferKind, RenderBufferSpec
 from .camera_isp import CameraISPMode
 from .tiled_camera import TiledCamera