Add dictionary and single-tensor processors for cameras and observations

docs/source/api/lab/isaaclab.envs.mdp.rst

@@ -9,6 +9,9 @@ Observations
 .. automodule:: isaaclab.envs.mdp.observations
     :members:
 
+.. automodule:: isaaclab.envs.mdp.sensor_observations
+    :members:
+
 Actions
 -------
 

docs/source/api/lab/isaaclab.managers.rst

@@ -103,7 +103,7 @@ On each recorded sample, each environment retains its lag with probability 0.8 a
 one. Zero redraws on every sample. Holding the lag keeps the latency constant while frames continue to
 advance; it does not freeze the returned frame. Reset always redraws the selected environments' lags.
 
-The processing order is observation function, modifiers, noise, clipping, scaling, delay, then history.
+The processing order is observation function, tensor processors, noise, clipping, scaling, delay, then history.
 The ``delay_min_lag`` and ``delay_max_lag`` field names and delay placement follow
 `mjlab's observation configuration <https://mujocolab.github.io/mjlab/main/source/observations.html>`_.
 ``delay_hold_prob`` follows mjlab's lag-retention semantics, with a default of 1.0 to preserve Isaac Lab's
@@ -115,7 +115,7 @@ For a delay buffer with no recorded sample after initialization or reset, the cu
 without recording it. Once recording starts, delays exceeding the available history return the oldest
 sample. Partial resets invalidate only the selected environments' histories.
 
-Observation outputs are independent snapshots: later simulation steps, modifier updates, and resets
+Observation outputs are independent snapshots: later simulation steps, processor updates, and resets
 do not overwrite previously returned tensors. The manager automatically copies borrowed storage before
 mutation or retention. Term and custom callback outputs are treated conservatively, without decorators
 or task-level copy settings. Clipping and scaling create independent storage when needed, which later
@@ -131,7 +131,7 @@ The manager checks the signature once during initialization. Only an explicitly
 ``out=None`` enables destination writes; accepting arbitrary ``**kwargs`` does not. The manager probes
 the ordinary call to infer shape, dtype and device, which must remain constant, then allocates a fresh
 contiguous destination for each computation. ``out`` is reserved and must not appear in term ``params``.
-Custom modifiers, noise and history retain their usual snapshot protections. Built-in
+Custom tensor processors, noise and history retain their usual snapshot protections. Built-in
 :class:`~isaaclab.envs.mdp.observations.image_rgb` terms use this interface automatically to avoid a
 second full-image copy for normalized uint8 images. Existing observation configurations require no changes.
 

docs/source/api/lab/isaaclab.sensors.camera.rst

@@ -13,7 +13,12 @@ The following classes are part of the public :mod:`isaaclab.sensors.camera` API.
 .. autosummary::
    :nosignatures:
 
+   CameraExposureCfg
    CameraISPMode
 
 .. autoclass:: CameraISPMode
    :show-inheritance:
+
+.. autoclass:: CameraExposureCfg
+   :members:
+   :exclude-members: __init__

docs/source/api/lab/isaaclab.utils.rst

@@ -17,7 +17,7 @@
       logger
       math
       mesh
-      modifiers
+      tensor_processors
       noise
       seed
       sensors
@@ -124,10 +124,19 @@ Mesh operations
    :imported-members:
    :show-inheritance:
 
-Modifier operations
-~~~~~~~~~~~~~~~~~~~
+Tensor processors
+~~~~~~~~~~~~~~~~~
+
+The processor interfaces live together in :mod:`isaaclab.utils.tensor_processors`:
+
+* ``TensorProcessor``: tensor to tensor.
+* ``DictTensorProcessor``: dictionary of named tensors to dictionary of named tensors.
+
+Dictionary tensors use ``ProxyArray`` for Warp and Torch access without copying. Processors own
+any persistent outputs and state. ``DictTensorProcessorCfg`` declares input/output keys for camera
+setup. Camera and observation consumers compose processors and preserve untouched keys.
 
-.. automodule:: isaaclab.utils.modifiers
+.. automodule:: isaaclab.utils.tensor_processors
    :members:
    :imported-members:
    :special-members: __call__

docs/source/api/lab_ppisp/isaaclab_ppisp.rst

@@ -14,10 +14,33 @@ The following classes are part of the public :mod:`isaaclab_ppisp` API.
    :nosignatures:
 
    PpispCfg
+   PpispProcessorCfg
+   PpispProcessor
+   PpispControllerWeights
+   PpispDiscoveryMode
    PpispPipeline
 
 .. autoclass:: PpispCfg
    :show-inheritance:
 
+.. autoclass:: PpispProcessorCfg
+   :members:
+   :inherited-members:
+   :exclude-members: __init__
+   :show-inheritance:
+
+.. autoclass:: PpispProcessor
+   :members:
+   :inherited-members:
+   :special-members: __call__
+   :show-inheritance:
+
+.. autoclass:: PpispControllerWeights
+   :members:
+
+.. autoclass:: PpispDiscoveryMode
+   :members:
+   :show-inheritance:
+
 .. autoclass:: PpispPipeline
    :show-inheritance:

docs/source/concepts/sensors/camera.rst

@@ -153,8 +153,9 @@ updating it adds frame-query overhead.
 Output types
 ------------
 
-The active renderer validates ``data_types`` and allocates the channel count and data type declared
-by its :class:`~isaaclab.renderers.RenderBufferSpec`.
+For raw outputs, the active renderer validates the requested names and allocates the channel count
+and data type declared by its :class:`~isaaclab.renderers.RenderBufferSpec`. A camera pipeline may
+produce additional output names; its processors allocate those buffers.
 
 .. list-table:: Common output contracts
    :header-rows: 1
@@ -168,7 +169,7 @@ by its :class:`~isaaclab.renderers.RenderBufferSpec`.
      - Low-dynamic-range color
    * - ``rgb_hdr``
      - 3, ``float32``
-     - Scene-linear high-dynamic-range color
+     - High-dynamic-range RGB using the active camera settings
    * - ``albedo``
      - 4, ``uint8``
      - Material base color
@@ -198,6 +199,12 @@ by its :class:`~isaaclab.renderers.RenderBufferSpec`.
 non-colorized segmentation uses one ``int32`` ID channel. Label and prim-path mappings are stored in
 ``camera_data.info[output_name]``.
 
+``rgb_hdr`` uses the camera's active exposure settings. PPISP requires scene-linear HDR;
+set ``CameraCfg.exposure=CameraExposureCfg()`` explicitly to disable RTX automatic exposure and
+use neutral exposure. Leaving ``exposure=None`` preserves authored camera settings. Selecting an
+output never changes exposure. PPISP requests ``rgb_hdr`` privately; include it in ``data_types``
+only to publish the raw input as well.
+
 .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/camera-renderer-isaac-rtx.webp
    :align: center
    :figwidth: 100%
@@ -263,6 +270,12 @@ current support matrix is:
      - Yes
      - No
 
+.. note::
+
+   Explicit exposure settings affect every output from the camera prim, including other sensors
+   sharing it. Use separate camera prims when different exposures are required. Newton's HDR
+   output is already scene-linear and does not consume RTX exposure attributes.
+
 Querying an unsupported output fails during camera initialization. Renderer configuration controls
 semantic filters, segmentation colorization, and depth clipping where those options are
 backend-specific.
@@ -305,47 +318,168 @@ background. Set a normalized RGB tuple to use a solid color for pixels that miss
 The setting is per camera. Cameras with renderer-default and solid backgrounds can coexist in one
 scene.
 
-Post-render image signal processing
------------------------------------
-
-:attr:`~sensors.CameraCfg.isp_cfg` optionally applies an image signal processing (ISP) pass to the
-renderer's scene-linear HDR output. The shipped implementation is PPISP (Physically Plausible Image
-Signal Processing), which applies responsivity, exposure, vignetting, color correction, and a camera
-response function before writing ``rgb`` or ``rgba``.
+Post-render image processing
+----------------------------
 
-The field accepts:
+:attr:`~sensors.CameraCfg.pipeline` composes dictionary processors after rendering.
+:class:`~isaaclab.utils.tensor_processors.DictTensorProcessor` takes a dictionary of named tensors
+and returns a dictionary of named tensors. It lives beside
+:class:`~isaaclab.utils.tensor_processors.TensorProcessor`, whose call takes and returns one tensor.
+:class:`~isaaclab.utils.tensor_processors.DictTensorProcessorCfg` declares the actual input and output
+keys needed for renderer setup. The shared dictionary interface also accepts nonvisual sensor data;
+each processor defines its layout and value semantics.
 
-* ``None`` to disable post-render ISP.
-* :class:`~isaaclab_ppisp.PpispCfg` for explicit coefficients or coefficients imported from a USD
-  camera.
-* :class:`~sensors.CameraISPMode` to discover ``ppisp:*`` attributes on a camera prim.
+The camera resolves external inputs before renderer initialization, including inputs needed only by the
+pipeline. It publishes the names in ``data_types`` after processing. For example, this camera requests
+private ``rgb_hdr`` from its renderer and publishes PPISP's processed ``rgb``:
 
 .. code-block:: python
 
-   from isaaclab.sensors.camera import CameraCfg, CameraISPMode
+   from isaaclab.sensors.camera import CameraExposureCfg
    from isaaclab.utils import replace
-   from isaaclab_ppisp import PpispCfg
+   from isaaclab_ppisp import PpispCfg, PpispProcessorCfg
+
+   processed_camera = replace(
+       front_camera,
+       data_types=["rgb"],
+       exposure=CameraExposureCfg(),
+       pipeline=[PpispProcessorCfg(params={"cfg": PpispCfg(inputs={"exposureOffset": 1.5})})],
+   )
+
+PPISP (Physically Plausible Image Signal Processing) applies responsivity, exposure, vignetting, color
+correction, and a camera response function. It consumes float32 scene-linear radiance and produces
+uint8 color. Set ``outputs=["rgba"]`` and ``params={"cfg": ppisp_cfg, "channels": 4}`` on the
+processor configuration for RGBA output. A pipeline-produced RGBA buffer also supplies a zero-copy
+RGB view when ``rgb`` is requested. A pipeline producing only RGB cannot satisfy an explicit RGBA request.
+
+For custom dictionary processing, subclass ``DictTensorProcessor``. Its constructor receives
+``(cfg, data)`` with actual ``ProxyArray`` tensors. Allocate persistent output tensors in
+``self.outputs`` without reading input values or advancing state. Implement ``__call__(data)``
+to fill and return those named outputs, and override ``reset`` or ``close`` only when needed.
+The pipeline passes the current dictionary to each processor and merges its returned outputs;
+untouched keys pass through. Shape, dtype and device come from the arrays directly.
+
+``PpispProcessor`` subclasses ``DictTensorProcessor`` and can also be constructed and called directly:
+
+.. code-block:: python
+
+   from isaaclab.utils import instantiate
+
+   cfg = PpispProcessorCfg()
+   processor = instantiate(cfg, {"rgb_hdr": hdr_tensor})
+   color = processor({"rgb_hdr": hdr_tensor})["rgb"]
+   processor.close()
+
+Here ``hdr_tensor`` is a scene-linear float32 ``ProxyArray`` with shape ``(N, H, W, 3)``.
+The returned color is uint8; copy it before another call if its values must be retained.
+
+Published camera outputs must keep the camera's batch size, height, width, and NHWC layout so pose
+and calibration metadata remain valid. Intermediate buffers may change shape. Use ``processed_sensor``
+below when the final observation should have a different image shape or layout.
+
+``camera.render_outputs`` exposes borrowed raw renderer buffers, including private inputs. Their
+``ProxyArray`` properties provide shape, dtype, and device. Reading raw outputs updates the camera
+when due. Neither a processor nor a caller may mutate borrowed inputs. Public processed
+buffers and raw buffers retain their storage until camera reinitialization, but later captures replace
+their contents; clone values that must survive another capture. Each camera owns its pipeline and processor
+workspace. Processors may share explicitly supplied immutable model resources.
+Configuration copying must preserve the identity of borrowed resources: PPISP provides
+``PpispControllerWeights`` for device weights. Custom processors can use a resource wrapper with
+identity-preserving ``__deepcopy__``; raw Warp allocations are not configuration values.
 
-   explicit_isp = replace(front_camera, data_types=["rgb"], isp_cfg=PpispCfg(inputs={"exposureOffset": 1.5}))
+Pipelines run after scalar and batched captures. When an asynchronous renderer republishes a capture,
+the camera retains its processed result and matching capture metadata. Camera reset forwards only the
+selected environment indices to processor reset callbacks; teardown closes the processors. For an
+observation-only pipeline, declare raw dependencies with ``camera.request_render_inputs(names)`` before
+simulation initialization. Requests for new inputs after initialization fail because renderer setup is
+already complete. Observation consumers use their own pipeline and snapshot results needed across captures.
 
-   discovered_isp = replace(front_camera, data_types=["rgb"], isp_cfg=CameraISPMode.AUTO_CAMERA)
+Processing sensor observations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-``AUTO_CAMERA`` checks the first matched camera prim. ``AUTO_ANY`` falls back to the first PPISP
-camera anywhere on the stage. Discovery happens once during camera construction.
+Use :class:`~isaaclab.envs.mdp.processed_sensor` for observation-only processing, including processors
+that combine several camera outputs or change image dimensions. The term currently binds sensors
+through their ``request_render_inputs()`` and ``render_outputs`` interface:
 
-PPISP is composed by Isaac RTX, OVRTX, and Newton Warp. It requires ``rgb`` or ``rgba`` output. A
-static configuration is shared by all cloned views in one camera batch; controller weights may
-predict per-view exposure and color parameters, while the remaining coefficients stay shared. ISP
-configuration and discovered USD attributes are fixed for the camera lifetime.
+.. code-block:: python
+
+   from isaaclab.envs import mdp
+   from isaaclab.managers import ObservationTermCfg, SceneEntityCfg
+   from isaaclab_ppisp import PpispProcessorCfg
+
+   front_camera.exposure = CameraExposureCfg()
+   policy_image = ObservationTermCfg(
+       func=mdp.processed_sensor,
+       params={
+           "sensor_cfg": SceneEntityCfg("front_camera"),
+           "pipeline": [PpispProcessorCfg()],
+           "output_name": "rgb",
+       },
+   )
+
+Manager-based environments collect the term's private renderer inputs before simulation reset. A
+standalone observation manager must call ``ObservationManager.prepare_scene(cfg, env)`` before reset.
+The term processes raw renderer outputs even when the camera has its own pipeline. To read the
+camera's published processed output, use an existing image term such as ``mdp.image_rgb`` instead.
+The observation pipeline executes on every term evaluation and returns a snapshot; its state belongs to
+that term rather than the camera's capture cadence.
+
+For a single incoming tensor, :class:`~isaaclab.utils.tensor_processors.DictTensorProcessorAdapterCfg` uses the
+same dictionary processors while preserving preceding observation transformations. This includes nonvisual
+observation tensors from sensors without the render-input interface. This form must preserve shape;
+use ``processed_sensor`` for multiple render inputs or changes to render-output shapes, or construct
+a ``DictTensorProcessor`` directly for other named buffers. The first incoming tensor constructs the
+dictionary processors; later calls reuse their output storage. Its values must meet each processor's
+documented input requirements, including any transformations from preceding tensor processors:
+
+.. code-block:: python
+
+   from isaaclab.utils import tensor_processors
+
+   # This camera publishes radiance for the ordinary image term to read.
+   front_camera = replace(front_camera, data_types=["rgb_hdr"], exposure=CameraExposureCfg(), pipeline=[])
+   policy_image = ObservationTermCfg(
+       func=mdp.image_rgb,
+       params={"sensor_cfg": SceneEntityCfg("front_camera"), "data_type": "rgb_hdr", "normalize": False},
+       tensor_processors=[
+           tensor_processors.DictTensorProcessorAdapterCfg(
+               pipeline=[PpispProcessorCfg()],
+               input_name="rgb_hdr",
+               output_name="rgb",
+           )
+       ],
+   )
 
-.. important::
+Migrating renderer-owned ISP configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-   With Isaac RTX and OVRTX, enabling ``isp_cfg`` makes PPISP the ISP authority. The renderer
-   disables RTX auto-exposure, authors neutral ``exposure:*`` values, and applies the
-   ``OmniRtxCameraAutoExposureAPI_1`` and ``OmniRtxCameraExposureAPI_1`` schemas on every matched
-   camera prim so RTX does not process the image a second time. Do not combine ``isp_cfg`` with
-   separately authored RTX exposure or tonemapping settings. When ``isp_cfg`` is ``None``, the
-   renderer leaves authored camera exposure unchanged.
+``CameraCfg.isp_cfg`` no longer executes image processing. A non-None value raises a migration error;
+replace it with ``CameraCfg.pipeline`` as above. ``CameraISPMode`` remains importable for legacy
+configuration values, but new code should resolve USD attributes explicitly before constructing the
+processor:
+
+.. code-block:: python
+
+   from isaaclab_ppisp import PpispDiscoveryMode, PpispProcessorCfg, resolve_and_normalize
+
+   ppisp_cfg = resolve_and_normalize(PpispDiscoveryMode.AUTO_CAMERA, stage, "/World/Camera")
+   discovered_camera = replace(
+       front_camera,
+       data_types=["rgb"],
+       exposure=CameraExposureCfg(),
+       pipeline=[PpispProcessorCfg(params={"cfg": ppisp_cfg})] if ppisp_cfg is not None else [],
+   )
+
+``AUTO_CAMERA`` checks the supplied camera prim; ``AUTO_ANY`` can fall back to the first PPISP camera
+on the supplied stage. Alternatively, pass an already resolved camera prim to
+``ppisp_cfg_from_usd_camera(camera_prim)``. Discovery has no implicit simulation or stage lookup inside
+the processor. Static coefficients apply to the whole camera batch; controller weights may predict
+per-view exposure and color parameters.
+
+For either processing attachment, configure the source camera with
+``exposure=CameraExposureCfg()`` before simulation initialization. PPISP's own exposure is set
+through ``PpispCfg.inputs["exposureOffset"]`` (stops). Processing does not discover or change
+renderer exposure settings.
 
 Run the ``ppisp-camera`` example for a complete PPISP workflow. It supports
 ``--renderer newton_renderer`` (Newton Warp, the default) and ``--renderer isaac_rtx`` (Isaac RTX).
@@ -414,3 +548,9 @@ A runnable camera example is available as ``camera``:
 
 For saving output to disk, see :doc:`/source/how-to/save_camera_output`. For renderer selection
 and customization, see :doc:`/source/how-to/configure_rendering`.
+
+The observation processor API is ``isaaclab.utils.tensor_processors``. Custom stateful processors
+inherit ``TensorProcessor`` and use ``TensorProcessorCfg``; attach configurations through
+``ObservationTermCfg.tensor_processors``. These replace ``ModifierBase``, ``ModifierCfg``, and
+``ObservationTermCfg.modifiers`` respectively. ``DictTensorProcessorAdapterCfg`` adapts a dictionary
+processor pipeline to that single-tensor interface.

examples/sensors/ppisp_camera.py

@@ -120,12 +120,13 @@
 import matplotlib.pyplot as plt
 import numpy as np
 import torch
-from isaaclab_ppisp.cfg import PpispCfg, ppisp_cfg_from_usd_camera
+from isaaclab_ppisp import PpispCfg, PpispProcessorCfg, ppisp_cfg_from_usd_camera
 
 import isaaclab.sim as sim_utils
 from isaaclab.assets import AssetBaseCfg, RigidObjectCfg
 from isaaclab.scene import InteractiveSceneCfg
 from isaaclab.sensors import CameraCfg
+from isaaclab.sensors.camera import CameraExposureCfg
 from isaaclab.sim.spawners.materials import UsdPhysicsRigidBodyMaterialCfg
 from isaaclab.utils import configclass, instantiate
 
@@ -374,7 +375,8 @@ def make_camera(camera_prim_path: str, *, ppisp_cfg: PpispCfg | None, width: int
         width=width,
         data_types=["rgb"],
         spawn=None,
-        isp_cfg=ppisp_cfg,
+        exposure=CameraExposureCfg() if ppisp_cfg is not None else None,
+        pipeline=[PpispProcessorCfg(params={"cfg": ppisp_cfg})] if ppisp_cfg is not None else [],
         renderer_cfg=make_renderer_cfg(),
     )
     return instantiate(camera_cfg)