[mataylor-nvidia]

Superseded by two stacked draft PRs:

1. Rename modifiers to tensor processors and add dictionary processing (#8693), targeting upstream develop.
2. Move PPISP processing from renderers into camera pipelines (mataylor-nvidia/IsaacLab#86), targeting the first PR's branch in the fork. Upstream repository rules prohibit creating the branch needed for a stacked base there. Once Rename modifiers and unify tensor processor APIs #8693 merges, the upper commit can be rebased onto develop and submitted upstream.

The cleanup defers the observation dictionary adapters, removes their preparation/teardown machinery, consolidates camera setup and duplicate tests, and retains deprecated modifier imports/configuration during migration. PPISP production changes are +627 / -629; tests and documentation account for the remaining net growth.

Both replacement branches are based on current develop and have passed focused tests, formatting/lint, changelog checks, and focused Sphinx validation. Full documentation is blocked by the installed Omniverse Kit EULA requirement; live OVRTX/Isaac RTX and remote GPU CI remain unrun. The original branch and review history are preserved.

[mataylor-nvidia]

Design proposal: Move image processing into RenderBufferPipe. Renderers supply raw buffers; cameras and observations decide when to process them and how to expose the results.

Draft: #8571

Renderer buffers → camera-owned pipe      → camera.data.output
Renderer buffers → observation-owned pipe → observation snapshot
Incoming tensor  → modifier-owned pipe    → observation snapshot

Buffers are ProxyArrays. Shape, dtype, and device come directly from storage.
PreparedBufferOperation describes executable processing:

output_specs
apply
reset
close

output_specs tells the pipe what to allocate. apply(inputs, outputs) writes into those allocations. Optional reset and close callbacks manage operation state and resources.
ProcessingBufferStepCfg connects that operation to named buffers:

prepare
inputs
outputs
params

Then map buffer names to operation roles. mechanism supports an operation with multiple inputs. Preparation happens once then execution loop looks like

current = dict(buffers)
for operation, inputs, outputs, destinations in self._operations:
    operation.apply({role: current[name] for role, name in inputs.items()}, destinations)
    current.update({name: destinations[role] for role, name in outputs.items()})
return current

The consumers have control over delivery

• Cameras publish borrowed buffers; observations return snapshots that survive subsequent processing.

• modifiers shouldn't know data names: The factory only gets arrays keyed by role, plus params. It never sees the step config (images.py:121)

• one operation, several inputs (fog: rgb + depth → rgb) inputs={"color": "rgb", "depth": ...}, outputs={"color": "rgb"}

• point Cosmos at specific outputs, each step names its inputs and outputs explicitly

• hide rgb_radiance

• The renderer is asked for it before it starts; it's in render_outputs but not in data.output unless listed

• plain list as a shortcut pipeline=[PpispStepCfg()] with default names

• Don't make every BaseModifier handle batches

• ModifierBase is unchanged; steps are a separate type, connected through RenderBufferPipeModifier

• scale before PPISP
  Works by putting RenderBufferPipeModifierCfg after a scale modifier, but only for single-input steps

• PpispControllerWeights are different per controller

The current form used is a list:

CameraCfg(
    data_types=["rgb", "depth"],
    # PpispStepCfg defaults: inputs={"radiance": "rgb_radiance"}, outputs={"color": "rgb"}
    pipeline=[PpispStepCfg(params={"cfg": ppisp_cfg})],
)

# The dict form Octi and Mustafa proposed (not implemented)
CameraCfg(data_types=["rgb", "depth"], modifiers={"rgb": [PpispModifierCfg()]})

[mataylor-nvidia]

Isaac Lab’s existing convention for callable MDP terms: classes such as image_depth, image_features, and stacked_image also use snake_case. They’re configured through the same func field as ordinary functions:

ObservationTermCfg(func=mdp.processed_sensor, ...)

[ooctipus]

are you sure this is needed for this pr?