Repository navigation
[Camera] Remove remaining hot-path host syncs and overhead [7B/7] - #8105
Conversation
- Log command, termination, curriculum and success-rate episode metrics as device scalars instead of host floats. - Select heading-controlled envs in UniformVelocityCommand with torch.where instead of nonzero(). - Cache event range tensors by value, and make reset_root_state_uniform honor the ranges passed at call time. - Replace the DiffIK zero-quaternion guard with a sync-free torch.where. - Batch RewardManager per-term updates and resets over a term matrix. - Skip defensive copies around built-in noise and delay buffers. - Shift small CircularBuffer frames in two kernels regardless of length. - Check event modes via dict membership in the env step.
- Skip ray-caster drift sampling on reset when drift ranges are zero and upload ray-cast drift ranges only when they change. - Stop eager per-step refreshes for debug-visualized sensors; debug callbacks now refresh outdated buffers lazily before drawing. - Return early from VisualizationMarkers.visualize when no backend is active. - Drop per-frame channel-compacting copies in the Isaac RTX renderer by indexing padded annotator tiles with their source channel count. - Remove the Newton Warp renderer read_output copy loop (outputs alias the camera buffers) and an unreachable PhysX ray-caster guard.
Store direct-task joint, body, and tendon ids as device tensors, precompute static limits and default poses, use the actuator target-command API, log success metrics as 0-d tensors, compute Lift contact terms from one stacked norm, cache the ANYmal symmetry permutations and constant tensors, and send only marker indices for static success markers.
|
run-ci |
…perf/camera-perf-07b-hot-path # Conflicts: # source/isaaclab_tasks/test/core/test_lift_env_cfg.py
|
run-ci |
|
| if not (sample_drift or sample_ray_cast_drift): | ||
| return |
There was a problem hiding this comment.
If a ray caster’s drift range changes from nonzero to zero, reset() leaves the previously sampled drift in its buffer. Later ray casts still use that offset, so disabling drift does not stop it from perturbing sensor measurements. Clear the selected drift buffer when its range becomes zero.
Knowledge Base Used: Simulation, rendering, and sensors
There was a problem hiding this comment.
Fixed in ec65628: when a drift range becomes zero, reset() now zeroes the corresponding buffer instead of skipping it (the skip only applies while nothing has ever been sampled).
| if self._static_success_vis_pos_w is not None: | ||
| # positions and environment ids were set at construction; resending them copies to the host | ||
| self.success_visualizer.visualize(marker_indices=success.int()) | ||
| return |
There was a problem hiding this comment.
If the lift command is created before a marker backend is active, its one-time position update is discarded. When a backend becomes active later, these calls send only marker_indices, so the success markers never receive their per-environment positions. Resending the positions on first backend activation would preserve the markers in that case.
Knowledge Base Used: Simulation, rendering, and sensors
There was a problem hiding this comment.
Fixed in ec65628: the static-marker shortcut is removed, so the success markers always receive their positions and environment ids on every visualize call and a late-activating backend gets them too.
There was a problem hiding this comment.
Isaac Lab Review Bot
The hot-path optimizations are broadly coherent, but CurriculumManager.get_active_iterable_terms() now returns cloned tensors while its public type contract still promises floats. Align this accessor’s implementation, annotation, documentation, and consumers before merge.
- Design and architecture: The device-resident logging and caching strategy is consistent with the performance goal. The remaining concern is that a reset-path synchronization optimization was also applied to a public inspection accessor without defining the resulting tensor-valued contract.
- API:
CurriculumManager.get_active_iterable_terms()is declared as returningSequence[tuple[str, Sequence[float]]], but changed tensor states from Python floats to 0-d tensors. Either retain.item()in this accessor or explicitly update its annotation, documentation, changelog guidance, and affected consumers. - Implementation: At
curriculum_manager.py:165and the equivalent scalar-state path,detach().clone()deterministically preserves tensor values rather than producing the floats promised by the signature. The implementation and public contract therefore need to be reconciled.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
There was a problem hiding this comment.
We're allocating new tensors here, we could avoid that if we pre-allocated
| """Per-environment contact force magnitude [N] of a single-body, single-filter contact sensor.""" | ||
| force = sensor.data.normal_force_matrix_w.torch.view(num_envs, 3) | ||
| return torch.linalg.norm(force, dim=-1) | ||
| def _contact_force_mags(env: ManagerBasedRLEnv, sensor_names: Sequence[str]) -> torch.Tensor: |
There was a problem hiding this comment.
_contact_force_mag made more sense?
|
run-ci |
… uv docs CLI (#8118, #8105, #8179) (#8210) ## Summary Backports three `develop` PRs to `release/3.0.0`, in merge order: 1. #8118 — towncrier changelog tooling 2. #8105 — camera hot-path cleanup [7B/7] 3. #8179 — uv CLI for Windows docs builds The automatic backport failed for #8118 and #8105. For #8179, this supersedes the bot's draft: - #8209 — bot draft backport of #8179 Once merged, the nightly compiles `release/3.0.0` again: it skips release branches without `tools/changelog/towncrier.toml`. # Description Each PR is cherry-picked with `-x`; every code file matches its source patch. Release-specific changes: 1. #8118 (`ee99db6a820`) - Each `CHANGELOG.rst` conflict resolved as the marker line above the release entries. - `uv.lock` unchanged: release's local package pins already match their `pyproject.toml`. - The bot failed here because `uv.lock` pushed its conflict context past the 1.5M-character inference limit. 2. #8105 (`cd917691992`) - Drops the `.github/workflows/build.yaml` hunk: it edits the `performance-smoke` job, which `release/3.0.0` does not have. The bot's validation rejected this omission. - Splits its 10 old-format fragments into the new format with `cli.py`'s `split_legacy`, keeping all 34 bullets; #8118's check rejects the old format. 3. #8179 (`c1a8bfa50b5`) - Deletes `docs/make.bat`; release held an older copy. - Splits its fragment the same way. - Release's `docs/_redirect/index.html` still hardcodes `./main/` (#7767 is not on this branch), so `isaaclab --docs_multi` writes that redirect unchanged, as release's `make multi-docs` does. ## Type of change - Release backport, carrying #8105's breaking change (`isaaclab` major bump) - New feature, bug fix and documentation update from the source PRs ## Validation 1. `cli.py check release/3.0.0` passes. 2. `tools/changelog` tests: 18 passed. 3. `pre-commit` over the range passes. 4. `cli.py compile` on a scratch copy of this branch: - 9 packages bumped, including `isaaclab` 25.1.0 → 26.0.0 and `isaaclab_tasks_experimental` 0.0.1 → 1.0.0 - every fragment consumed, only its outputs staged - `uv.lock` changed in version lines only 5. Runtime coverage for #8105 and #8179 is left to CI. ## Release backport This PR targets the release branch; no additional backport is needed. ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) on the changed files - [x] I have made corresponding changes to the documentation (carried from the source PRs) - [ ] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage (source PRs' tests carried unchanged) - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (carried from the source PRs; #8105 and #8179 converted to the new format) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
## Description The ANYmal symmetry augmentation used by the `rsl_rl_with_symmetry` agents hard-coded joint indices for the PhysX joint order. The velocity tasks default to Newton, where ANYmal's native order is depth-first, so every mirrored joint observation and action came from the wrong joint. This PR resolves the left-right and front-back permutations and signs from the robot's `joint_names`, using the ANYmal `<L|R><F|H>_<HAA|HFE|KFE>` naming, and feeds them into the cached permutation tables from #8105 (now keyed by joint order). The sign conventions are unchanged: HAA is negated for left-right, and HFE/KFE for front-back. With the PhysX order the tables equal the previous ones. Resolving by name matches the ordering contract, where public order is backend-native unless `joint_ordering` is set, and works for any `joint_ordering`; ANYmal-B/C use the same joint names. Pinning `joint_ordering="physx"` in the task would add a per-step reorder on Newton and change the observation and action layout of existing Newton checkpoints. Fixes #8204 Verification on `Isaac-Velocity-Flat-AnymalD` and `Isaac-Velocity-Rough-AnymalD`, on both PhysX and Newton, at the current head: - Wrong mirrored joints go from 12/12 to 0/12 on Newton, and stay at 0/12 on PhysX. - The physical mirror check (base-frame feet of the mirrored configuration against the mirrored feet) goes from 0.17-0.42 m to below 1e-5 m on Newton. - `test_anymal_symmetry.py` keeps the PhysX reference check from #8105 and adds one check that the Newton order is mirrored like the PhysX order with the joint columns reordered; it fails on develop and passes with this change. - The per-call path is unchanged: one cached gather and multiply per transform. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Release backport - [x] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump package versions — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com>
## Description The ANYmal symmetry augmentation used by the `rsl_rl_with_symmetry` agents hard-coded joint indices for the PhysX joint order. The velocity tasks default to Newton, where ANYmal's native order is depth-first, so every mirrored joint observation and action came from the wrong joint. This PR resolves the left-right and front-back permutations and signs from the robot's `joint_names`, using the ANYmal `<L|R><F|H>_<HAA|HFE|KFE>` naming, and feeds them into the cached permutation tables from #8105 (now keyed by joint order). The sign conventions are unchanged: HAA is negated for left-right, and HFE/KFE for front-back. With the PhysX order the tables equal the previous ones. Resolving by name matches the ordering contract, where public order is backend-native unless `joint_ordering` is set, and works for any `joint_ordering`; ANYmal-B/C use the same joint names. Pinning `joint_ordering="physx"` in the task would add a per-step reorder on Newton and change the observation and action layout of existing Newton checkpoints. Fixes #8204 Verification on `Isaac-Velocity-Flat-AnymalD` and `Isaac-Velocity-Rough-AnymalD`, on both PhysX and Newton, at the current head: - Wrong mirrored joints go from 12/12 to 0/12 on Newton, and stay at 0/12 on PhysX. - The physical mirror check (base-frame feet of the mirrored configuration against the mirrored feet) goes from 0.17-0.42 m to below 1e-5 m on Newton. - `test_anymal_symmetry.py` keeps the PhysX reference check from #8105 and adds one check that the Newton order is mirrored like the PhysX order with the joint columns reordered; it fails on develop and passes with this change. - The per-call path is unchanged: one cached gather and multiply per transform. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Release backport - [x] <!-- backport-active-release --> Backport this pull request to the active release branch after it merges into `develop` ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `uv run isaaclab --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have validated the changed behavior with focused coverage and applied the [test-audit criteria](https://github.com/isaac-sim/IsaacLab/blob/develop/skills/developer/test-audit/SKILL.md) to any test changes - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump package versions — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com> (cherry picked from commit abe7db4)
Description
Part 7B of the camera performance series, based on the latest
develop(including #8039, #8042, and #8087). A sweep of the per-step hot path found remaining host-device synchronizations and code that costs more than it gives. This PR fixes the verified findings: it removes the synchronizations, launch overhead, and dead or over-defensive code, and fixes the verified bugs.Profiling on
developmotivated this. The GPU sat idle 20–27% of each step, because the CPU blocked on synchronizations and then launched hundreds of small kernels. OnIsaac-Velocity-Rough-AnymalD(4096 environments), for example, the CPU was blocked 12.8 ms of a 25.4 ms step.Changes
Host synchronizations removed
survival_success_rate, the pose commands, and task success rates (Cabinet, Locomotion, Lift deformable/cable, Fourbar) log 0-d device tensors instead of reading back.item()/.cpu()on every reset.RewardManageralready logged tensors, and the RL wrappers accept them. Curriculum states are logged as copies, because rsl_rl keeps episode infos for a whole rollout. The inspection accessorget_active_iterable_terms()still returns Python scalars.torch.where) instead ofis_heading_env.nonzero()every step. Output is identical.sample_uniform_from_ranges()to draw named components directly. One private, bounded bounds cache replaces both sets of tensor-building helpers; it is keyed by values, resolved device, and dtype. Curriculum edits take effect on the next call, and every call draws fresh samples in the existing order.reset()no longer uploads its range tensor or samples drift when the drift ranges are zero (the default). In rough locomotion this reset runs almost every step. When drift is disabled after sampling, reset clears the selected environments’ old drift.if ee_quat.norm() != 0guard inapply_actions, which synchronized every physics substep, is now atorch.wherewith the same all-or-nothing behavior.actuators.target_commandAPI.walk_target_wcaches its device target (it synchronized 4 times per step).EpisodeErrorRecorderusestorch.wherewith NaN reductions instead of boolean masks.Launch overhead
RewardManager: each term writes its weighted value into a step-reward matrix column, followed by one sum and one batched episode-sum update. Reset is one mean and one fill. With 20 terms at 4096 environments, compute drops from 121 to 23 kernels (471 → 152 µs) and reset from 80 to 4 kernels (504 → 49 µs).ObservationManager: the built-in noise functions (constant_noise,gaussian_noise,uniform_noise) are out-of-place, so the pre-noise clone and the second clone are skipped. Safe built-in noise functions are identified during_prepare_terms; custom noise functions, noise models, and modifiers still clone, including when a caller replaces a noise function after setup. The noise sampling order is unchanged.CircularBuffer: one staged shift replaces K−1 slot copies for frames up to 2 MiB. Image-sized frames keep the slot loop, which is faster for them.contacts()runs 5 times per step in Kuka Allegro, and this removes about 80 kernels per step.rgb_hdr, and simple shading no longer copy each frame into a contiguous channel slice before reshaping.doneson the host side.Overengineered code removed
NewtonManager.step()ran the solver-internals reset every physics step.forward()already performs it when kinematics are dirty, so the extra call only launched with an all-False mask.WrenchComposer.reset()returns early when inactive. Its buffers are already zero, and this removes about 7 memsets per rigid object per physics step (21 µs → 0.07 µs per call).SensorBase: sensors withdebug_vis=Trueno longer refresh on every physics substep. That refresh existed because debug callbacks once read private data. The callbacks now refresh the outdated buffers themselves. The original eager update was introduced in commit4565902f5to make visualization work without adataread; it did not pump rendering.VisualizationMarkers.visualize()returns early when no backend is active.body_state_bindex conversion (redundant since Cache scene selections with automatic Torch dispatch #8087).Bug fixes
body_com_acc_w: the finite difference divided by the physics timestep even when an update spans several physics steps (Newton owning the decimation loop). That made the acceleration, andbody_lin_acc_l2,decimation× too large. It now uses the elapsed time, matchingjoint_acc. The OvPhysX rigid-object implementation uses the same elapsed-time formula. PhysX and OvPhysX articulations read acceleration from the backend, so they do not need this finite-difference change.reset_root_state_uniformbuilt its range tensors in__init__and ignored call-timepose_range/velocity_range, so curriculum updates were silently dropped.Investigated and not included
nonzero: these are now the only synchronizations left inIsaac-Velocity-Rough-AnymalD. Each converts a mask into variable-length environment IDs consumed across the reset, command, and event APIs. Removing them requires a mask-based API redesign and is left for a separate design.joint_actionsallocation is needed for LEAPP export tracing.Type of change
float()or.item()where a Python number is needed.actuated_dof_indices,finger_bodies,finger_wrench_bodies) andresolve_actuated_tendonsids are tensors.EpisodeErrorRecorderreports NaN for an empty selection after its first update._debug_vis_callback.Performance
RTX PRO 6000 Blackwell Workstation Edition (driver 595.91.07); PyTorch 2.12.0+cu130; Warp 1.17.0; Newton 1.6.0; GPU otherwise idle.
developat60bc09f17compared with this PR on that base, with the sameuv.lock. The branch has since mergeddevelopatf25d3f09b(#8042, #8090, #8102). The numbers below were measured before that merge; #8042 changes the Kuka camera observations and #8090 touches the simulation context, so they have not been re-measured on the merged branch or after review fixes. Static Lift success markers now resend positions to support late backend activation; the earlier marker-index-only optimization is not included. Runtime benchmark (isaaclab benchmark runtime), Newton/MJWarp, seed 42, no visualizer. Three fresh processes per revision, run in alternating order. Figures are the mean step time across processes; every workload's per-process ranges are disjoint.developIsaac-Ant, 4096 envsIsaac-Open-Drawer-Franka-Direct, 4096 envsIsaac-Reorient-Cube-Allegro-Direct, 4096 envsIsaac-Velocity-Rough-AnymalD, 4096 envsIsaac-Lift-KukaAllegro-Camera, 256 envs,presets=duo_camera,rgb128, Newton rendererPer-process step times (
develop→ PR):3.367, 3.359, 3.382→2.639, 2.638, 2.611ms.11.113, 11.102, 11.104→10.099, 10.100, 10.100ms.27.136, 27.136, 27.070→25.217, 25.250, 25.197ms.23.081, 23.050, 23.103→22.551, 22.568, 22.599ms.12.389, 12.374, 12.424→12.373, 12.119, 12.113ms.Host synchronizations per
env.step(), attributed to call sites with PyTorch sync-debug mode over 170 steps:developIsaac-Velocity-Rough-AnymalDnonzeroIsaac-Lift-KukaAllegro-Cameranonzero; curriculum difficulty.item()(0.04 per step)PyTorch labels sync-debug mode a prototype that does not detect every synchronization, and it does not see synchronizations issued by Warp, so these counts are lower bounds.
Validation
isaacsimextra, and OvPhysX usedovphysxwith separate CPU/GPU processes.uv run isaaclab -fpassed, including changelog checks against the actual PR base, andgit diff --checkpassed.Release backport
developChecklist
uv run --frozen isaaclab -f.CONTRIBUTORS.md.