[Task Clean-up] Dexterous Part 3/9: Add success-rate metrics to the reorientation Direct tasks by hujc7 · Pull Request #6413 · isaac-sim/IsaacLab

hujc7 · 2026-07-08T11:41:55Z

Review Map

Exact changes: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution.
Links pin specific SHAs and can go stale after a branch update — the table on [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference) #6324 is refreshed first.

PR	Depends on	Exact changes
#6411 Part 1/9: Newton cloner/cubric/visualizer fixes	—	merged
#6412 Part 2/9: OVPhysX articulation + manager runtime	—	merged
📌 #6413 Part 3/9: Reorient Direct, torch (this PR)	—	changes
#6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct	#6413	changes
#6418 Part 5/9: Reorient manager counterparts	#6413	changes
#6421 Part 6/9: Handover + camera manager counterparts	#6413, #6414, #6418	changes
#6410 Part 7/9: Environment overview docs	#6421	changes
#6415 Part 8/9: Benchmark success-rate utilities	#6421	changes
#6582 Part 9/9: Warp variants → experimental (draft; merges last)	#6413	changes
#6324 [DO-NOT-MERGE] Lumped validation reference	ALL	—

Summary

Adds a behavioral Metrics/success_rate signal (goal-reach streaks per episode) and threshold-independent episode orientation-error diagnostics to the Direct reorientation environments; success gates task health, reward stays diagnostic.
Task logic is torch-first per the mainline convention (plain torch buffers; .torch accessors only at the core-lib boundary). This supersedes the earlier warp-first revision of this PR; the warp implementation moved to isaaclab_tasks_experimental ([Warp] Part 7/8: Add a Shadow Hand reorientation Warp twin and fix the Allegro twin #6582).
Lands shared helpers in isaaclab_tasks.core.utils (EpisodeErrorRecorder, sample_joint_positions_within_limits) with torch math tests; fixes hand resets below lower joint limits.

Stacking

First stack PR; based on develop.

Validation

Kit-free torch math + utils tests green; Direct state training on Newton: success rate 0.85–1.00, reward ≈3600–3900 at 2000 iterations, frame-verified policy video. Full integrated evidence: [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference) #6324.

Review history

The earlier warp-era review rounds on this PR are superseded by the torch restack (2026-07-16); the warp implementation and its review-polished kernels live on in [Warp] Part 7/8: Add a Shadow Hand reorientation Warp twin and fix the Allegro twin #6582.

greptile-apps · 2026-07-08T11:49:36Z

Greptile Summary

This PR adds behavioral Metrics/success_rate and threshold-independent Diagnostics/episode_min_orientation_error_* metrics to the Direct reorientation environments, and fixes hand resets that could place joints below their lower position limits. New shared helpers (EpisodeErrorRecorder, sample_joint_positions_within_limits) land in isaaclab_tasks.core.utils with unit tests.

EpisodeErrorRecorder tracks per-episode minimum orientation error across all active environments, emitting mean/median/p90 aggregates on reset; the recorder carries no success threshold, keeping measurement decoupled from policy.
sample_joint_positions_within_limits replaces the old reset logic that could violate lower joint limits by interpolating uniformly between the default position and each joint's sampled limit endpoint, with a final clamp as a safety net.
The core reward computation is refactored into direct_reorient_reward in the MDP rewards module, and the compute_rewards wrapper now delegates to it; find_joints/find_bodies replace the manual index-and-sort loops for joint and body discovery.

Confidence Score: 4/5

Safe to merge; all new behaviour is additive (metrics, diagnostics, safer resets) and the reward math is functionally unchanged from the original.

The core reward logic, success tracking, and joint-reset fix are all correct. The two findings are cosmetic: an unreachable torch.abs() on a provably non-negative value, and a repeated quaternion-distance computation (up to three times per step) that produces identical results each time. Neither affects training correctness or runtime safety.

reorient_direct_env.py and mdp/rewards.py are worth a second look for the redundant evaluate_reorient_success calls; no other files require special attention.

Important Files Changed

Filename	Overview
source/isaaclab_tasks/isaaclab_tasks/core/utils.py	New shared utilities: EpisodeErrorRecorder for threshold-independent episode-minimum tracking, and sample_joint_positions_within_limits for safe joint resets; logic is correct and well-guarded.
source/isaaclab_tasks/isaaclab_tasks/core/reorient/mdp/rewards.py	Adds direct_reorient_rotation_distance, evaluate_reorient_success, and direct_reorient_reward helpers; a redundant torch.abs() on an already-non-negative value is present in evaluate_reorient_success, and the orientation error is computed multiple times per step.
source/isaaclab_tasks/isaaclab_tasks/core/reorient/reorient_direct_env.py	Wires success-rate metrics, orientation-error diagnostics, and fixed joint-reset sampling into the env; evaluate_reorient_success is called up to three times per step (once in _get_dones, once in _get_rewards for the recorder, once inside direct_reorient_reward).
source/isaaclab_tasks/test/core/test_core_utils.py	Unit tests for EpisodeErrorRecorder and sample_joint_positions_within_limits; covers boundary cases, invalid inputs, and empty-episode edge cases.
source/isaaclab_tasks/changelog.d/task-cleanup-dex-part03.minor.rst	Changelog fragment documenting the added success metrics and joint-limit reset fix.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Env as ReorientDirectEnv
    participant Dones as _get_dones()
    participant Rewards as _get_rewards()
    participant DRR as direct_reorient_reward()
    participant Rec as EpisodeErrorRecorder
    participant Reset as _reset_idx()

    Env->>Dones: step()
    Dones->>Dones: "evaluate_reorient_success() [if max_consecutive_success > 0]"
    Dones-->>Env: (terminated, truncated)

    Env->>Rewards: _get_rewards()
    Rewards->>Rewards: evaluate_reorient_success() → orientation_error
    Rewards->>Rec: update(orientation_error)
    Rewards->>DRR: compute_rewards() → direct_reorient_reward()
    DRR->>DRR: evaluate_reorient_success() [3rd call, same data]
    DRR-->>Rewards: reward, goal_resets, successes, consecutive_successes
    Rewards-->>Env: total_reward

    Env->>Reset: _reset_idx(env_ids)
    Reset->>Reset: "_last_episode_success = successes >= threshold"
    Reset->>Rec: "reset(env_ids) → {mean, median, p90}"
    Reset->>Reset: log Metrics/success_rate
    Reset->>Reset: "log Diagnostics/episode_min_orientation_error_*"
    Reset->>Reset: sample_joint_positions_within_limits()
    Reset-->>Env: done

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Env as ReorientDirectEnv
    participant Dones as _get_dones()
    participant Rewards as _get_rewards()
    participant DRR as direct_reorient_reward()
    participant Rec as EpisodeErrorRecorder
    participant Reset as _reset_idx()

    Env->>Dones: step()
    Dones->>Dones: "evaluate_reorient_success() [if max_consecutive_success > 0]"
    Dones-->>Env: (terminated, truncated)

    Env->>Rewards: _get_rewards()
    Rewards->>Rewards: evaluate_reorient_success() → orientation_error
    Rewards->>Rec: update(orientation_error)
    Rewards->>DRR: compute_rewards() → direct_reorient_reward()
    DRR->>DRR: evaluate_reorient_success() [3rd call, same data]
    DRR-->>Rewards: reward, goal_resets, successes, consecutive_successes
    Rewards-->>Env: total_reward

    Env->>Reset: _reset_idx(env_ids)
    Reset->>Reset: "_last_episode_success = successes >= threshold"
    Reset->>Rec: "reset(env_ids) → {mean, median, p90}"
    Reset->>Reset: log Metrics/success_rate
    Reset->>Reset: "log Diagnostics/episode_min_orientation_error_*"
    Reset->>Reset: sample_joint_positions_within_limits()
    Reset-->>Env: done

Comments Outside Diff (1)

source/isaaclab_tasks/isaaclab_tasks/core/reorient/reorient_direct_env.py, line 202-230 (link)

Triple orientation-error computation per step

evaluate_reorient_success is called up to three times each step with the same inputs: once here for the recorder (line 203), once inside direct_reorient_reward via compute_rewards (line 210), and once in _get_dones when max_consecutive_success > 0 (line 252). All three calls produce identical results because no simulation step occurs between them. The raw orientation_error returned by the direct_reorient_reward path is already available inside that function; surfacing it as a fifth return value (or computing it once before the call and passing it in) would eliminate the duplicate work.

_{Reviews (1): Last reviewed commit: "Add success-rate metrics to the reorient..." | Re-trigger Greptile}

greptile-apps · 2026-07-08T11:49:40Z

+    orientation_error = direct_reorient_rotation_distance(object_quat, target_quat)
+    return torch.abs(orientation_error) <= success_tolerance, orientation_error


torch.abs() is redundant here. direct_reorient_rotation_distance computes 2 * arcsin(clamp(norm, max=1)), where the clamped norm is in [0, 1] and arcsin returns [0, π/2], so the product is always in [0, π]. Applying torch.abs on a provably non-negative tensor is a no-op and slightly misleads readers into thinking the value could be negative.

Suggested change

orientation_error = direct_reorient_rotation_distance(object_quat, target_quat)

return torch.abs(orientation_error) <= success_tolerance, orientation_error

orientation_error = direct_reorient_rotation_distance(object_quat, target_quat)

return orientation_error <= success_tolerance, orientation_error

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

AntoineRichard

Do we want to move more of our logic to pure warp? Worth to discuss during our Thursday architecture meeting. I am also seeing dramatic speed-ups for newton when we start writing more kernels in the tasks. One of my concerns would mostly be on readability this is quite loaded, and some users may struggle to figure things out with this little information to go on. I think we should be a bit more didactic in the kernel descriptions.

AntoineRichard · 2026-07-14T14:12:01Z

+    i, j = wp.tid()
+    num_joints = joint_pos.shape[1]
+    num_fingers = finger_ids.shape[0]
+    # segment boundaries, in column order
+    end_joint = 2 * num_joints
+    end_object = end_joint + 13
+    end_goal = end_object + 11
+    end_tip_pos = end_goal + 3 * num_fingers
+    end_tip_quat = end_tip_pos + 4 * num_fingers
+    end_tip_vel = end_tip_quat + 6 * num_fingers
+    end_wrench = end_tip_vel
+    if with_forces != 0:
+        end_wrench += 6 * num_fingers
+    # hand: normalized DOF positions, scaled DOF velocities
+    if j < num_joints:
+        out[i, j] = 2.0 * (joint_pos[i, j] - lower[i, j]) / (upper[i, j] - lower[i, j]) - 1.0
+    elif j < end_joint:
+        out[i, j] = vel_scale * joint_vel[i, j - num_joints]
+    # object pose and velocities (environment frame position)
+    elif j < end_object:
+        k = j - end_joint
+        if k < 3:
+            p = object_pos_w[i] - env_origins[i]
+            out[i, j] = p[k]
+        elif k < 7:
+            out[i, j] = object_quat[i][k - 3]
+        elif k < 10:
+            out[i, j] = object_lin_vel[i][k - 7]
+        else:
+            out[i, j] = vel_scale * object_ang_vel[i][k - 10]
+    # goal: in-hand anchor, goal rotation, and the goal-to-object rotation difference
+    elif j < end_goal:
+        k = j - end_object
+        if k < 3:
+            out[i, j] = in_hand_pos_e[i][k]
+        elif k < 7:
+            out[i, j] = goal_quat[i][k - 3]
+        else:
+            # quat_inverse == conjugate for these unit quaternions, matching
+            # isaaclab.utils.math.quat_mul/quat_conjugate semantics
+            qe = object_quat[i] * wp.quat_inverse(goal_quat[i])
+            out[i, j] = qe[k - 7]
+    # fingertips: environment-frame positions, rotations, spatial velocities
+    elif j < end_tip_pos:
+        out[i, j] = fingertip_pos_col(body_pos_w, env_origins, finger_ids, i, j - end_goal)
+    elif j < end_tip_quat:
+        out[i, j] = fingertip_quat_col(body_quat_w, finger_ids, i, j - end_tip_pos)
+    elif j < end_tip_vel:
+        out[i, j] = fingertip_vel_col(body_vel_w, finger_ids, i, j - end_tip_quat)
+    # fingertip force/torque sensors (full state only; absent when with_forces == 0)
+    elif j < end_wrench:
+        if with_forces == 1:
+            k = j - end_tip_vel
+            c = k % 6
+            if c < 3:
+                out[i, j] = force_scale * force[i, wrench_ids[k // 6]][c]
+            else:
+                out[i, j] = force_scale * torque[i, wrench_ids[k // 6]][c - 3]
+        else:
+            # full state requested but the sensor has no data yet: zero block
+            out[i, j] = 0.0
+    # actions
+    else:
+        out[i, j] = actions[i, j - end_wrench]
+
+
+@wp.kernel
+def _reduced_obs_kernel(
+    body_pos_w: wp.array2d(dtype=wp.vec3f),
+    env_origins: wp.array(dtype=wp.vec3f),
+    finger_ids: wp.array(dtype=wp.int32),
+    object_pos_w: wp.array(dtype=wp.vec3f),
+    object_quat: wp.array(dtype=wp.quatf),
+    goal_quat: wp.array(dtype=wp.quatf),
+    actions: wp.array2d(dtype=wp.float32),
+    out: wp.array2d(dtype=wp.float32),
+):
+    """Direct reduced (OpenAI) observation, matching the torch concatenation order."""
+    i = wp.tid()
+    num_fingers = finger_ids.shape[0]
+    for f in range(num_fingers):
+        fp = body_pos_w[i, finger_ids[f]] - env_origins[i]
+        out[i, 3 * f + 0] = fp[0]
+        out[i, 3 * f + 1] = fp[1]
+        out[i, 3 * f + 2] = fp[2]
+    idx = 3 * num_fingers
+    p = object_pos_w[i] - env_origins[i]
+    # quat_inverse == conjugate for these unit quaternions, matching
+    # isaaclab.utils.math.quat_mul/quat_conjugate semantics
+    qe = object_quat[i] * wp.quat_inverse(goal_quat[i])
+    out[i, idx + 0] = p[0]
+    out[i, idx + 1] = p[1]
+    out[i, idx + 2] = p[2]
+    out[i, idx + 3] = qe[0]
+    out[i, idx + 4] = qe[1]
+    out[i, idx + 5] = qe[2]
+    out[i, idx + 6] = qe[3]
+    idx += 7
+    for a in range(actions.shape[1]):
+        out[i, idx + a] = actions[i, a]


Why are they not in the re-orient kernel file?

It's direct env specific. Is it still prefer to be moved given it's not used by others?

AntoineRichard · 2026-07-14T14:12:38Z

        self.successes = torch.zeros(self.num_envs, dtype=torch.float, device=self.device)
        self.consecutive_successes = torch.zeros(1, dtype=torch.float, device=self.device)
        self._last_episode_success = torch.zeros(self.num_envs, dtype=torch.bool, device=self.device)


If we are going full warp why not use wp.zeros?

warp envs will be seperate (under warp/), so we can keep the torch for core/ and contrib/

AntoineRichard · 2026-07-14T14:13:21Z

+        wp.launch(
+            ema_actuation_kernel,
+            dim=(self.num_envs, self._actuated_dof_ids_wp.shape[0]),
+            inputs=[
+                wp.from_torch(self.actions),
+                self._lower_limits_wp,
+                self._upper_limits_wp,
+                self._actuated_dof_ids_wp,
+                self.cfg.act_moving_average,
+                self._prev_targets_wp,
+                self._cur_targets_wp,
+            ],
+            outputs=[self._compact_targets_wp],
+            device=self._compact_targets_wp.device,


For extra speeeeeed we could cache the kernel on the first exec using the record launch command.

Will defer those for now. is there a util class merged?

AntoineRichard · 2026-07-14T14:14:38Z

+        # RSL-RL holds the observation reference across the next env.step, so hand out
+        # a per-step snapshot rather than the persistent kernel output buffer.
+        observations = {"policy": obs.clone()}
        if self.cfg.asymmetric_obs:
-            observations["critic"] = self.compute_full_state()
+            observations["critic"] = self.compute_full_state().clone()
        return observations


Rather than cloning outside the kernel launch the kernel launch could write to safe pre-allocated warp arrays with cached torch bindings?

rl lib could take the output and keep it more than one step. So properly using a persistent buffer would require a bit more changes. Deferring it for now.

AntoineRichard · 2026-07-14T14:15:26Z

            self.episode_length_buf = torch.where(
-                torch.abs(rot_dist) <= self.cfg.success_tolerance,
+                goal_reached,
                torch.zeros_like(self.episode_length_buf),
                self.episode_length_buf,
            )


Can't the kernel above do the torch.where?

AntoineRichard · 2026-07-14T14:15:45Z

        self.extras.setdefault("log", {})["Metrics/success_rate"] = (
            self._last_episode_success[env_ids].float().mean().item()
        )


This will force a cuda sync, there is a way to avoid this,

What's the suggested way?

AntoineRichard · 2026-07-14T14:16:36Z

Should have UTs on these kernels?

Add a behavioral Metrics/success_rate signal (goal-reach streaks per episode) and threshold-independent episode orientation-error diagnostics to the Direct reorientation environments, with the shared helpers in isaaclab_tasks.core.utils and torch math tests. The task logic is torch-first per the mainline convention; success gates task health while reward stays diagnostic. Also fix hand resets that could initialize joints below their lower position limits.

…nager runtime (#6412) ## Summary - Fixes OVPhysX actuator joint indices to follow the common actuator indexing contract. - Fixes OVPhysX initialization alongside Kit by reusing Kit's registered PhysX schema provider. - Fixes the OVPhysX manager to support both the declared public runtime API and the current runtime API. - Regression tests included. Validated by full dexterous training runs on the OVPhysX backend; split out of the lumped validation branch #6324 (Part 2 of 11). ## Dependencies - None. ## Series review map Full integrated diff + training/validation evidence: the lumped validation PR #6324 (DO-NOT-MERGE). | Part | PR | |---|---| | Docs: regenerate the environment overview table | #6410 | | Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz teardown) | #6411 | | **Part 2/11: OVPhysX runtime fixes (this PR)** | #6412 | | Part 3/11: success-rate metrics for the Direct reorientation tasks | #6413 | | Part 4/11: RSL-RL training for the handover Direct task | #6414 | | Part 5/11: success-rate support in the benchmark utilities | #6415 | | Part 6/11: renderer presets for the Direct camera task | #6416 | | Part 7/11: OVPhysX presets for the dexterous tasks | #6417 | | Part 8/11: Allegro manager counterpart | #6418 | | Part 9/11: Shadow + OpenAI manager counterparts | #6419 | | Part 10/11: Shadow camera manager counterpart | #6420 | | Part 11/11: Shadow handover manager counterpart | #6421 | --- ### Exact changes in this PR - OVPhysX backend changes + tests: 1f7a433

hujc7 · 2026-07-18T09:34:53Z

Do we want to move more of our logic to pure warp? Worth to discuss during our Thursday architecture meeting. I am also seeing dramatic speed-ups for newton when we start writing more kernels in the tasks. One of my concerns would mostly be on readability this is quite loaded, and some users may struggle to figure things out with this little information to go on. I think we should be a bit more didactic in the kernel descriptions.

@AntoineRichard For this PR, I am updating the torch impl. The already implemented warp one is moving to experimental folder

Fold the reviewed lump changes that belong to this part's content: - Share per-family sim settings through task-cfg base mixins. - Deduplicate backend scene presets via inner-class defaults and nest single-consumer helper cfgs in the shadow-hand Direct cfg. - Compute the orientation error through isaaclab.utils.math.quat_error_magnitude and delete the local direct_reorient_rotation_distance primitive. - Rename direct_reorient_reward to reorient_reward: shared symbols carry no paradigm prefix. Source commits on the lump branch: 2c22af0af0d, 792e400dad0, 42675b64f13, c6140f990d7, 173e9dc5a8d.

Part 3 share of the lump readability round (cdaadac660c): the Direct cfg files keep only task values; asset and marker cfgs, name lists, backend presets, and noise cfgs move to shadow_hand_common and allegro_hand_common, task geometry to reorient_common, and reorient_task_base is removed. The Shadow Direct files carry the workflow marker in their names.

…eanup-dex-part03

The shadow_hand_env_cfg -> shadow_hand_direct_env_cfg rename landed at this layer while two consumers kept importing the removed module name, so the part tree no longer stood alone (caught by arm-ci kitless rendering collection). Pin the camera cfg import to shadow_hand_direct_env_cfg and the handover robot cfg import to shadow_hand_common, matching the final tree.

The rename round updated the camera gym-registration entry-point strings to shadow_hand_direct_camera_env* at this layer, but the camera modules here still carry their pre-rename names, so every registry-driven cfg load failed at import (isaaclab_tasks suites, forbidden-imports test, registered-tasks rendering). Point the registrations back at the modules that exist at this layer.

…isualizer teardown (#6411) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — the table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6410 [Docs] Environment overview regen | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=) | — | — | | 📌 #6411 Part 1/8: Newton cloner/cubric/visualizer fixes (this PR) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=) | — | — | | #6412 Part 2/8: OVPhysX articulation + manager runtime | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=) | — | — | | #6413 Part 3/8: Reorient Direct, torch | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=) | — | — | | #6414 Part 4/8: MARL-to-single-agent fix + handover/camera Direct | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/6e8a63e4e028b2d43676ea30c446b9dc9068c7b5..5cb00e7cb5cc813b202521272e043007cd255194) | | #6418 Part 5/8: Reorient manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/79f87501ac4c81de93a71dab00dc443da62113aa..e7c9a9a3fae3a7972b0c5165ae683abffb7d0e0f) | | #6421 Part 6/8: Handover + camera manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=) | #6413, #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/01c9f4d8c5c35a5688b2a5bb90209e16b8f81b99..835a5815ec49b11aada1d20a76c177054505e6e7) | | #6415 Part 7/8: Benchmark success-rate utilities + docs | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=) | #6413, #6414, #6418, #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/e1abb6b1790ccc57af42551eebccf743633f1f13..d6348539aa8032d9668c20a8fea462c5d88d3af9) | | #6582 Part 8/8: Warp variants → experimental (draft; merges last) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/21dbb1769c4e30c8e9e5b0f563c2dae24c230349..83e1587cadd9712a60615ed2a3cb2d177c2ac24d) | | #6324 [DO-NOT-MERGE] Lumped validation reference | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=) | ALL | — | ## Summary - Fixes Newton cloner label rows, the cubric IAdapter version audit (exact-match fallback to the CPU hierarchy path), and visualizer teardown. - Retains an in-tree `ignore_paths` workaround for custom-frequency USD traversal; it becomes redundant once the Newton pin advance (#6584) merges — this PR then only needs a rebase. ## Stacking - Independent; based on `develop`. ## Review history - Approved. The Newton pin + MuJoCo overrides were split out to #6584 via revert commits (2026-07-17) so this PR's CI runs against develop's pins.

kellyguo11 · 2026-07-23T04:01:59Z

can you also update the environments.rst doc page to include the new physics backends supported? they should also apply to all the shadow hand variants right?

StafaH

Thanks @hujc7, left some comments but nothing major/blocking.

StafaH · 2026-07-23T05:21:35Z

+        ),
+        init_state=RigidObjectCfg.InitialStateCfg(pos=(0.0, -0.17, 0.56), rot=(0.0, 0.0, 0.0, 1.0)),
+    )
+    default = physx


Let's default to newton_mjwarp (here and elsewhere) since our base installation for IsaacLab 3.0 will have newton and not physx

StafaH · 2026-07-23T05:22:02Z

+            cone="elliptic",
+            update_data_interval=2,
+            iterations=100,
+            # save_to_mjcf="AllegroHand.xml",


StafaH · 2026-07-23T05:25:17Z

+            impratio=10.0,
+            cone="elliptic",
+            update_data_interval=2,
+            iterations=100,


For values matching the defaults we can just remove from here (iterations, solver, debug mode).

StafaH · 2026-07-23T05:26:04Z

+            nconmax=70,
+            impratio=10.0,
+            cone="elliptic",
+            update_data_interval=2,


is 2 deliberate here? It's an interesting choice, what does 2 do vs 1 (updating state every step)?

This is common for mywarp config in the repo. seems to improve stability.

StafaH · 2026-07-23T05:40:16Z

+    reward = torch.where(goal_distance >= fall_distance, reward + fall_penalty, reward)
+    resets = torch.where(goal_distance >= fall_distance, torch.ones_like(reset_buf), reset_buf)
+    num_resets = torch.sum(resets)
+    finished_successes = torch.sum(successes * resets.float())


finished_successes = (successes * resets).sum()

StafaH · 2026-07-23T05:40:40Z

+    resets = torch.where(goal_distance >= fall_distance, torch.ones_like(reset_buf), reset_buf)
+    num_resets = torch.sum(resets)
+    finished_successes = torch.sum(successes * resets.float())
+    consecutive_successes = torch.where(


mean_successes = finished_successes / num_resets.clamp_min(1) updated_consecutive_successes = ( averaging_factor * mean_successes + (1.0 - averaging_factor) * consecutive_successes ) consecutive_successes = torch.where( num_resets > 0, updated_consecutive_successes, consecutive_successes, )

StafaH · 2026-07-23T05:40:55Z

@@ -0,0 +1,32 @@
+# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.


Similar comment about common.py

StafaH · 2026-07-23T05:41:56Z

        self.successes = torch.zeros(self.num_envs, dtype=torch.float, device=self.device)
        self.consecutive_successes = torch.zeros(1, dtype=torch.float, device=self.device)
        self._last_episode_success = torch.zeros(self.num_envs, dtype=torch.bool, device=self.device)


warp envs will be seperate (under warp/), so we can keep the torch for core/ and contrib/

StafaH · 2026-07-23T05:44:05Z

@@ -0,0 +1,137 @@
+# Copyright (c) 2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).


Let's avoid adding files in isaaclab_tasks other than the tasks themselves. If its useful for more than one, let's promote it to the global MDP and global utils in isaaclab core

Applies the P3-owned share of the lump review-response commits: - Default the dexterous tasks to newton_mjwarp (from 3f9ce324). - Drop solver defaults matching the backend + inline the scene preset via preset() instead of a wrapper class (from 29e189bf). - Simplify the Direct reorientation reward computation (from bfb73735). - Source the actuated-joint and fingertip body-name lists from the robot assets instead of the config module (from 34676102). Deferred to their owning layers: the core/utils relocation (S17) lands at the manager layer where reorient/mdp/events.py is introduced, and the handover default/preset changes land with the handover part.

StafaH · 2026-07-24T00:49:44Z

 from isaaclab.utils.configclass import configclass

-from isaaclab_tasks.core.reorient.config.shadow_hand.shadow_hand_env_cfg import ShadowHandRobotCfg
+from isaaclab_tasks.core.reorient.config.shadow_hand.shadow_hand_common import ShadowHandRobotCfg


Why does the base env config need to import the robot config? This should be handled in the per-robot config, and probably just use MISSING until then

defer the fixes to next PR. CI has been slow

StafaH · 2026-07-24T00:51:48Z

+
+
+@configclass
+class ObjectCfg(PresetCfg):


nit: CubeObjectCfg or CubeCfg?

StafaH · 2026-07-24T00:52:25Z

+        actuators={},
+        articulation_root_prim_path="",
+    )
+    ovphysx = RigidObjectCfg(


looks identical to physx, ovphysx = physx?

StafaH · 2026-07-24T00:54:20Z

+    OBJECT_CFG,
+    OPENAI_ACTION_NOISE_CFG,
+    OPENAI_OBSERVATION_NOISE_CFG,
+    ROBOT_CFG,


Some stuff is capital, some stuff is not, seems inconsistent

The vision env renders through the Isaac RTX tiled camera, whose render products require the Fabric cloning path. The Newton backend disables Fabric cloning, so under Newton the rgb annotator has no render products at num_envs > 1 and the default RGB/depth/semantic render fails with "Annotator rgb is not attached to any render products". The shared PhysicsCfg/RobotCfg/ObjectCfg now default to Newton, so the camera env inherited a Newton default it cannot render with. Override the camera env's backend PresetCfgs to default to PhysX in __post_init__; Newton stays selectable via physics=newton_mjwarp for the depth-only Newton-warp-renderer benchmark path.

hujc7 · 2026-07-24T05:34:58Z

can you also update the environments.rst doc page to include the new physics backends supported? they should also apply to all the shadow hand variants right?

defer to #6410

… and enable handover Direct RSL-RL (#6414) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. This PR (#6414) is now rebased directly onto `develop`, so its own **Files changed** tab is its contribution. - Links pin specific SHAs and can go stale after a branch update — the table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6411 Part 1/9: Newton cloner/cubric/visualizer fixes | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=) | — | merged | | #6412 Part 2/9: OVPhysX articulation + manager runtime | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=) | — | merged | | #6413 Part 3/9: Reorient Direct, torch | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=) | — | merged | | 📌 #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct (this PR) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=) | — (on develop) | Files changed tab | | #6418 Part 5/9: Reorient manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | #6421 Part 6/9: Handover + camera manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=) | #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | #6410 Part 7/9: Environment overview docs | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=) | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6415 Part 8/9: Benchmark success-rate utilities | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=) | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 9/9: Warp variants → experimental (draft; merges last) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=) | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=) | ALL | — | ## Summary Converts the two-hand Shadow Hand **handover** task from multi-agent (MARL) to **single-agent** (enabling RSL-RL training), fixes the underlying MARL→single-agent observation bridge, and consolidates the shared Shadow Hand definitions into the asset. - **MARL → single-agent bridge (general, broader than this task).** Single-agent RL libraries train `DirectMARLEnv` tasks via `multi_agent_to_single_agent`; the bridge dropped the latest observations from the public buffer. Fixed generally — RSL-RL observations read from the env-owned `obs_buf`, stored by `reset` like `step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the experimental warp base). Every MARL task + single-agent runner benefits; handover is the first consumer. - **Handover Direct → single-agent + RSL-RL.** Single-agent conversion, RSL-RL runner config, shared identity in `handover_common`, success-rate reward metrics, and a fix for its Newton construction failure (see below). - **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp) robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside `SHADOW_HAND_CFG`); reorient and handover consume it as **siblings** (neither imports the other). Reorient uses the default gains; handover overrides only the finger stiffness/damping to `20/2` for its catch. - **Shadow camera cleanup.** Renamed the camera Direct modules to `shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the shared, unit-tested `reorient/mdp/observations.py` (replacing the per-env `compute_keypoints` shim); and added OVPhysX physics presets to the handover and camera Direct environments. The camera env config is otherwise unchanged from develop. ### Behavior changes **Reorient-on-Newton is unchanged.** The Newton hand definition moves **verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions, effort limits, gains, friction and armature — so its config diff vs `develop` is purely the move, and the Direct env config has no diff at all. Handover changes in three ways: - Its **default physics backend moves from PhysX to Newton** (MJWarp), matching the reorientation tasks. Pass `physics=physx` for the previous backend. - Its `distal_passive` actuator group is **removed**. It targeted `robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the environment raised `No joints found for actuator group`. The `fingers` group already drives those joints, and the comment justifying the group (USD-baked `stiffness=286/damping=57`) does not apply to this asset, which authors no drive stiffness at all. - Two orientation fixes: the goal orientation was initialized to a 180-degree rotation instead of identity, and the Newton root rotation replaced the asset's baked rotation instead of composing with it, leaving both palms rotated 90 degrees. Two pre-existing discrepancies were found while verifying against the asset and are left for a follow-up, since each is a behavior change needing its own validation: the config sets `armature=2e-3` where the asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by a fixed tendon that the MJWarp solver currently skips — the actuator gains are what hold that pair together today, which is why the `J4` knuckle-abduction joints cannot simply be added to the driven set. ## Notes - Rebased onto **latest develop**; clean 3-commit history: `Add single-agent flattening for MARL Direct environments` / `Convert Shadow Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow reorient/camera tasks and consolidate the Newton hand`. ## Validation - Kit-free suite green: handover, keypoint math, the MARL adapter, and new checks pinning the RSL-RL wrapper observation contract (verified to fail against the previous implementation). - Camera Direct reaches training takeoff under `rsl_rl` (frame-verified videos); its env config now matches develop's established vision env. - **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500 iterations, 4 runs per config): this branch spans 524–664 reward, `develop` spans 654–822 — overlapping distributions whose per-config spreads (140–168) exceed the gap between them. The move is additionally verified at the config level: the Newton robot configuration is character-for-character identical to `develop`'s. - **Handover-on-Newton** trains end-to-end on this branch (reward 1314), which the `develop` config cannot do at all — it fails during construction. Full evidence: #6324.

…nager runtime (isaac-sim#6412) ## Summary - Fixes OVPhysX actuator joint indices to follow the common actuator indexing contract. - Fixes OVPhysX initialization alongside Kit by reusing Kit's registered PhysX schema provider. - Fixes the OVPhysX manager to support both the declared public runtime API and the current runtime API. - Regression tests included. Validated by full dexterous training runs on the OVPhysX backend; split out of the lumped validation branch isaac-sim#6324 (Part 2 of 11). ## Dependencies - None. ## Series review map Full integrated diff + training/validation evidence: the lumped validation PR isaac-sim#6324 (DO-NOT-MERGE). | Part | PR | |---|---| | Docs: regenerate the environment overview table | isaac-sim#6410 | | Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz teardown) | isaac-sim#6411 | | **Part 2/11: OVPhysX runtime fixes (this PR)** | isaac-sim#6412 | | Part 3/11: success-rate metrics for the Direct reorientation tasks | isaac-sim#6413 | | Part 4/11: RSL-RL training for the handover Direct task | isaac-sim#6414 | | Part 5/11: success-rate support in the benchmark utilities | isaac-sim#6415 | | Part 6/11: renderer presets for the Direct camera task | isaac-sim#6416 | | Part 7/11: OVPhysX presets for the dexterous tasks | isaac-sim#6417 | | Part 8/11: Allegro manager counterpart | isaac-sim#6418 | | Part 9/11: Shadow + OpenAI manager counterparts | isaac-sim#6419 | | Part 10/11: Shadow camera manager counterpart | isaac-sim#6420 | | Part 11/11: Shadow handover manager counterpart | isaac-sim#6421 | --- ### Exact changes in this PR - OVPhysX backend changes + tests: isaac-sim@1f7a433

…eorientation Direct tasks (isaac-sim#6413) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — the table on isaac-sim#6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | isaac-sim#6411 Part 1/9: Newton cloner/cubric/visualizer fixes | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=) | — | merged | | isaac-sim#6412 Part 2/9: OVPhysX articulation + manager runtime | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=) | — | merged | | 📌 isaac-sim#6413 Part 3/9: Reorient Direct, torch (this PR) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=) | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71) | | isaac-sim#6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=) | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8) | | isaac-sim#6418 Part 5/9: Reorient manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=) | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | isaac-sim#6421 Part 6/9: Handover + camera manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=) | isaac-sim#6413, isaac-sim#6414, isaac-sim#6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | isaac-sim#6410 Part 7/9: Environment overview docs | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=) | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | isaac-sim#6415 Part 8/9: Benchmark success-rate utilities | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=) | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | isaac-sim#6582 Part 9/9: Warp variants → experimental (draft; merges last) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=) | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | isaac-sim#6324 [DO-NOT-MERGE] Lumped validation reference | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=) | ALL | — | ## Summary - Adds a behavioral `Metrics/success_rate` signal (goal-reach streaks per episode) and threshold-independent episode orientation-error diagnostics to the Direct reorientation environments; success gates task health, reward stays diagnostic. - Task logic is **torch-first** per the mainline convention (plain torch buffers; `.torch` accessors only at the core-lib boundary). This supersedes the earlier warp-first revision of this PR; the warp implementation moved to `isaaclab_tasks_experimental` (isaac-sim#6582). - Lands shared helpers in `isaaclab_tasks.core.utils` (`EpisodeErrorRecorder`, `sample_joint_positions_within_limits`) with torch math tests; fixes hand resets below lower joint limits. ## Stacking - First stack PR; based on `develop`. ## Validation - Kit-free torch math + utils tests green; Direct state training on Newton: success rate 0.85–1.00, reward ≈3600–3900 at 2000 iterations, frame-verified policy video. Full integrated evidence: isaac-sim#6324. ## Review history - The earlier warp-era review rounds on this PR are superseded by the torch restack (2026-07-16); the warp implementation and its review-polished kernels live on in isaac-sim#6582.

… and enable handover Direct RSL-RL (isaac-sim#6414) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. This PR (isaac-sim#6414) is now rebased directly onto `develop`, so its own **Files changed** tab is its contribution. - Links pin specific SHAs and can go stale after a branch update — the table on isaac-sim#6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | isaac-sim#6411 Part 1/9: Newton cloner/cubric/visualizer fixes | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=) | — | merged | | isaac-sim#6412 Part 2/9: OVPhysX articulation + manager runtime | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=) | — | merged | | isaac-sim#6413 Part 3/9: Reorient Direct, torch | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=) | — | merged | | 📌 isaac-sim#6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct (this PR) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=) | — (on develop) | Files changed tab | | isaac-sim#6418 Part 5/9: Reorient manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=) | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | isaac-sim#6421 Part 6/9: Handover + camera manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=) | isaac-sim#6414, isaac-sim#6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | isaac-sim#6410 Part 7/9: Environment overview docs | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=) | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | isaac-sim#6415 Part 8/9: Benchmark success-rate utilities | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=) | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | isaac-sim#6582 Part 9/9: Warp variants → experimental (draft; merges last) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=) | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | isaac-sim#6324 [DO-NOT-MERGE] Lumped validation reference | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=) | ALL | — | ## Summary Converts the two-hand Shadow Hand **handover** task from multi-agent (MARL) to **single-agent** (enabling RSL-RL training), fixes the underlying MARL→single-agent observation bridge, and consolidates the shared Shadow Hand definitions into the asset. - **MARL → single-agent bridge (general, broader than this task).** Single-agent RL libraries train `DirectMARLEnv` tasks via `multi_agent_to_single_agent`; the bridge dropped the latest observations from the public buffer. Fixed generally — RSL-RL observations read from the env-owned `obs_buf`, stored by `reset` like `step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the experimental warp base). Every MARL task + single-agent runner benefits; handover is the first consumer. - **Handover Direct → single-agent + RSL-RL.** Single-agent conversion, RSL-RL runner config, shared identity in `handover_common`, success-rate reward metrics, and a fix for its Newton construction failure (see below). - **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp) robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside `SHADOW_HAND_CFG`); reorient and handover consume it as **siblings** (neither imports the other). Reorient uses the default gains; handover overrides only the finger stiffness/damping to `20/2` for its catch. - **Shadow camera cleanup.** Renamed the camera Direct modules to `shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the shared, unit-tested `reorient/mdp/observations.py` (replacing the per-env `compute_keypoints` shim); and added OVPhysX physics presets to the handover and camera Direct environments. The camera env config is otherwise unchanged from develop. ### Behavior changes **Reorient-on-Newton is unchanged.** The Newton hand definition moves **verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions, effort limits, gains, friction and armature — so its config diff vs `develop` is purely the move, and the Direct env config has no diff at all. Handover changes in three ways: - Its **default physics backend moves from PhysX to Newton** (MJWarp), matching the reorientation tasks. Pass `physics=physx` for the previous backend. - Its `distal_passive` actuator group is **removed**. It targeted `robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the environment raised `No joints found for actuator group`. The `fingers` group already drives those joints, and the comment justifying the group (USD-baked `stiffness=286/damping=57`) does not apply to this asset, which authors no drive stiffness at all. - Two orientation fixes: the goal orientation was initialized to a 180-degree rotation instead of identity, and the Newton root rotation replaced the asset's baked rotation instead of composing with it, leaving both palms rotated 90 degrees. Two pre-existing discrepancies were found while verifying against the asset and are left for a follow-up, since each is a behavior change needing its own validation: the config sets `armature=2e-3` where the asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by a fixed tendon that the MJWarp solver currently skips — the actuator gains are what hold that pair together today, which is why the `J4` knuckle-abduction joints cannot simply be added to the driven set. ## Notes - Rebased onto **latest develop**; clean 3-commit history: `Add single-agent flattening for MARL Direct environments` / `Convert Shadow Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow reorient/camera tasks and consolidate the Newton hand`. ## Validation - Kit-free suite green: handover, keypoint math, the MARL adapter, and new checks pinning the RSL-RL wrapper observation contract (verified to fail against the previous implementation). - Camera Direct reaches training takeoff under `rsl_rl` (frame-verified videos); its env config now matches develop's established vision env. - **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500 iterations, 4 runs per config): this branch spans 524–664 reward, `develop` spans 654–822 — overlapping distributions whose per-config spreads (140–168) exceed the gap between them. The move is additionally verified at the config level: the Newton robot configuration is character-for-character identical to `develop`'s. - **Handover-on-Newton** trains end-to-end on this branch (reward 1314), which the `develop` config cannot do at all — it fails during construction. Full evidence: isaac-sim#6324.

…anager counterparts (#6418) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — this table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6411 Part 1/10: Newton cloner/cubric/visualizer fixes | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=) | — | merged | | #6412 Part 2/10: OVPhysX articulation + manager runtime | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=) | — | merged | | #6413 Part 3/10: Reorient Direct, torch | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=) | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71) | | #6414 Part 4/10: MARL-to-single-agent fix + handover/camera Direct | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8) | | 📌 #6418 Part 5/10: Reorient manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/3ea6f7bbf6c..0271783ea17) | | #6421 Part 6/10: Handover + camera manager counterparts | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=) | #6413, #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/b31b04eaf71..0cae3d87be4) | | #6410 Part 7/10: Environment overview docs | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=) | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6954 Part 8/10: OpenAI Shadow variants → contrib (draft) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6954?label=) | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6954/changes/0cae3d87be4..b0d2d8448af) | | #6415 Part 9/10: Benchmark success-rate utilities | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=) | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 10/10: Warp variants → experimental (draft; merges last) | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=) | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference | ![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=) | ALL | — | ## Summary Adds manager-based counterparts for the Shadow cube reorientation task and aligns the existing Allegro manager task with the Direct contracts. - **3 manager tasks** mirror their Direct counterparts: `Isaac-Reorient-Cube-Shadow`, `-Shadow-Camera` and `-Allegro`. - The OpenAI variants move to the contributed tasks **unchanged**. They encode a paper-specific training regime, so they get no manager counterpart. - `Metrics/success_rate` now means the same thing in both workflows. - The in-hand cube is one object per hand instead of one per physics backend. ## 1. Manager tasks match the Direct contracts Observations, actions, rewards, terminations, reset distributions and timing were brought to the Direct values. A value-parity test asserts the decimation / episode length / simulation step triple, the orientation success tolerance and the fall distance, so drift on either side fails CI. The Allegro observation space changes size, so existing manager checkpoints must be retrained. Domain randomization becomes opt-in through `presets=randomized`, replacing a configuration-file flag that the command line could not override. ## 2. OpenAI variants move to the contributed tasks `Isaac-Reorient-Cube-Shadow-OpenAI-{FF,LSTM}-Direct` become `IsaacContrib-Reorient-Cube-Shadow-OpenAI-{FF,LSTM}-Direct`. Both identifiers are kept: they share one environment configuration and differ only in their agent configuration, so collapsing them would leave the recurrent policy unreachable — the training entry point selects an agent configuration by workflow, not by name. ## 3. One success rate, defined once `Metrics/success_rate` named two quantities. The manager tasks reported goals reached over goals presented; the Direct tasks reported a per-episode success bit drawn at `success_count_threshold`. The curves were not comparable despite sharing a name and measuring the same task. Both now report the per-attempt rate, and `success_count_threshold` is removed as unused. The goal accounting moves into a `SuccessTracker` beside `EpisodeErrorRecorder`, which also suppresses success detection for the single step after an automatic reset — where the goal has been drawn but no physics has run against it, so a reset pose inside the success threshold would otherwise bank a goal the policy never earned. ## 4. One in-hand cube per hand, not per backend The cube was a `PresetCfg` with a variant per physics backend, so each hand manipulated a different object depending on the backend. On Shadow the branches disagreed on size and spawn height: PhysX spawned a 60 mm rigid body at `z=0.6`, Newton a 54 mm **articulation with no joints** at `z=0.535`. A policy trained on one backend was not solving the same task as one trained on the other. Each hand now declares a single `CUBE_CFG`. Shadow keeps the PhysX values; Allegro keeps its 1.2 scale. The `mass_props` density overrides are dropped: the asset authors an absolute `physics:mass` of 0.216 kg, which takes precedence over density, so they never applied on either backend. ## 5. Configuration hierarchy flattened, classes named by role Every manager task overrode all seven of the shared base's sub-configurations, so the base carried pre-alignment defaults that each task then undid. The environment configurations now derive from `ManagerBasedRLEnvCfg` directly. Removed: `ReorientObjectEnvCfg`, the shared sub-configurations no task constructed, and `reorient_common`. `isaaclab_tasks.core.utils` moves to `isaaclab_tasks.core.reorient.utils`, which the hand-over task already imports from. One naming convention now applies throughout, since `SceneCfg`, `EventCfg` and `ObservationsCfg` are each defined in dozens of files and gave no clue which base a task was specializing. Classes shared across files carry the task prefix; the `Base` suffix marks only those a subclass must complete by supplying fields the parent leaves unset, so the observation halves are named as parts rather than templates. Classes nested inside another configuration drop the prefix. Registered and preset-selectable configurations always name their task and hold fields only, with their helpers declared beside them. ## Validation Eight arms, 1500 iterations, 8192 environments, `physics=newton_mjwarp`, one L40 each. All exited cleanly with no regression in reward. | Arm | reward before | reward after | |---|---|---| | `Isaac-Reorient-Cube-Shadow` | 67.08 | 66.24 | | `-Shadow` `presets=asymmetric` | 42.08 | 52.95 | | `-Shadow-Direct` | 3532.95 | 3967.58 | | `-Allegro` | 25.22 | 34.03 | | `-Allegro` `presets=randomized` | 16.03 | 39.59 | | `-Allegro-Direct` | 656.78 | 1003.11 | | `IsaacContrib-…-OpenAI-FF-Direct` | 2757.56 | 4533.21 | | `IsaacContrib-…-OpenAI-LSTM-Direct` | — | 2585.54 | The LSTM arm resolves `ShadowHandAsymLSTMPPORunnerCfg`, confirming the recurrent policy is reachable through its own identifier. ### Known failure `test_rendering_shadow_hand[newton-isaacsim_rtx-rgb]` fails; it passes on `develop`. Only the `rgb` modality is affected — depth, normals and all three segmentations pass on the same backend, so the cube move is not an obvious explanation. Root cause is not yet established.

…#7970) # Description The `reset_hand` kernel in the experimental Warp reorientation env sampled hand joint reset positions around the **lower joint limit** instead of between the lower and upper limits: ```python dof_pos_noise = wp.randf(rng_state[env_id], -1.0, 1.0) rand_delta = delta_min + (delta_max - delta_min) * 0.5 * dof_pos_noise # centred on delta_min ``` As a result, every hand joint reset at or below its default position. With a large enough `reset_dof_pos_noise` it could also end up outside the joint limits. For example, with limits `[-1, 1]`, default `0.1` and noise scale `0.2`, resets fell in `[-0.32, 0.08]` instead of `[-0.12, 0.28]`. The torch reorient and handover tasks were already fixed in #6413 via `sample_joint_positions_within_limits`. This PR applies the same `[-1, 1] -> [0, 1]` mapping in the Warp kernel, so both versions of the task reset joints over the same range. This was the last copy of the formula from the issue on `develop`. Fixes #4404 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Test I added `test_warp_reset_hand_samples_the_same_range_as_the_torch_helper` to the existing `source/isaaclab_tasks/test/core/test_reorient_utils.py`. It launches `reset_hand` on CPU for 512 envs and checks two things: - every sample stays inside the range that `sample_joint_positions_within_limits` produces; - each joint gets samples on both sides of its default. The test fails without the fix and passes with it (`uv run python -m pytest source/isaaclab_tasks/test/core/test_reorient_utils.py`: 12 passed). ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump `extension.toml` — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there - [x]  Backport this pull request to the active release branch after it merges into `develop` --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…#7970) # Description The `reset_hand` kernel in the experimental Warp reorientation env sampled hand joint reset positions around the **lower joint limit** instead of between the lower and upper limits: ```python dof_pos_noise = wp.randf(rng_state[env_id], -1.0, 1.0) rand_delta = delta_min + (delta_max - delta_min) * 0.5 * dof_pos_noise # centred on delta_min ``` As a result, every hand joint reset at or below its default position. With a large enough `reset_dof_pos_noise` it could also end up outside the joint limits. For example, with limits `[-1, 1]`, default `0.1` and noise scale `0.2`, resets fell in `[-0.32, 0.08]` instead of `[-0.12, 0.28]`. The torch reorient and handover tasks were already fixed in #6413 via `sample_joint_positions_within_limits`. This PR applies the same `[-1, 1] -> [0, 1]` mapping in the Warp kernel, so both versions of the task reset joints over the same range. This was the last copy of the formula from the issue on `develop`. Fixes #4404 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Test I added `test_warp_reset_hand_samples_the_same_range_as_the_torch_helper` to the existing `source/isaaclab_tasks/test/core/test_reorient_utils.py`. It launches `reset_hand` on CPU for 512 envs and checks two things: - every sample stays inside the range that `sample_joint_positions_within_limits` produces; - each joint gets samples on both sides of its default. The test fails without the fix and passes with it (`uv run python -m pytest source/isaaclab_tasks/test/core/test_reorient_utils.py`: 12 passed). ## Checklist - [x] I have read and understood the [contribution guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html) - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have added a changelog fragment under `source/<pkg>/changelog.d/` for every touched package (do **not** edit `CHANGELOG.rst` or bump `extension.toml` — CI handles that) - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there - [x]  Backport this pull request to the active release branch after it merges into `develop` --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit 244496d)

github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 8, 2026

This was referenced Jul 8, 2026

[Task Clean-up] Dexterous Part 4/9: Fix MARL-to-single-agent training and enable handover Direct RSL-RL #6414

Merged

[Task Clean-up][Benchmark] Dexterous Part 9/10: Add success-rate support to the benchmark utilities #6415

Closed

greptile-apps Bot reviewed Jul 8, 2026

View reviewed changes

AntoineRichard reviewed Jul 14, 2026

View reviewed changes

hujc7 force-pushed the jichuanh/task-cleanup-dex-part03 branch from b1e2a42 to cf21603 Compare July 15, 2026 09:12

hujc7 force-pushed the jichuanh/task-cleanup-dex-part03 branch from ef2b22c to 6e8a63e Compare July 17, 2026 00:44

hujc7 changed the title ~~[Task Clean-up] Dexterous Part 3/11: Add success-rate metrics to the reorientation Direct tasks~~ [Task Clean-up] Dexterous Part 3: Add success-rate metrics to the reorientation Direct tasks Jul 17, 2026

hujc7 changed the title ~~[Task Clean-up] Dexterous Part 3: Add success-rate metrics to the reorientation Direct tasks~~ [Task Clean-up] Dexterous Part 3/11: Add success-rate metrics to the reorientation Direct tasks Jul 17, 2026

hujc7 mentioned this pull request Jul 17, 2026

[Warp] Part 7/8: Add a Shadow Hand reorientation Warp twin and fix the Allegro twin #6582

Draft

6 of 8 tasks

hujc7 changed the title ~~[Task Clean-up] Dexterous Part 3/11: Add success-rate metrics to the reorientation Direct tasks~~ [Task Clean-up] Dexterous Part 3/8: Add success-rate metrics to the reorientation Direct tasks Jul 17, 2026

hujc7 requested a review from StafaH July 17, 2026 23:51

hujc7 requested a review from AntoineRichard July 18, 2026 09:34

hujc7 requested a review from ooctipus as a code owner July 18, 2026 11:10

hujc7 requested a review from pascal-roth as a code owner July 18, 2026 11:10

hujc7 requested a review from a team July 18, 2026 11:10

hujc7 added 4 commits July 20, 2026 15:32

Merge remote-tracking branch 'upstream/develop' into jichuanh/task-cl…

a3e8045

…eanup-dex-part03

kellyguo11 approved these changes Jul 23, 2026

View reviewed changes

StafaH approved these changes Jul 23, 2026

View reviewed changes

github-actions Bot added the asset New asset feature or request label Jul 23, 2026

Add isaaclab_assets changelog fragment for the name-list additions

d29afc7

hujc7 changed the title ~~[Task Clean-up] Dexterous Part 3/8: Add success-rate metrics to the reorientation Direct tasks~~ [Task Clean-up] Dexterous Part 3/9: Add success-rate metrics to the reorientation Direct tasks Jul 23, 2026

StafaH approved these changes Jul 24, 2026

View reviewed changes

hujc7 merged commit 4c9e5b7 into isaac-sim:develop Jul 24, 2026
41 checks passed

hujc7 mentioned this pull request Aug 7, 2026

[Task Clean-up][Contrib] Dexterous Part 8/10: Move the OpenAI Shadow Hand variants to contrib #6954

Closed

This was referenced Sep 23, 2026

fix: correct joint position sampling range in dexterous hand envs #4680

Closed

[Tasks] Fix Warp reorient reset sampling around the lower joint limit #7970

Merged

		orientation_error = direct_reorient_rotation_distance(object_quat, target_quat)
		return torch.abs(orientation_error) <= success_tolerance, orientation_error

		@@ -0,0 +1,32 @@
		# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
		# All rights reserved.

		@@ -0,0 +1,137 @@
		# Copyright (c) 2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).

Conversation

hujc7 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Map

Summary

Stacking

Validation

Review history

Uh oh!

greptile-apps Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Sequence Diagram

Comments Outside Diff (1)

Uh oh!

greptile-apps Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

AntoineRichard left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hujc7 commented Jul 18, 2026

Uh oh!

kellyguo11 commented Jul 23, 2026

Uh oh!

StafaH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hujc7 commented Jul 8, 2026 •

edited

Loading

greptile-apps Bot commented Jul 8, 2026 •

edited

Loading

AntoineRichard left a comment •

edited

Loading

hujc7 commented Jul 24, 2026 •

edited

Loading