Repository navigation
[Task Clean-up][Docs] Dexterous Part 7/10: Environment overview updates for the reorientation series - #6410
Closed
hujc7 wants to merge 27 commits into
Closed
[Task Clean-up][Docs] Dexterous Part 7/10: Environment overview updates for the reorientation series#6410hujc7 wants to merge 27 commits into
hujc7 wants to merge 27 commits into
Conversation
github-actions
Bot
added
the
documentation
Contributor
This was referenced Jul 8, 2026
Merged
[Task Clean-up] Dexterous Part 3/9: Add success-rate metrics to the reorientation Direct tasks
#6413
Merged
Closed
Closed
Merged
[Task Clean-up][Manager] Dexterous Part 9/11: Add Shadow state and OpenAI manager counterparts
#6419
Closed
Add a behavioral Metrics/success_rate signal (goal-reach streaks per episode) and threshold-independent episode orientation-error diagnostics to the Direct reorientation environments, with the shared helpers in isaaclab_tasks.core.utils and torch math tests. The task logic is torch-first per the mainline convention; success gates task health while reward stays diagnostic. Also fix hand resets that could initialize joints below their lower position limits.
Add the RSL-RL runner configuration for the Shadow handover Direct task and success-rate metrics on the torch-first path, fix handover construction on Newton (renamed distal joints), and land the camera Direct renderer presets with configuration validation. RSL-RL observations now read from the public environment-owned obs_buf on all Direct env bases (reset stores the buffer like step), replacing the adapter-side private hook.
hujc7
added a commit
that referenced
this pull request
Jul 17, 2026
…nager runtime (#6412) ## Summary - Fixes OVPhysX actuator joint indices to follow the common actuator indexing contract. - Fixes OVPhysX initialization alongside Kit by reusing Kit's registered PhysX schema provider. - Fixes the OVPhysX manager to support both the declared public runtime API and the current runtime API. - Regression tests included. Validated by full dexterous training runs on the OVPhysX backend; split out of the lumped validation branch #6324 (Part 2 of 11). ## Dependencies - None. ## Series review map Full integrated diff + training/validation evidence: the lumped validation PR #6324 (DO-NOT-MERGE). | Part | PR | |---|---| | Docs: regenerate the environment overview table | #6410 | | Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz teardown) | #6411 | | **Part 2/11: OVPhysX runtime fixes (this PR)** | #6412 | | Part 3/11: success-rate metrics for the Direct reorientation tasks | #6413 | | Part 4/11: RSL-RL training for the handover Direct task | #6414 | | Part 5/11: success-rate support in the benchmark utilities | #6415 | | Part 6/11: renderer presets for the Direct camera task | #6416 | | Part 7/11: OVPhysX presets for the dexterous tasks | #6417 | | Part 8/11: Allegro manager counterpart | #6418 | | Part 9/11: Shadow + OpenAI manager counterparts | #6419 | | Part 10/11: Shadow camera manager counterpart | #6420 | | Part 11/11: Shadow handover manager counterpart | #6421 | --- ### Exact changes in this PR - OVPhysX backend changes + tests: 1f7a433
6 of 8 tasks
Fold the reviewed lump changes that belong to this part's content: - Share the handover sim settings through the HandoverTaskCfgBase mixin instead of a module-level cfg variable. - Reuse isaaclab.utils.math.quat_mul for the Newton initial-rotation composition instead of a local xyzw product helper. - Nest the single-consumer tiled-camera helper cfg inside the camera task cfg. - Drop the stale kit-less OpenUSD bullet from the isaaclab changelog fragment. Source commits on the lump branch: 2c22af0af0d, 2c5ea5bb6c7, 42675b64f13, 1f05f850886.
Fold the reviewed lump changes that belong to this part's content: - Share per-family sim settings through task-cfg base mixins. - Deduplicate backend scene presets via inner-class defaults and nest single-consumer helper cfgs in the shadow-hand Direct cfg. - Compute the orientation error through isaaclab.utils.math.quat_error_magnitude and delete the local direct_reorient_rotation_distance primitive. - Rename direct_reorient_reward to reorient_reward: shared symbols carry no paradigm prefix. Source commits on the lump branch: 2c22af0af0d, 792e400dad0, 42675b64f13, c6140f990d7, 173e9dc5a8d.
Part 3 share of the lump readability round (cdaadac660c): the Direct cfg files keep only task values; asset and marker cfgs, name lists, backend presets, and noise cfgs move to shadow_hand_common and allegro_hand_common, task geometry to reorient_common, and reorient_task_base is removed. The Shadow Direct files carry the workflow marker in their names.
…eanup-dex-part04 # Conflicts: # source/isaaclab_tasks/test/core/test_dexterous_task_math.py
Part 4 share of the lump readability round (cdaadac660c): the camera Direct files carry the workflow marker in their names and read the camera geometry from reorient_common; handover_task_base is renamed handover_common and the handover files take the Shadow identity from shadow_hand_common. Sim settings are declared inline per file.
The shadow_hand_env_cfg -> shadow_hand_direct_env_cfg rename landed at this layer while two consumers kept importing the removed module name, so the part tree no longer stood alone (caught by arm-ci kitless rendering collection). Pin the camera cfg import to shadow_hand_direct_env_cfg and the handover robot cfg import to shadow_hand_common, matching the final tree.
|
|
||
| Removed | ||
| ^^^^^^^ | ||
|
|
Collaborator
There was a problem hiding this comment.
[AI Review][Important] Blocking per the repository API policy: public APIs cannot be removed until they have been deprecated in a prior release. A major changelog entry does not replace that transition period. Please retain ReorientObjectEnvCfg, the exported MDP terms, and the renamed public modules as forwarding compatibility shims that emit DeprecationWarning; remove them only after the documented deprecation window.
| state_space = 290 | ||
|
|
||
| # simulation | ||
| # simulation — values mirrored by the manager cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] I cannot find the claimed value-parity test, while these simulation and task constants are duplicated again in HandoverManagerEnvCfg. This comment is therefore factually misleading and the values can drift silently. Prefer a shared task-parameter source; if duplication is unavoidable, add an explicit Direct/manager contract test and keep production comments focused on behavior rather than test implementation.
| ccd_iterations=50, # bumped from default 35 for multi-finger contact geometry | ||
| ), | ||
| num_substeps=2, | ||
| # 4 substeps (vs the single-agent port's 2): sustained ball-palm contact |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This is an experiment log rather than durable source documentation: it embeds a particular environment count, probe duration, and observed failure percentage to justify one value. Please keep only the functional invariant here, such as that four substeps prevent non-finite sustained ball–palm contacts, and move the benchmark history/evidence to the PR or benchmark documentation. Please apply the same cleanup to the other provenance-heavy comments in this configuration.
| random_values = math_utils.sample_uniform(-1.0, 1.0, (count, 2), device=device) | ||
| x_unit = torch.tensor([1.0, 0.0, 0.0], device=device).repeat(count, 1) | ||
| y_unit = torch.tensor([0.0, 1.0, 0.0], device=device).repeat(count, 1) | ||
| return math_utils.quat_mul( |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] random_xy_rotation() at lines 58-74 and randomize_rotation() at lines 77-82 implement the same ordered composition: sample/accept two values in [-1, 1], scale each by pi, build X- and Y-axis quaternions, then return quat_mul(x_rotation, y_rotation). The manager reset calls the first helper at line 119, while ReorientDirectEnv calls the second at lines 271 and 300, so the supposedly shared Direct/manager reset contract has two math implementations that can drift. Keep random_xy_rotation() only as the sampling wrapper and return randomize_rotation(random_values[:, 0], random_values[:, 1], x_unit, y_unit) from it. Add a deterministic helper-level assertion using fixed samples to lock the order and (x, y, z, w) convention.
AntoineRichard
requested changes
Jul 28, 2026
AntoineRichard
left a comment
AntoineRichard
left a comment
Collaborator
There was a problem hiding this comment.
[AI Review]
Requesting changes for the inline correctness, API-compatibility, comment-quality, and duplication findings.
One additional blocking issue cannot be attached inline because the affected line is unchanged: scripts/demos/hands.py:53 still imports the deleted shadow_hand_env_cfg module, so the demo now fails during import. Please update the call site and keep the old module as a deprecated forwarding shim.
PR hygiene: the pinned exact-change range ends at af259c0, while the current head is f1afcc6; the docs-only bot summary therefore misses the latest runtime change. Please refresh the review map.
Verification performed: full pre-commit passed; 24 CPU unit tests and 108 kitless registered-task config tests passed. Camera runtime tests were unavailable because Isaac Sim is not installed in the review environment.
| physx_cfg = SHADOW_HAND_CFG.replace(prim_path=prim_path).replace( | ||
| init_state=ArticulationCfg.InitialStateCfg(pos=init_pos, rot=init_rot, joint_pos={".*": 0.0}) | ||
| ) | ||
| # Newton's importer bakes the asset's native root orientation into the |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This block mixes the durable requirement (compose the task rotation with Newton’s baked root rotation) with debugging history and comparison to the previous Warp calculation. Please state only the functional constraint and required quaternion order here; move the former behavior and bit-for-bit provenance to the PR or a regression test.
| state_space = 187 + 27 # asymmetric states + vision CNN embedding | ||
|
|
||
| def __post_init__(self): | ||
| # The vision env renders through the Isaac RTX tiled camera, whose render |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This seven-line rationale is longer than the configuration it explains and narrates backend failure and benchmark history. Keep the source comment to the enforceable functional constraint—for example, “RTX tiled cameras require Fabric cloning; default camera tasks to the PhysX/Fabric scene.”—and move the failure details and benchmark-path provenance to the PR or documentation. The code must also establish clone_in_fabric=True, as noted in the correctness thread above.
|
|
||
|
|
||
| @configclass | ||
| class ReorientObjectEnvCfg(ManagerBasedRLEnvCfg): |
Collaborator
There was a problem hiding this comment.
[AI Review][Important] This is the concrete public-class deletion covered by the compatibility finding. ReorientObjectEnvCfg was importable from isaaclab_tasks.core.reorient.reorient_manager_env_cfg; deleting the whole module makes downstream imports fail before users can migrate. Keep this module and class as a deprecated forwarding compatibility layer for at least one release, point it to the supported replacement where semantics permit, and add an import-level regression test for the legacy path.
| # | ||
| # SPDX-License-Identifier: BSD-3-Clause | ||
|
|
||
| """Shadow Hand identity shared by the Direct and manager-based reorientation tasks. |
Collaborator
There was a problem hiding this comment.
[AI Review][Important] Moving these definitions from shadow_hand_env_cfg.py to shadow_hand_common.py removes the old import path without a deprecation window. This already breaks scripts/demos/hands.py:53, and skills/user/use-sensors-actuators/examples.md:58 still directs users to the deleted file. Update those references now and retain shadow_hand_env_cfg.py as a deprecated forwarding module that re-exports the moved public names; add a test that imports ShadowHandRobotCfg through the legacy path.
| def __post_init__(self): | ||
| self.decimation = 2 | ||
| self.episode_length_s = 7.5 | ||
| # simulation — mirrors the Direct cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This is the second copy of the handover simulation settings named in the Direct-config thread. The duplicated contract is decimation=2, episode_length_s=7.5, dt=1/120, render_interval=2, friction (1.0, 1.0), and PhysicsCfg(), but no parity test exists in source/isaaclab_tasks/test. Move these task-defining values to one shared configuration source consumed by both workflows, or add a parameterized test that resolves both registered task configs and compares each field. Until that safeguard exists, remove the false “guarded” claim.
| obs_type = "full" | ||
| # simulation | ||
|
|
||
| # simulation — values mirrored by the manager cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This comment claims a safeguard that is not present: there is no Direct/manager value-parity test under source/isaaclab_tasks/test. The same Allegro values are assigned independently again in AllegroCubeEnvCfg.__post_init__ (allegro_hand_manager_env_cfg.py:295-302), including decimation, episode length, timestep, render interval, friction, and PhysicsCfg. Put these values in a shared task-parameter config used by both classes, or add a parameterized registered-config contract test that compares every duplicated field; otherwise this comment will conceal drift instead of preventing it.
| def __post_init__(self): | ||
| self.decimation = 4 | ||
| self.episode_length_s = 10.0 | ||
| # simulation — mirrors the Direct cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This is the manager-side duplicate referenced in the Direct-config thread, and it is not guarded by any test. These assignments independently reproduce AllegroHandEnvCfg.sim plus its control timing, so changing either workflow can silently change training dynamics relative to the other. Consume a shared parameter object here, or add an explicit contract test covering decimation, episode_length_s, sim.dt, sim.render_interval, both friction coefficients, and the resolved physics preset. Remove the “guarded” wording unless that test is added in this PR.
| asymmetric_obs = False | ||
| obs_type = "full" | ||
|
|
||
| # simulation — values mirrored by the manager cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] No value-parity test guards this state-task configuration. ShadowHandManagerEnvCfg.__post_init__ independently repeats the control and simulation values at shadow_hand_manager_env_cfg.py:218-225, while scene/backend defaults are declared through a separate preset. A future one-line tuning change can therefore make Direct and manager tasks diverge while this comment still promises parity. Share the task-defining values, or add a registered-config contract test covering timing, material, physics, and backend-resolved scene defaults; remove the unsupported test claim otherwise.
| def __post_init__(self): | ||
| self.decimation = 2 | ||
| self.episode_length_s = 10.0 | ||
| # simulation — mirrors the Direct cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This is the manager-side copy of ShadowHandEnvCfg timing and simulation state, but the asserted parity test does not exist. Please eliminate the independent assignments through a shared task-parameter source, or add a contract test that resolves Isaac-Reorient-Cube-Shadow and Isaac-Reorient-Cube-Shadow-Direct and compares decimation, episode duration, timestep, render interval, material coefficients, physics backend, and scene-cloning choice. Without one of those safeguards, delete the misleading “guarded” claim.
| asymmetric_obs = True | ||
| obs_type = "openai" | ||
|
|
||
| # simulation — values mirrored by the manager cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This OpenAI variant makes a second unsupported parity claim. Its decimation=3, episode_length_s=8.0, dt=1/60, render interval, material, and physics values are repeated in ShadowHandOpenAIManagerEnvCfg.__post_init__ (shadow_hand_openai_manager_env_cfg.py:222-229), but no test compares the resolved configs. Share these parameters or add a contract test for the FF/LSTM Direct and manager task IDs; otherwise remove the factual claim that a value-parity test guards them.
| def __post_init__(self): | ||
| self.decimation = 3 | ||
| self.episode_length_s = 8.0 | ||
| # simulation — mirrors the Direct cfg (guarded by the value-parity test) |
Collaborator
There was a problem hiding this comment.
[AI Review][Moderate] This is the manager-side copy for the OpenAI FF/LSTM family, and it is not guarded by a parity test. Because both task IDs share this class while the Direct variants use ShadowHandOpenAIEnvCfg, drift here changes the control frequency and physics seen by one workflow only. Centralize the six repeated timing/simulation fields, or add a parameterized test over both FF and LSTM Direct/manager registrations that compares their fully resolved configs. Remove the “guarded” wording if the test is not part of this change.
| @@ -6,6 +6,7 @@ | |||
|
|
|||
Collaborator
There was a problem hiding this comment.
[AI Review][Important] This new runtime module replaces the public shadow_hand_camera_env.py import path, but the old module is removed immediately. Existing code importing ShadowHandCameraEnv from that path will fail even though the class itself still exists here. Retain shadow_hand_camera_env.py as a deprecated forwarding module that re-exports ShadowHandCameraEnv from this module (and preserves the separately documented compute_keypoints deprecation), then add a legacy-import test so the shim survives future cleanup.
| # | ||
| # SPDX-License-Identifier: BSD-3-Clause | ||
|
|
||
| from __future__ import annotations |
Collaborator
There was a problem hiding this comment.
[AI Review][Important] This configuration module replaces shadow_hand_camera_env_cfg.py, but deleting the old path breaks imports of ShadowHandTiledCameraCfg, ShadowHandCameraEnvCfg, and the benchmark/play configs before any deprecation period. Keep shadow_hand_camera_env_cfg.py as a deprecated forwarding module that re-exports the corresponding names from this Direct-camera module, and test at least the legacy module import plus construction of ShadowHandCameraEnvCfg. The new manager module can remain additive; it does not justify removing the established Direct path.
| class EpisodeErrorRecorder: | ||
| """Record the minimum physical error reached in each episode. | ||
|
|
||
| def success_bonus( |
Collaborator
There was a problem hiding this comment.
[AI Review][Important] These are the actual public reward-term removals summarized in the changelog: success_bonus here, track_pos_l2 at line 51, and track_orientation_inv_l2 at line 75. They were exported through the package’s lazy MDP namespace, so external manager configs can reference them directly and will now fail at import/config resolution. Preserve the three callables with their existing signatures during the deprecation window—wrapping a new equivalent only where semantics are genuinely identical—and add import/call compatibility tests. Migration guidance to ReorientReward is not sufficient because that class does not preserve these function signatures.
| from .commands import ReorientCommand | ||
|
|
||
|
|
||
| def max_consecutive_success(env: ManagerBasedRLEnv, num_success: int, command_name: str) -> torch.Tensor: |
Collaborator
There was a problem hiding this comment.
[AI Review][Important] These are the concrete termination-term removals behind the changelog entry: max_consecutive_success here, object_away_from_goal at line 35, and object_away_from_robot at line 63. All three were reachable from isaaclab_tasks.core.reorient.mdp; deleting them breaks downstream manager configurations before migration code can run. Retain deprecated compatibility callables with the old names and signatures for the required release window, document the exact replacement for each function separately, and add a legacy-import/config-resolution test. A single pointer to ReorientTimeout does not demonstrate equivalent behavior for all three contracts.
hujc7
added a commit
that referenced
this pull request
Jul 29, 2026
… and enable handover Direct RSL-RL (#6414) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. This PR (#6414) is now rebased directly onto `develop`, so its own **Files changed** tab is its contribution. - Links pin specific SHAs and can go stale after a branch update — the table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |  | — | merged | | #6412 Part 2/9: OVPhysX articulation + manager runtime |  | — | merged | | #6413 Part 3/9: Reorient Direct, torch |  | — | merged | | 📌 #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct (this PR) |  | — (on develop) | Files changed tab | | #6418 Part 5/9: Reorient manager counterparts |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | #6421 Part 6/9: Handover + camera manager counterparts |  | #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | #6410 Part 7/9: Environment overview docs |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6415 Part 8/9: Benchmark success-rate utilities |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 9/9: Warp variants → experimental (draft; merges last) |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary Converts the two-hand Shadow Hand **handover** task from multi-agent (MARL) to **single-agent** (enabling RSL-RL training), fixes the underlying MARL→single-agent observation bridge, and consolidates the shared Shadow Hand definitions into the asset. - **MARL → single-agent bridge (general, broader than this task).** Single-agent RL libraries train `DirectMARLEnv` tasks via `multi_agent_to_single_agent`; the bridge dropped the latest observations from the public buffer. Fixed generally — RSL-RL observations read from the env-owned `obs_buf`, stored by `reset` like `step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the experimental warp base). Every MARL task + single-agent runner benefits; handover is the first consumer. - **Handover Direct → single-agent + RSL-RL.** Single-agent conversion, RSL-RL runner config, shared identity in `handover_common`, success-rate reward metrics, and a fix for its Newton construction failure (see below). - **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp) robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside `SHADOW_HAND_CFG`); reorient and handover consume it as **siblings** (neither imports the other). Reorient uses the default gains; handover overrides only the finger stiffness/damping to `20/2` for its catch. - **Shadow camera cleanup.** Renamed the camera Direct modules to `shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the shared, unit-tested `reorient/mdp/observations.py` (replacing the per-env `compute_keypoints` shim); and added OVPhysX physics presets to the handover and camera Direct environments. The camera env config is otherwise unchanged from develop. ### Behavior changes **Reorient-on-Newton is unchanged.** The Newton hand definition moves **verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions, effort limits, gains, friction and armature — so its config diff vs `develop` is purely the move, and the Direct env config has no diff at all. Handover changes in three ways: - Its **default physics backend moves from PhysX to Newton** (MJWarp), matching the reorientation tasks. Pass `physics=physx` for the previous backend. - Its `distal_passive` actuator group is **removed**. It targeted `robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the environment raised `No joints found for actuator group`. The `fingers` group already drives those joints, and the comment justifying the group (USD-baked `stiffness=286/damping=57`) does not apply to this asset, which authors no drive stiffness at all. - Two orientation fixes: the goal orientation was initialized to a 180-degree rotation instead of identity, and the Newton root rotation replaced the asset's baked rotation instead of composing with it, leaving both palms rotated 90 degrees. Two pre-existing discrepancies were found while verifying against the asset and are left for a follow-up, since each is a behavior change needing its own validation: the config sets `armature=2e-3` where the asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by a fixed tendon that the MJWarp solver currently skips — the actuator gains are what hold that pair together today, which is why the `J4` knuckle-abduction joints cannot simply be added to the driven set. ## Notes - Rebased onto **latest develop**; clean 3-commit history: `Add single-agent flattening for MARL Direct environments` / `Convert Shadow Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow reorient/camera tasks and consolidate the Newton hand`. ## Validation - Kit-free suite green: handover, keypoint math, the MARL adapter, and new checks pinning the RSL-RL wrapper observation contract (verified to fail against the previous implementation). - Camera Direct reaches training takeoff under `rsl_rl` (frame-verified videos); its env config now matches develop's established vision env. - **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500 iterations, 4 runs per config): this branch spans 524–664 reward, `develop` spans 654–822 — overlapping distributions whose per-config spreads (140–168) exceed the gap between them. The move is additionally verified at the config level: the Newton robot configuration is character-for-character identical to `develop`'s. - **Handover-on-Newton** trains end-to-end on this branch (reward 1314), which the `develop` config cannot do at all — it fails during construction. Full evidence: #6324.
matthewtrepte
pushed a commit
to matthewtrepte/IsaacLab
that referenced
this pull request
Aug 4, 2026
…nager runtime (isaac-sim#6412) ## Summary - Fixes OVPhysX actuator joint indices to follow the common actuator indexing contract. - Fixes OVPhysX initialization alongside Kit by reusing Kit's registered PhysX schema provider. - Fixes the OVPhysX manager to support both the declared public runtime API and the current runtime API. - Regression tests included. Validated by full dexterous training runs on the OVPhysX backend; split out of the lumped validation branch isaac-sim#6324 (Part 2 of 11). ## Dependencies - None. ## Series review map Full integrated diff + training/validation evidence: the lumped validation PR isaac-sim#6324 (DO-NOT-MERGE). | Part | PR | |---|---| | Docs: regenerate the environment overview table | isaac-sim#6410 | | Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz teardown) | isaac-sim#6411 | | **Part 2/11: OVPhysX runtime fixes (this PR)** | isaac-sim#6412 | | Part 3/11: success-rate metrics for the Direct reorientation tasks | isaac-sim#6413 | | Part 4/11: RSL-RL training for the handover Direct task | isaac-sim#6414 | | Part 5/11: success-rate support in the benchmark utilities | isaac-sim#6415 | | Part 6/11: renderer presets for the Direct camera task | isaac-sim#6416 | | Part 7/11: OVPhysX presets for the dexterous tasks | isaac-sim#6417 | | Part 8/11: Allegro manager counterpart | isaac-sim#6418 | | Part 9/11: Shadow + OpenAI manager counterparts | isaac-sim#6419 | | Part 10/11: Shadow camera manager counterpart | isaac-sim#6420 | | Part 11/11: Shadow handover manager counterpart | isaac-sim#6421 | --- ### Exact changes in this PR - OVPhysX backend changes + tests: isaac-sim@1f7a433
matthewtrepte
pushed a commit
to matthewtrepte/IsaacLab
that referenced
this pull request
Aug 4, 2026
…isualizer teardown (isaac-sim#6411) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — the table on isaac-sim#6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | isaac-sim#6410 [Docs] Environment overview regen |  | — | — | | 📌 isaac-sim#6411 Part 1/8: Newton cloner/cubric/visualizer fixes (this PR) |  | — | — | | isaac-sim#6412 Part 2/8: OVPhysX articulation + manager runtime |  | — | — | | isaac-sim#6413 Part 3/8: Reorient Direct, torch |  | — | — | | isaac-sim#6414 Part 4/8: MARL-to-single-agent fix + handover/camera Direct |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/6e8a63e4e028b2d43676ea30c446b9dc9068c7b5..5cb00e7cb5cc813b202521272e043007cd255194) | | isaac-sim#6418 Part 5/8: Reorient manager counterparts |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/79f87501ac4c81de93a71dab00dc443da62113aa..e7c9a9a3fae3a7972b0c5165ae683abffb7d0e0f) | | isaac-sim#6421 Part 6/8: Handover + camera manager counterparts |  | isaac-sim#6413, isaac-sim#6414, isaac-sim#6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/01c9f4d8c5c35a5688b2a5bb90209e16b8f81b99..835a5815ec49b11aada1d20a76c177054505e6e7) | | isaac-sim#6415 Part 7/8: Benchmark success-rate utilities + docs |  | isaac-sim#6413, isaac-sim#6414, isaac-sim#6418, isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/e1abb6b1790ccc57af42551eebccf743633f1f13..d6348539aa8032d9668c20a8fea462c5d88d3af9) | | isaac-sim#6582 Part 8/8: Warp variants → experimental (draft; merges last) |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/21dbb1769c4e30c8e9e5b0f563c2dae24c230349..83e1587cadd9712a60615ed2a3cb2d177c2ac24d) | | isaac-sim#6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary - Fixes Newton cloner label rows, the cubric IAdapter version audit (exact-match fallback to the CPU hierarchy path), and visualizer teardown. - Retains an in-tree `ignore_paths` workaround for custom-frequency USD traversal; it becomes redundant once the Newton pin advance (isaac-sim#6584) merges — this PR then only needs a rebase. ## Stacking - Independent; based on `develop`. ## Review history - Approved. The Newton pin + MuJoCo overrides were split out to isaac-sim#6584 via revert commits (2026-07-17) so this PR's CI runs against develop's pins.
matthewtrepte
pushed a commit
to matthewtrepte/IsaacLab
that referenced
this pull request
Aug 4, 2026
…eorientation Direct tasks (isaac-sim#6413) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — the table on isaac-sim#6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | isaac-sim#6411 Part 1/9: Newton cloner/cubric/visualizer fixes |  | — | merged | | isaac-sim#6412 Part 2/9: OVPhysX articulation + manager runtime |  | — | merged | | 📌 isaac-sim#6413 Part 3/9: Reorient Direct, torch (this PR) |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71) | | isaac-sim#6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8) | | isaac-sim#6418 Part 5/9: Reorient manager counterparts |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | isaac-sim#6421 Part 6/9: Handover + camera manager counterparts |  | isaac-sim#6413, isaac-sim#6414, isaac-sim#6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | isaac-sim#6410 Part 7/9: Environment overview docs |  | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | isaac-sim#6415 Part 8/9: Benchmark success-rate utilities |  | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | isaac-sim#6582 Part 9/9: Warp variants → experimental (draft; merges last) |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | isaac-sim#6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary - Adds a behavioral `Metrics/success_rate` signal (goal-reach streaks per episode) and threshold-independent episode orientation-error diagnostics to the Direct reorientation environments; success gates task health, reward stays diagnostic. - Task logic is **torch-first** per the mainline convention (plain torch buffers; `.torch` accessors only at the core-lib boundary). This supersedes the earlier warp-first revision of this PR; the warp implementation moved to `isaaclab_tasks_experimental` (isaac-sim#6582). - Lands shared helpers in `isaaclab_tasks.core.utils` (`EpisodeErrorRecorder`, `sample_joint_positions_within_limits`) with torch math tests; fixes hand resets below lower joint limits. ## Stacking - First stack PR; based on `develop`. ## Validation - Kit-free torch math + utils tests green; Direct state training on Newton: success rate 0.85–1.00, reward ≈3600–3900 at 2000 iterations, frame-verified policy video. Full integrated evidence: isaac-sim#6324. ## Review history - The earlier warp-era review rounds on this PR are superseded by the torch restack (2026-07-16); the warp implementation and its review-polished kernels live on in isaac-sim#6582.
matthewtrepte
pushed a commit
to matthewtrepte/IsaacLab
that referenced
this pull request
Aug 4, 2026
… and enable handover Direct RSL-RL (isaac-sim#6414) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. This PR (isaac-sim#6414) is now rebased directly onto `develop`, so its own **Files changed** tab is its contribution. - Links pin specific SHAs and can go stale after a branch update — the table on isaac-sim#6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | isaac-sim#6411 Part 1/9: Newton cloner/cubric/visualizer fixes |  | — | merged | | isaac-sim#6412 Part 2/9: OVPhysX articulation + manager runtime |  | — | merged | | isaac-sim#6413 Part 3/9: Reorient Direct, torch |  | — | merged | | 📌 isaac-sim#6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct (this PR) |  | — (on develop) | Files changed tab | | isaac-sim#6418 Part 5/9: Reorient manager counterparts |  | isaac-sim#6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | isaac-sim#6421 Part 6/9: Handover + camera manager counterparts |  | isaac-sim#6414, isaac-sim#6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | isaac-sim#6410 Part 7/9: Environment overview docs |  | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | isaac-sim#6415 Part 8/9: Benchmark success-rate utilities |  | isaac-sim#6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | isaac-sim#6582 Part 9/9: Warp variants → experimental (draft; merges last) |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | isaac-sim#6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary Converts the two-hand Shadow Hand **handover** task from multi-agent (MARL) to **single-agent** (enabling RSL-RL training), fixes the underlying MARL→single-agent observation bridge, and consolidates the shared Shadow Hand definitions into the asset. - **MARL → single-agent bridge (general, broader than this task).** Single-agent RL libraries train `DirectMARLEnv` tasks via `multi_agent_to_single_agent`; the bridge dropped the latest observations from the public buffer. Fixed generally — RSL-RL observations read from the env-owned `obs_buf`, stored by `reset` like `step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the experimental warp base). Every MARL task + single-agent runner benefits; handover is the first consumer. - **Handover Direct → single-agent + RSL-RL.** Single-agent conversion, RSL-RL runner config, shared identity in `handover_common`, success-rate reward metrics, and a fix for its Newton construction failure (see below). - **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp) robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside `SHADOW_HAND_CFG`); reorient and handover consume it as **siblings** (neither imports the other). Reorient uses the default gains; handover overrides only the finger stiffness/damping to `20/2` for its catch. - **Shadow camera cleanup.** Renamed the camera Direct modules to `shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the shared, unit-tested `reorient/mdp/observations.py` (replacing the per-env `compute_keypoints` shim); and added OVPhysX physics presets to the handover and camera Direct environments. The camera env config is otherwise unchanged from develop. ### Behavior changes **Reorient-on-Newton is unchanged.** The Newton hand definition moves **verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions, effort limits, gains, friction and armature — so its config diff vs `develop` is purely the move, and the Direct env config has no diff at all. Handover changes in three ways: - Its **default physics backend moves from PhysX to Newton** (MJWarp), matching the reorientation tasks. Pass `physics=physx` for the previous backend. - Its `distal_passive` actuator group is **removed**. It targeted `robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the environment raised `No joints found for actuator group`. The `fingers` group already drives those joints, and the comment justifying the group (USD-baked `stiffness=286/damping=57`) does not apply to this asset, which authors no drive stiffness at all. - Two orientation fixes: the goal orientation was initialized to a 180-degree rotation instead of identity, and the Newton root rotation replaced the asset's baked rotation instead of composing with it, leaving both palms rotated 90 degrees. Two pre-existing discrepancies were found while verifying against the asset and are left for a follow-up, since each is a behavior change needing its own validation: the config sets `armature=2e-3` where the asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by a fixed tendon that the MJWarp solver currently skips — the actuator gains are what hold that pair together today, which is why the `J4` knuckle-abduction joints cannot simply be added to the driven set. ## Notes - Rebased onto **latest develop**; clean 3-commit history: `Add single-agent flattening for MARL Direct environments` / `Convert Shadow Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow reorient/camera tasks and consolidate the Newton hand`. ## Validation - Kit-free suite green: handover, keypoint math, the MARL adapter, and new checks pinning the RSL-RL wrapper observation contract (verified to fail against the previous implementation). - Camera Direct reaches training takeoff under `rsl_rl` (frame-verified videos); its env config now matches develop's established vision env. - **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500 iterations, 4 runs per config): this branch spans 524–664 reward, `develop` spans 654–822 — overlapping distributions whose per-config spreads (140–168) exceed the gap between them. The move is additionally verified at the config level: the Newton robot configuration is character-for-character identical to `develop`'s. - **Handover-on-Newton** trains end-to-end on this branch (reward 1314), which the `develop` config cannot do at all — it fails during construction. Full evidence: isaac-sim#6324.
hujc7
changed the title
hujc7
added a commit
that referenced
this pull request
Aug 11, 2026
…anager counterparts (#6418) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — this table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6411 Part 1/10: Newton cloner/cubric/visualizer fixes |  | — | merged | | #6412 Part 2/10: OVPhysX articulation + manager runtime |  | — | merged | | #6413 Part 3/10: Reorient Direct, torch |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71) | | #6414 Part 4/10: MARL-to-single-agent fix + handover/camera Direct |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8) | | 📌 #6418 Part 5/10: Reorient manager counterparts |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/3ea6f7bbf6c..0271783ea17) | | #6421 Part 6/10: Handover + camera manager counterparts |  | #6413, #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/b31b04eaf71..0cae3d87be4) | | #6410 Part 7/10: Environment overview docs |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6954 Part 8/10: OpenAI Shadow variants → contrib (draft) |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6954/changes/0cae3d87be4..b0d2d8448af) | | #6415 Part 9/10: Benchmark success-rate utilities |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 10/10: Warp variants → experimental (draft; merges last) |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary Adds manager-based counterparts for the Shadow cube reorientation task and aligns the existing Allegro manager task with the Direct contracts. - **3 manager tasks** mirror their Direct counterparts: `Isaac-Reorient-Cube-Shadow`, `-Shadow-Camera` and `-Allegro`. - The OpenAI variants move to the contributed tasks **unchanged**. They encode a paper-specific training regime, so they get no manager counterpart. - `Metrics/success_rate` now means the same thing in both workflows. - The in-hand cube is one object per hand instead of one per physics backend. ## 1. Manager tasks match the Direct contracts Observations, actions, rewards, terminations, reset distributions and timing were brought to the Direct values. A value-parity test asserts the decimation / episode length / simulation step triple, the orientation success tolerance and the fall distance, so drift on either side fails CI. The Allegro observation space changes size, so existing manager checkpoints must be retrained. Domain randomization becomes opt-in through `presets=randomized`, replacing a configuration-file flag that the command line could not override. ## 2. OpenAI variants move to the contributed tasks `Isaac-Reorient-Cube-Shadow-OpenAI-{FF,LSTM}-Direct` become `IsaacContrib-Reorient-Cube-Shadow-OpenAI-{FF,LSTM}-Direct`. Both identifiers are kept: they share one environment configuration and differ only in their agent configuration, so collapsing them would leave the recurrent policy unreachable — the training entry point selects an agent configuration by workflow, not by name. ## 3. One success rate, defined once `Metrics/success_rate` named two quantities. The manager tasks reported goals reached over goals presented; the Direct tasks reported a per-episode success bit drawn at `success_count_threshold`. The curves were not comparable despite sharing a name and measuring the same task. Both now report the per-attempt rate, and `success_count_threshold` is removed as unused. The goal accounting moves into a `SuccessTracker` beside `EpisodeErrorRecorder`, which also suppresses success detection for the single step after an automatic reset — where the goal has been drawn but no physics has run against it, so a reset pose inside the success threshold would otherwise bank a goal the policy never earned. ## 4. One in-hand cube per hand, not per backend The cube was a `PresetCfg` with a variant per physics backend, so each hand manipulated a different object depending on the backend. On Shadow the branches disagreed on size and spawn height: PhysX spawned a 60 mm rigid body at `z=0.6`, Newton a 54 mm **articulation with no joints** at `z=0.535`. A policy trained on one backend was not solving the same task as one trained on the other. Each hand now declares a single `CUBE_CFG`. Shadow keeps the PhysX values; Allegro keeps its 1.2 scale. The `mass_props` density overrides are dropped: the asset authors an absolute `physics:mass` of 0.216 kg, which takes precedence over density, so they never applied on either backend. ## 5. Configuration hierarchy flattened, classes named by role Every manager task overrode all seven of the shared base's sub-configurations, so the base carried pre-alignment defaults that each task then undid. The environment configurations now derive from `ManagerBasedRLEnvCfg` directly. Removed: `ReorientObjectEnvCfg`, the shared sub-configurations no task constructed, and `reorient_common`. `isaaclab_tasks.core.utils` moves to `isaaclab_tasks.core.reorient.utils`, which the hand-over task already imports from. One naming convention now applies throughout, since `SceneCfg`, `EventCfg` and `ObservationsCfg` are each defined in dozens of files and gave no clue which base a task was specializing. Classes shared across files carry the task prefix; the `Base` suffix marks only those a subclass must complete by supplying fields the parent leaves unset, so the observation halves are named as parts rather than templates. Classes nested inside another configuration drop the prefix. Registered and preset-selectable configurations always name their task and hold fields only, with their helpers declared beside them. ## Validation Eight arms, 1500 iterations, 8192 environments, `physics=newton_mjwarp`, one L40 each. All exited cleanly with no regression in reward. | Arm | reward before | reward after | |---|---|---| | `Isaac-Reorient-Cube-Shadow` | 67.08 | 66.24 | | `-Shadow` `presets=asymmetric` | 42.08 | 52.95 | | `-Shadow-Direct` | 3532.95 | 3967.58 | | `-Allegro` | 25.22 | 34.03 | | `-Allegro` `presets=randomized` | 16.03 | 39.59 | | `-Allegro-Direct` | 656.78 | 1003.11 | | `IsaacContrib-…-OpenAI-FF-Direct` | 2757.56 | 4533.21 | | `IsaacContrib-…-OpenAI-LSTM-Direct` | — | 2585.54 | The LSTM arm resolves `ShadowHandAsymLSTMPPORunnerCfg`, confirming the recurrent policy is reachable through its own identifier. ### Known failure `test_rendering_shadow_hand[newton-isaacsim_rtx-rgb]` fails; it passes on `develop`. Only the `rgb` modality is affected — depth, normals and all three segmentations pass on the same backend, so the cube move is not an obvious explanation. Root cause is not yet established.
hujc7
added a commit
that referenced
this pull request
Aug 17, 2026
…mera manager counterparts (#6421) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — this table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | 📌 #6421 Part 6/10: Handover + camera manager counterparts |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/b31b04eaf71..0cae3d87be4) | | #6410 Part 7/10: Environment overview docs |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6415 Part 9/10: Benchmark success-rate utilities |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 10/10: Warp variants → experimental (draft; merges last) |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary Completes manager coverage of the dexterous task families by adding manager-based counterparts for the Shadow handover and Shadow camera reorientation tasks. - **2 manager tasks** added: `Isaac-Shadow-Handover` and `Isaac-Reorient-Cube-Shadow-Camera`. - **The last 2 `-Play` registrations** in `isaaclab_tasks` are removed. ## 1. Handover manager counterpart The fused handover reward became a plain reward term, with success and goal-distance bookkeeping moved to `HandoverCommand`, which owns the goal. It reports `Metrics/success_rate` as a per-episode bit, matching the handover Direct environment. `HandoverCommand.command` builds its pose per call rather than returning a persistent buffer, so consumers that store it across steps are not aliased to live data. The task keeps no observation, reset or event code of its own. Every observation is a framework term or a reorientation one: fingertip pose is `body_pose_w`, the object block is the root pose, root velocities, the generated command and the reorientation goal-quaternion difference, and only the fingertip velocity stays task-local because the framework has no `body_vel_w`. Both hands reset through the reorientation hand reset, one term per hand. The object and goal block was previously observed once per agent with identical values, so a single copy now carries the same information. `presets=randomized` adds the domain-randomization terms to the reset, matching how the reorientation manager tasks expose theirs. The reset and action observations use the framework terms rather than task-local ones, following Part 5: the object resets through `reset_root_state_with_random_orientation` and both hands observe `last_action`, leaving only the two-hand joint and PD-target reset task-local. This samples the object's reset orientation uniformly over SO(3) where the Direct task composes X/Y rotations, and reports zero actions on the reset step where the Direct task reports the pre-reset action. ## 2. Camera manager counterpart The camera task runs on PhysX by default: the RTX render modalities require Fabric cloning, which Newton does not support, so the inherited default could not render. Newton remains selectable with `physics=newton_mjwarp` for the state-only observation groups. The feature-extractor observation term ignored its declared `feature_extractor_cfg` parameter and read the environment configuration instead; it now honours the parameter. ## 3. Camera playback moves to `play_mode` `Isaac-Reorient-Cube-Shadow-Camera-Play` and `-Camera-Direct-Play` were the last `-Play` registrations left in `isaaclab_tasks`; every other task moved to `play_mode` in #6676 — Remove the _PLAY configs. Their playback environment count and inference feature extractor now live in `play_mode` overrides on the manager and Direct camera configurations. The override mutates the feature extractor rather than replacing it. Replacing the object resets every field the caller does not name, which silently re-enabled the CNN in `ShadowHandCameraBenchmarkEnvCfg` — a configuration whose entire purpose is to disable it. ## 4. Reorientation action configuration The action term is named through a module path, so loading a task configuration no longer imports the USD bindings. ## Validation Trained at 4096 environments, 1500 iterations, seed 42, on one L40 (`jichuanh-autorun-8gpu`). Success rate is logged per reset batch, so the figures below are the mean and range over the final eight batches rather than a single value. | Task | Backend | Success rate | Range | |---|---|---|---| | `Isaac-Shadow-Handover` (manager) | `newton_mjwarp` | 0.756 | 0.745 - 0.779 | | `Isaac-Shadow-Handover-Direct` (reference) | `newton_mjwarp` | 0.697 | 0.563 - 0.875 | | `Isaac-Shadow-Handover` (manager) | `isaacsim_physx` | 0.999 | 0.992 - 1.000 | The manager task matches the Direct reference it replaces on the same backend. The spread between backends is a pre-existing property of the Newton Shadow Hand actuation, not something this pull request introduces; it is tracked separately. `Isaac-Reorient-Cube-Shadow-Camera` was smoke-tested rather than trained to convergence: 300 iterations at 256 environments through the Kit renderer completes cleanly. Its rendering matrix and golden images are covered separately. `presets=randomized` converges more slowly than the default and does not plateau within 1500 iterations. A longer run is in progress to set an appropriate iteration budget.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review Map