Skip to content

[Backport release/3.0.0] Complete task, demo, and benchmark backports - #7852

Closed
kellyguo11 wants to merge 14 commits into
isaac-sim:release/3.0.0from
kellyguo11:codex/backport-missing-tasks-demos-release-3.0.0
Closed

kellyguo11 wants to merge 14 commits into
isaac-sim:release/3.0.0from
kellyguo11:codex/backport-missing-tasks-demos-release-3.0.0

Conversation

@kellyguo11

@kellyguo11 kellyguo11 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Completes the task, demo, benchmark, and Newton task changes from develop that were not fully present on release/3.0.0.

Backported source PRs: #7333, #7375, #7433, #7607, #7623, #7639, #7756, #7702, and #7520.

The heterogeneous multitask manipulation backport from #7375 was previously incomplete: five configuration files were present on the release branch, while its task modules, registrations, assets, and tests were missing. This PR restores the complete feature and groups the remaining related task/demo changes with it.

Validation

  • 19 selected kitless task and benchmark tests passed
  • changed-file pre-commit suite passed (Git LFS pointer hook skipped because git-lfs is unavailable locally; new images are ordinary, non-LFS blobs)
  • changelog fragment validation passed
  • git diff --check

One selected Newton test requires the unavailable newton package on macOS, and Isaac Sim-dependent collection cannot run on this host. CI remains authoritative for simulator-dependent coverage.

Full backport re-audit

The five multitask configuration modules now use the public isaaclab.utils.configclass export, completing #7646 for files that arrived after its original release backport.

The later #7375 backport also reintroduced multitask_manipulation.jpg after #7822 had removed repository-hosted task previews in favor of the S3 image base. The stale JPEG is removed again so #7822 remains fully effective after these PRs are combined.

Ruff, formatting, Python compilation, the S3 preview contract, and git diff --check pass for the follow-up.

Latest develop sync (2026-09-17)

Backports #7866 and #7870 exactly: Franka Pour, Lift, and Reorient explicitly select the convex-hull collider variants required by their task/reset contracts. Both cherry-picked patch IDs match develop exactly. The three focused config regressions pass, as do Ruff, formatting, aggregate git diff --check, and combined-tree provenance checks.

rebeccazhang0707 and others added 9 commits September 16, 2026 16:48
# Description

This PR reduces the number of physics views created by
`scripts/demos/bin_packing.py`. It keeps the existing scene
configuration, clone combinations, and randomized layout flow, but
changes each grocery slot to a spawn-only `AssetBaseCfg` and manages all
spawned groceries through one flat PhysX `RigidBodyView`.

This is a demo-only performance optimization and adds no dependencies.

## Type of change

- Performance improvement (non-breaking change)

## Testing

- `uv run isaaclab -f`
- `uv run python -m py_compile scripts/demos/bin_packing.py`
- `uv run isaaclab -p scripts/demos/bin_packing.py --num_envs 2
--visualizer none`

The 2-environment PhysX smoke test completed scene setup, created one
flat grocery view, and ran without errors.

## Screenshots

Not applicable; the rendered scene and randomized layouts are unchanged.

## Checklist

- [x] I have read and understood the contribution guidelines.
- [x] I have run the pre-commit checks with `uv run isaaclab -f`.
- [x] My changes generate no new warnings.
- [x] I have tested the updated demo with the PhysX backend.
- [x] Documentation changes are not required for this internal demo
implementation update.
- [x] A changelog fragment is not required because no source package is
changed.
- [x] My name already exists in `CONTRIBUTORS.md`.

(cherry picked from commit 9c209c0)
…c-sim#7375)

This PR adds `IsaacContrib-Multitask-Manipulation`, a contributed
manager-based environment for training and evaluating one RSL-RL policy
across three heterogeneous manipulation layouts:

- OpenArm cube lifting
- Franka drawer opening
- UR10 end-effector reaching

The environment uses selection-aware physics-view indexing so that each
asset only exists in its task subset while observations and actions
remain fixed-width in global environment order.

- Adds `SceneEntitySelectionCfg` to resolve task-local assets and map
physics-view indices into global environment order, enabling
heterogeneous scene subsets across reset, observation, action, reward,
command, and termination terms.
- Supports multitask training, task-subset evaluation, and single-task
evaluation.
- Adds a shared observation backbone with task-specific Gaussian policy
and value heads.
- Masks inactive action dimensions in sampling, entropy,
log-probability, KL, and last-action observations.
- Adds bounded targets, task-specific timeouts, lift and cabinet success
metrics, and playback visualization.

The three tasks share manipulation structure but differ in robot state,
action semantics, and reward scale. A fully shared policy with one
padded action distribution would train irrelevant action dimensions and
allow the task with the largest reward scale to dominate the update. The
design therefore shares representations where transfer is useful while
routing task-specific outputs and normalization where the contracts
differ.

| Challenge | Solution | Motivation |
| --- | --- | --- |
| Heterogeneous scene ownership | Add `SceneEntitySelectionCfg`, which
maps global environment IDs to each asset's physics-view instance rows
and scatters task-local values back into global environment order. | An
asset exists only in its task subset, so global environment indices
cannot directly index its compact physics view. Explicit mapping
prevents incorrect or out-of-range indexing and zero-fills environments
where the asset is absent. |
| Varied observation dimensions | Concatenate a task ID and zero-filled
task blocks into one 95-dimensional policy observation: `3 task ID + 28
lift + 23 cabinet + 19 reach + 22 last action`. Process it with a shared
feature extractor. | Gives RSL-RL one fixed observation contract while
allowing the backbone to reuse features across tasks. |
| Varied action dimensions | Use task-specific Gaussian action heads
with dimensions `(8, 8, 6)` and task-specific scalar value heads. Mask
inactive action dimensions in actions, log-probability, entropy, KL, and
last-action observations. | Prevents unrelated joints from contributing
policy gradients while retaining shared actor and critic backbones. |
| Varied reward scaling | Normalize PPO advantages independently for
lift, cabinet, and reach rollouts. | Prevents a task with larger reward
magnitudes from dominating the shared PPO update. |

No new dependencies are required.

Train:

```bash
uv run isaaclab train \
    --rl_library rsl_rl \
    --task IsaacContrib-Multitask-Manipulation \
    --num_envs 16384 \
    --seed 42 \
    --run_name multitask_manipulation
```

Evaluate a specific checkpoint:

```bash
uv run isaaclab play \
    --rl_library rsl_rl \
    --task IsaacContrib-Multitask-Manipulation \
    --num_envs 3 \
    --checkpoint logs/rsl_rl/multitask_manipulation/<run-dir>/model_3000.pt \
    --viz kit
```
<img width="768" height="432"
alt="multitask_manipulation_eval_compressed"
src="https://github.com/user-attachments/assets/c491e548-2eb2-4943-be75-5eb97099c964"
/>

- New feature (non-breaking change which adds functionality)
- Documentation update

![Multitask manipulation
environment](https://github.com/rebeccazhang0707/IsaacLab/blob/reb/multitask_manip_demo/docs/source/_static/tasks/manipulation/multitask_manipulation.jpg?raw=true)

- `uv run isaaclab -f`
- `uv run --extra test --frozen python -m pytest
source/isaaclab_tasks/test/contrib/test_multitask_manipulation_env_cfg.py
-q` (`18 passed`)
- `uv run --extra test --frozen python -m pytest
source/isaaclab_tasks/test/contrib/test_contrib_environments.py -q -k
IsaacContrib-Multitask-Manipulation` (`1 passed`)
- `uv run python scripts/environments/list_envs.py --keyword Multitask
--show_presets`
- `uv run --isolated --extra test -- make -C docs current-docs`

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html).
- [x] I have run the pre-commit checks.
- [x] I have made corresponding changes to the documentation.
- [x] My changes generate no new task-local warnings.
- [x] I have added tests that cover the new behavior.
- [x] I have added a changelog fragment under
`source/isaaclab_tasks/changelog.d/`.
- [x] My name already exists in `CONTRIBUTORS.md`.

(cherry picked from commit 13296b1)
…n run (isaac-sim#7433)

# Description

`DataGenConfig.max_num_failures` is documented as *"Maximum number of
failures allowed before stopping generation"* and eighteen shipped Mimic
environment configs set it to `25`. Nothing reads it.

```console
$ git grep -c max_num_failures
source/isaaclab/isaaclab/envs/mimic_env_cfg.py:1          # the definition
source/isaaclab_mimic/isaaclab_mimic/envs/*.py:18         # eighteen assignments
```

No third entry: there is no read anywhere in the repository, and the
field has been inert since Isaac Lab Mimic was introduced in isaac-sim#179.
`env_loop` counts `num_failures` and then never consults it; its only
termination is

```python
check_val = num_success if generation_guarantee else num_attempts
if check_val >= generation_num_trials:
```

With `generation_guarantee = True` (which those same eighteen configs
also set), a task whose success rate is low retries without any bound,
and the one knob that claims to stop that does nothing. On our own data
generation this cost 337 attempts for 10 demos before we discovered the
cap was not real.

### Reproduction, on a stock task

```bash
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
  --task Isaac-Stack-Cube-Franka-IK-Rel-Mimic-v0 \
  --input_file ./datasets/annotated_dataset.hdf5 \
  --output_file /tmp/out.hdf5 \
  --generation_num_trials 30 --num_envs 10 --headless
```

`FrankaCubeStackIKRelMimicEnvCfg` sets `max_num_failures = 25`.
Observed:

| | successes | failures | attempts |
|---|---|---|---|
| before | 30 | **50** | 80 |
| after (`max_num_failures = 25`) | 12 | **25** | 37 |

Before the change the run passed the configured cap and kept going,
ending at twice it. After, it stops on the attempt that reaches 25
failures and says so:

```
Reached 25 failures (max_num_failures=25) after 12/30 successes. Exiting.
```

### On the default

Wiring the field up exactly as written would abort the primary
documented workflow. At the ~36% success rate measured above,
`--generation_num_trials 1000` needs on the order of 1800 failures, so a
cap of 25 would end the run after roughly forty attempts. Since the
eighteen assignments were inert when they were written and no current
behaviour depends on them, they are removed here and the default becomes
`None`, meaning no limit. **Runs behave exactly as they do today unless
a limit is explicitly requested.**

If maintainers would rather the shipped configs keep an active cap, that
is a one-line change per config and I am happy to make it — it just
needs a value that suits large runs.

### Tests

`source/isaaclab_mimic/test/test_generation_failure_cap.py` drives
`env_loop` with a fake environment whose `step()` consumes scripted
attempt outcomes and refills the action queue, so the loop's own
termination logic runs unmodified with no simulator or dataset behind
it; a step-count fuse turns an unbounded loop into a failed assertion.
Six cases: the cap stops a run that never succeeds; `None` keeps the
guarantee unbounded as before; enough successes still end the run first;
a cap reached short of the target ends it; attempt-based termination
with the guarantee off is untouched (×2).

Against the unpatched loop the two cap-dependent cases fail (`'fuse' ==
'exited'`, and `(3, 4) == (2, 4)` — the loop overran the cap and
collected an extra success) and the four asserting unchanged behaviour
pass.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Checklist

- [x] I have read and understood the contribution guidelines
- [ ] I have run the `pre-commit` checks with `./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01JFz91VMgmrjS4XR6f9Q9Z2

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 59df9b8)
## Description

Remove backend-specific task and training changes from rough velocity
configurations so physics selection no longer silently changes the
comparison inputs.

- Remove MJWarp-only actuator armature overrides from Anymal-C, Cassie,
Go1, and Go2.
- Use the same 5,000-iteration G1 PPO budget for every physics backend.
- Replace the one-option base-COM PresetCfg with its owned EventTerm and
update the robot-specific callers.
- Remove tests whose only contract was the deleted G1 budget or Go2
armature special case. Keep the legacy Newton alias check tied to the
selected solver instead of an actuator side effect.

Downstream configurations that intentionally require different values
can still set them explicitly.

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)

## Release backport

- [ ] Backport this pull request to the active release branch after it
merges into develop

## Testing

- `uv run --extra test python -m pytest
source/isaaclab_tasks/test/core/test_hydra.py
source/isaaclab_tasks/test/core/test_velocity_newton_cfg.py` — 88
passed.
- `uv run isaaclab -f` — passed.
- No new tests were added.

## Checklist

- [x] I have read and understood the contribution guidelines.
- [x] I have run the pre-commit checks.
- [x] Documentation changes are not required.
- [x] My changes generate no new warnings.
- [x] I removed tests that encoded the deleted backend-specific tuning.
- [x] I added an isaaclab_tasks changelog fragment.
- [x] My name already exists in CONTRIBUTORS.md.

(cherry picked from commit b0c3941)
)

# Description

`test-contrib-environments` failed on isaac-sim#7610 (run
[34042243157](https://github.com/isaac-sim/IsaacLab/actions/runs/34042243157/job/101511058208))
in `test_contrib_environments[IsaacContrib-DrLegs-Walk]` with
`AssertionError: Invalid data` on the policy observation. The PR itself
only touched `source/isaaclab/test/install_ci`, so this is unrelated to
that change.

**Root cause.** The first row of the failing observation is NaN across
projected gravity, base angular velocity and joint positions, i.e. the
robot state itself is non-finite, not a single observation term.
Reproducing the test scenario locally (2 envs, 20 random-action steps,
fresh env per seed) shows the Kamino P-ADMM solver diverging
mid-episode: in 1 of 12 seeds the entire state of env 0 (root position,
root quaternion, joint positions and velocities) turns NaN at step 6.
The heading-driven velocity command then inherits the NaN through
`heading_w`, and the `root_height` / `bad_orientation` termination terms
cannot fire because every comparison against NaN is false, so the
episode is never reset. The sibling `IsaacContrib-DrLegs-HoldPose` task
(same robot, same solver preset, same random actions) stayed finite in 9
of 9 local trials and passed in the same CI job, as did the deprecated
alias id of the Walk task, which is why the failure is intermittent
rather than deterministic.

**Change.** Add `DrLegs-Walk` to `_SKIPPED_TASK_SUBSTRINGS` in
`test_contrib_environments.py` with the reason documented inline. The
substring covers both `IsaacContrib-DrLegs-Walk` and its deprecated
alias `Isaac-DrLegs-Walk-v0`, which the test also enumerates. HoldPose
keeps running. This is a test-side skip; the solver instability itself
needs to be addressed in Kamino/Newton (or by retuning the DR Legs
solver preset), and the inline comment says to re-enable once that
lands.

## Validation

- Reproduction script (launches once, rebuilds the env per seed, mirrors
`env_test_utils._run_environments`): Walk 1/12 seeds NaN at step 6 with
root pose, joint state and the velocity command all non-finite; HoldPose
0/9.
- `pytest
source/isaaclab_tasks/test/contrib/test_contrib_environments.py -k
DrLegs-Walk -rs`: both ids report `SKIPPED ... Kamino solver
intermittently produces NaN robot state under random actions.`; HoldPose
ids still collect as runnable.
- Ruff and ruff-format pass; `isaaclab_tasks` changelog `.skip` fragment
included.

## Type of change

- Test fix (non-breaking)

## Checklist

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package (do **not** edit
`CHANGELOG.rst` or bump `extension.toml` — CI handles that)
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

(cherry picked from commit 63d579e)
# Description

Use the default `GroundPlaneCfg` appearance in the granular, two-way
coupling, and snowball MPM demos. This removes per-demo size and color
overrides and keeps the examples aligned with the standard Isaac Lab
ground plane.

No new dependencies.

## Type of change

- Documentation/example cleanup (non-breaking)

## Release backport

- [ ] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Screenshots

Not applicable; this intentionally adopts the standard ground-plane
appearance.

## Checklist

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) on
all changed files
- [x] Documentation changes are not required
- [x] My changes generate no new warnings
- [x] Tests are not required for configuration-only demo cleanup
- [x] Changelog fragments are not required because no source package is
changed
- [x] My name already exists in `CONTRIBUTORS.md`

(cherry picked from commit bb3f366)
# Fixed locomanipulation NuRec SDG generation

Hugging Face NuRec assets were updated to v0.2. This PR updates
locomanip_sdg to use the v0.2 assets and fixes several issues in the
NuRec SDG workflow:

- Fixed navigation pose projection and recorder initial-state capture
after placement.
- Synced randomized static fixture poses so downstream success checks
use the placed table pose
  - Applied Isaac RTX Gaussian camera settings for NuRec camera capture.
- Automatically enabled placement randomization when a NuRec scene is
used.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Release backport

- [ ] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Screenshots

<img width="692" height="437" alt="image"
src="https://github.com/user-attachments/assets/5491a85a-cd94-4fa0-818e-66e4983cdd6c"
/>

## Checklist

Docker and GPU tests run on demand. Push the commits you want tested,
then
comment `run-ci` on the pull request.

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package (do **not** edit
`CHANGELOG.rst` or bump `extension.toml` — CI handles that)
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

(cherry picked from commit b9618b6)
# Description

Adds a rendering benchmark for comparing OVRTX and Newton/Warp renderer
performance on an animated Franka + Sektion-cabinet scene, plus the
plumbing needed to measure it.

- `scripts/benchmarks/benchmark_renderer.py`: sweeps a declarative list
of renderer profiles (OVRTX old/new pipeline × shading presets, Newton
BVH constructor combinations) under `nsys`, parses
per-frame render times from the exported NVTX ranges, and reports
pixels/second plus median/mean/min/max/stdev either as a table or as
JSON.
- `source/isaaclab/isaaclab/renderers/render_context.py`: adds a
backend-agnostic NVTX range (`RENDER_PROFILE_SCOPE`) around
`RenderContext.render_into_camera`'s renderer call, gated by the
`ISAACLAB_RENDER_PROFILE` env var so it has no effect (and no
device-sync cost) in normal runs.
- `source/isaaclab_tasks/isaaclab_tasks/core/render_benchmark/`: a new
direct-workflow task, `Isaac-RenderBenchmark-Franka-Cabinet`, with no
policy or reward — all articulated joints (Franka +
cabinet drawers/doors) are driven by a deterministic sinusoid so every
frame has moving geometry to render. Supports `newton_mjwarp`/`physx`
physics presets and
`rgb`/`albedo`/`depth`/`simple_shading_*` camera presets.

## Type of change

- New feature (non-breaking change which adds functionality)

## Release backport

- [ ] Backport this pull request to the active release branch after it
merges into `develop`

## Checklist

- [x] I have read and understood the contribution guidelines
- [x] I have run the `pre-commit` checks with `./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------

Co-authored-by: Matthew Taylor <mataylor@nvidia.com>
(cherry picked from commit 487b5ca)
…y tasks (isaac-sim#7520)

## The problem

`ArticulationData._sim_bind_joint_pos` is bound straight to
`ArticulationView.get_dof_positions()`:

```python
# isaaclab_newton/assets/articulation/articulation_data.py:1594
self._sim_bind_joint_pos = self._root_view.get_dof_positions(SimulationManager.get_state_0())[:, 0]

# newton/_src/utils/selection.py:1587
def get_dof_positions(self, source):
    """Get the joint coordinate positions (DoF positions) ..."""
    return self._get_attribute_values("joint_q", source)
```

The name says DOF, the return is `joint_q` which is coordinate space.
Newton (like MuJoCo) stores a
ball joint as a 4-component unit quaternion against 3 DOFs, so for an
articulation containing one the
array is wider than `num_joints`, while `joint_names`, `joint_vel`,
`default_joint_pos` and every
joint gain stay in DOF space. On Agility Digit (6 ball joints) that is
**56 coordinates against 50
DOFs**.

PhysX is unaffected: its articulations are reduced-coordinate, a
spherical joint is 3 scalars,
`nq == nv`, and the string `joint_q` does not appear anywhere in
`isaaclab_physx`. The MDP layer was
written against that assumption and the Newton backend inherited it
without a conversion.

IsaacLab addresses joints by DOF index everywhere in `joint_names`,
`find_joints`,
`SceneEntityCfg.joint_ids`, so both directions break, and they break
silently.

### Reads

Every MDP term indexes both arrays with the same ids:

```python
# isaaclab/envs/mdp/rewards.py:186
asset.data.joint_pos.torch[:, asset_cfg.joint_ids] - asset.data.default_joint_pos.torch[:, asset_cfg.joint_ids]
```

The two arrays are in different spaces, so past the first ball joint
they refer to different joints.
On Digit `joint_pos[13]`, `left_leg_knee`, reads a quaternion `w`, which
barely leaves 1.0, so the
knee angle never reaches the policy or the rewards.
`sync_torque_telemetry` carries the same offset.

### Writes

`write_joint_position_to_sim_*` scatters DOF-indexed values into
coordinate slots, so
`reset_joints_by_scale` corrupts the pose on every reset: the
loop-closure residual goes from
0.0004 mm to 828 mm and the solver diverges within 10 steps.

## The fix

`joint_coordinates.py` adds `JointCoordinateMap`, which builds the
coordinate ↔ DOF index tables
once per articulation and converts with four small Warp kernels. Ball
joints go through
rotation vector ↔ quaternion; the gather forces `w >= 0` first, because
a quaternion double-covers
SO(3) and without that a fixed pose can decode to `+θ` one step and
`-(2π − θ)` the next, injecting
discontinuities into the observation. Rotation vector is the
representation consistent with
`joint_qd`, which already holds angular velocity in the joint frame.

`required` is False when every joint has one coordinate per DOF — i.e.
every articulation without a
ball joint. Those keep the existing zero-copy view onto `joint_q` and
cost nothing.

Four call sites: the binding in `articulation_data.py`, a lazy
`_refresh_joint_positions()` in
`update()` and the `joint_pos` property, a `_flush_joint_positions()`
after each of the four joint
position writers in `articulation.py`, and a forced refresh in
`_post_actuator` (it runs inside the
step, before `update()` bumps the timestamp).

The scatter is scoped to the environments each write touched. Resets are
staggered, so an unscoped
scatter would round-trip every environment's ball joints through the log
map on nearly every step,
and the float32 rounding lands on the loop-closure constraints.

## The Digit tasks

`DigitPhysicsCfg` gains a `newton_mjwarp` branch, so
`presets=newton_mjwarp` selects it on the
shipped tasks; the PhysX default is unchanged. Digit was never
PhysX-only by design — that missing
branch was the whole implementation of it. Four items specific to
`digit_v4.usd`, all gated on the
preset:

* 32 `CollisionAPI` prims, every one a `/Visual/` decoration mesh on a
RealSense camera mount. They
become 57% of the robot's shapes and produced 3e7 N contact forces 1.4 m
above the ground.
* Ten joints ship with armature below MJWarp's explicit-damping bound
`c·h/I < 2` — `wrist_yaw` at
  7.86, eight more at 2.74. They get their own actuator group with
  `armature = preset(default=None, newton_mjwarp=0.10)`.
* No `articulation_props`, so Newton filtered all 253 intra-articulation
shape pairs and the legs
  passed through each other.
* `entropy_coef = preset(default=0.01, newton_mjwarp=0.005)`. MJWarp is
less forgiving of the action
tail than PhysX, and at 0.01 a seed occasionally diverges late in
training while the policy itself
  is still healthy.

## Result on the task this came from

Digit was listed in `docs/source/refs/issues.rst` as "not currently
validated" on Newton, attributed
to its closed kinematic loops. The loops import and solve fine; this was
the actual blocker. With it
fixed (3 seeds each, 4096 envs, `Metrics/success_rate` at the final
iteration):

| | before | after |
|---|---|---|
| Velocity-Flat-Digit | 0.000 / 0.000 / 0.000 | **1.000 / 1.000 /
1.000** |
| Velocity-Rough-Digit | 0.019 / 0.005 / 0.002 | **0.992 / 0.993 /
1.000** |

Episode length 985–1000 of 1000, tip-over termination 0.008 (was 1.000
on rough), terrain curriculum
climbing to 3.5 (was pinned at 0).

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] bug fixed
- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have added a changelog fragment under
`source/<pkg>/changelog.d/` for every touched package (do **not** edit
`CHANGELOG.rst` or bump `extension.toml` — CI handles that)
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------

Co-authored-by: ooctipus <zhengyuz@nvidia.com>
(cherry picked from commit 9d1896d)
@kellyguo11
kellyguo11 requested a review from a team September 16, 2026 16:11
@github-actions github-actions Bot added documentation Improvements or additions to documentation asset New asset feature or request isaac-mimic Related to Isaac Mimic team labels Sep 16, 2026
@greptile-apps

greptile-apps Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

The PR appears safe to merge after considering one non-blocking edge case in the optional Mimic failure-limit configuration.

Findings

  1. P2 Failure Cap Allows Attempt ▶

Summary

This backport completes several task, benchmark, Mimic, renderer, and Newton changes from the development branch.

  • Restores the heterogeneous multitask manipulation task, registration, task-specific policy/value heads, and selection-aware MDP terms.
  • Adds a render benchmark task and CLI plus a shared renderer profiling scope.
  • Adds Newton ball-joint coordinate/DOF conversion and enables Digit velocity tasks on MJWarp.
  • Updates NuRec locomanipulation generation, Mimic failure limiting, demos, documentation, and associated tests.
  • One lower-impact configuration edge case remains in the new Mimic failure limit.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Tasks[Task catalog] --> Multi[Multitask manipulation]
    Tasks --> Bench[Render benchmark task]
    Bench --> Camera[Tiled camera]
    Camera --> Scope[Shared render timing scope]
    Scope --> Report[Benchmark statistics]
    Mimic[Mimic generation] --> FailureCap[Optional failure cap]
    Mimic --> NuRec[NuRec scene placement and capture]
    Newton[Newton articulation] --> Convert[Coordinate / DOF conversion]
    Convert --> Digit[Digit MJWarp tasks]
Loading

Reviews (1) · Last reviewed commit: "Fix Newton joint positions in DOF space,..."

Comment on lines +147 to +148
max_num_failures = env.cfg.datagen_config.max_num_failures
if generation_guarantee and max_num_failures is not None and num_failures >= max_num_failures:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Failure Cap Allows Attempt

With max_num_failures=0 or a negative value, this check still allows one generation attempt because it runs only after env.step(). That conflicts with the documented maximum: zero should allow no failures, and negative limits should be rejected. Validate that the value is non-negative and check an already exhausted limit before stepping.

Knowledge Base Used: Imitation learning and data generation

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The backport is broadly coherent, but two maintainability/documentation defects need correction: the SDG scene helper can permanently cache an empty pre-cloning pose, and the Newton changelog references a class name that the added module does not define.

  • Design and architecture: The SDG SceneAsset cache conflicts with its lazy view lifecycle. Once get_pose() caches a result, it bypasses _get_xform_view(), preventing the documented rebuild of a view created before environment cloning.
  • API: The Newton ball-joint changelog links to JointCoordinateMap, while the implementation exposes BallJointCoordinateMap. The release note should reference the actual symbol, or the symbol should be renamed/exported consistently.
  • Implementation: Move the SDG pose-cache check behind validation or rebuilding of the transform view so an empty pre-cloning result cannot remain permanently cached. The other reviewed paths do not contribute additional accepted findings.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.


def get_pose(self):
"""Get the 3D pose of the entity."""
if self._cached_pose is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Implementation — Pose cache bypasses lazy view rebuild

get_pose now returns _cached_pose before calling _get_xform_view, so the rebuild path documented in this file (a view created before environment cloning is rebuilt once cloned prims exist, clearing the cache) becomes unreachable once any pose is cached. A first read while the view still has count == 0 caches an empty tensor that every later read returns, since only set_pose refreshes it. Populate the cache only after obtaining a non-empty view.

array was wider than ``num_joints`` while ``joint_names``, ``joint_vel``, ``default_joint_pos``
and the joint gains stayed in DOF space, and every consumer indexing them with the same joint ids
read a different joint past the first ball joint. Added
:class:`~isaaclab_newton.assets.articulation.joint_coordinates.JointCoordinateMap`, which converts

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Api — Changelog names a nonexistent Newton class

The fragment cross-references isaaclab_newton.assets.articulation.joint_coordinates.JointCoordinateMap, but the added module defines BallJointCoordinateMap. The Sphinx reference will not resolve and readers following the release note cannot import the documented symbol. Update the fragment to the actual class name, or rename/export the class to match the documented name consistently.

@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 16, 2026
@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 17, 2026
maxkra15 and others added 2 commits September 17, 2026 17:18
# Description

Pin the Franka Pour robot asset to its authored `Colliders=convex_hulls`
variant. The primitive default omits the arm collision meshes (`link0_c`
through `link7_c`) required by the task, preventing the current Newton
MPM environment from starting correctly.

This keeps the current 70% / 735-particle layout unchanged.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Release backport

- [x] <!-- backport-active-release --> Backport this pull request to the
active release branch after it merges into `develop`

## Screenshots

Not applicable.

## Validation

- `uv run --extra test python -m pytest -q
source/isaaclab_tasks/test/contrib/test_franka_pour_env_cfg.py` (13
passed)
- `uv run isaaclab -f`
- Local Newton GL playback with the published `model_6997.pt` checkpoint
and one current-layout environment

## Checklist

- [x] I have read and understood the contribution guidelines
- [x] I have run the pre-commit checks
- [x] Documentation changes are not required for this task configuration
fix
- [x] My changes generate no new warnings
- [x] I have added a test that proves the asset variant contract
- [x] I have added a changelog fragment for `isaaclab_tasks`
- [x] My name already exists in `CONTRIBUTORS.md`

(cherry picked from commit ad8b371)
…aac-sim#7870)

Franka Lift can stall while filling its reset-state bank: the updated
`franka_panda.usda` defaults to `Colliders=primitives`, whose first-link
capsule extends 6 cm below the ground. The existing ground-clearance
criterion rejects every candidate, leaving the RSL-RL export batch at
`0/2048` until its 600-second timeout. This PR selects
`Colliders=convex_hulls` on the robot configuration shared by rigid
Franka Lift and Reorient, restoring the previous collision geometry and
successful reset sampling.

The affected asset and reset setup originated in
[isaac-sim#6457](isaac-sim#6457)
([564a008](isaac-sim@564a008)),
then moved into the current Lift tasks in
[isaac-sim#6732](isaac-sim#6732)
([73106da](isaac-sim@73106da)).
The triggering change is external: the production Franka asset's top
layer now selects primitive colliders and reports `Last-Modified:
2026-09-15 22:49:27 GMT`. Controlled comparisons found 0 valid states in
128 draws with the new default; explicit convex hulls and the preserved
old asset both produced 63 valid states from the same draws.

The same reset hang appeared in CI for isaac-sim#7838, isaac-sim#7608, isaac-sim#7832, isaac-sim#7834, and
isaac-sim#7866. [The isaac-sim#7866 RL
log](https://github.com/isaac-sim/IsaacLab/actions/runs/35202342707/job/105139978507)
shows the batch blocked on Franka Lift after environment setup. isaac-sim#7866
fixes the independently configured Franka Pour task; it does not change
Lift's robot configuration. This change covers Lift/Reorient only.

Validation:

- Fresh regression check on this branch: both Lift/Reorient cases failed
without the production fix; all 9 tests in `test_lift_env_cfg.py` passed
with it.
- `uv run isaaclab -f` passed.
- Earlier GPU validation of the identical Lift change completed the
original 2,048-state harvest and reset in 22.169 seconds; the existing
four-task RSL-RL export batch passed in 107.92 seconds.
- Local GPU validation used PyTorch 2.11+cu128, Newton 1.6rc1, and
RSL-RL 5.4.1. The identical Lift fix also passed [the full
locked-environment RL CI job in
isaac-sim#7865](https://github.com/isaac-sim/IsaacLab/actions/runs/35207648168/job/105158711330).

Reset-bank sizes, clearance criteria, export behavior, the global Franka
defaults, and Pour remain unchanged.

(cherry picked from commit c5d33ef)
@kellyguo11

Copy link
Copy Markdown
Contributor Author

run-ci

@isaaclab-bot isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Sep 17, 2026
@kellyguo11

Copy link
Copy Markdown
Contributor Author

Superseded by the consolidated backport in #7851 so CI validates the complete dependency/runtime/task/tooling fix set together.

@kellyguo11 kellyguo11 closed this Sep 17, 2026
kellyguo11 added a commit that referenced this pull request Sep 17, 2026
# Description

Consolidates the complete `develop` to `release/3.0.0` backport set
previously split across #7851, #7852, and #7853. Keeping the changes
together lets CI validate the runtime, task, tooling, dependency, and
renderer prerequisites as one coherent release tree.

The aggregate includes all audited develop changes through `e8363313c`,
including the latest #7866, #7867, #7868, #7869, and #7870 fixes. It
also includes the exact net patch from #7871 (`412cac434` / patch ID
`0f69afa6`) because the externally updated Franka asset already breaks
the release legacy-rendering matrix; #7871 is still pending merge to
`develop` at the time of this update.

Release CI now uses the same immutable Isaac Sim `latest-develop` digest
as `develop`:

```yaml
isaacsim_image_tag: latest-develop@sha256:223adbb0a6f1897ed78d40597dac2efc740843e8051779ffdcbb021a9edc1217
```

This supersedes #7852 and #7853.

## Type of change

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Documentation update
- Release integration/backport

## Release backport

- [ ] <!-- backport-active-release --> Not applicable: this PR targets
`release/3.0.0` directly.

## Screenshots

Not applicable.

## Validation

- Latest five develop commits are patch-equivalent in the aggregate
tree.
- #7871 renderer fix matches its source patch ID exactly.
- `.github/workflows/config.yaml` matches `develop`.
- 33 OVRTX USD/export tests passed; 19 optional backend tests skipped
locally.
- 3 focused Franka Pour/Lift/Reorient collider regressions passed.
- 45 standalone supervisor tests passed; 386 simulator launch cases
skipped locally.
- All 23 repository skills validated.
- Ruff check and formatting check passed on all manually
merged/latest-fix Python files.
- Workflow YAML parsing, `uv lock --check`, and aggregate `git diff
--check` passed.
- Full `uv run isaaclab -f` and local docs build cannot run on macOS
ARM64 because the lock supports Linux x86_64/aarch64 and Windows AMD64;
Linux CI is authoritative for those checks.

## Checklist

- [x] I have read and understood the contribution guidelines.
- [ ] I have run the full pre-commit command locally
(platform-incompatible; focused Ruff/format checks passed and CI is
running).
- [x] I have made corresponding documentation changes.
- [x] My focused local checks generate no new warnings.
- [x] I have added/backported tests that prove the fixes are effective.
- [x] I have included changelog fragments for every touched source
package.
- [x] Contributor attribution is preserved from the source changes.

---------

Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Rebecca Zhang <168459200+rebeccazhang0707@users.noreply.github.com>
Co-authored-by: Kai Pei <2047767028@qq.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: ooctipus <zhengyuz@nvidia.com>
Co-authored-by: Mustafa H <34825877+StafaH@users.noreply.github.com>
Co-authored-by: Maximilian Krause <99733341+maxkra15@users.noreply.github.com>
Co-authored-by: Ege Sekkin <72572910+nvsekkin@users.noreply.github.com>
Co-authored-by: Antoine RICHARD <antoiner@nvidia.com>
Co-authored-by: Piotr Barejko <pbarejko@nvidia.com>
Co-authored-by: nvsekkin <esekkin@nvidia.com>
Co-authored-by: camevor <camevor@nvidia.com>
Co-authored-by: hujc <jichuanh@nvidia.com>
Co-authored-by: marcodiiga <1969828+marcodiiga@users.noreply.github.com>
Co-authored-by: Alesiani Marco <marcodiiga@users.noreply.github.com>
Co-authored-by: Diego Ferigo <dferigo@rai-inst.com>
Co-authored-by: Matthew Taylor <mataylor@nvidia.com>
Co-authored-by: Yuchen Deng <yuchendeng@nvidia.com>
Co-authored-by: Daniela Hase <116915287+daniela-hase@users.noreply.github.com>
Co-authored-by: Henry Hu <yukanghu7@163.com>
Co-authored-by: r-schmitt <139814266+r-schmitt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

asset New asset feature or request documentation Improvements or additions to documentation isaac-mimic Related to Isaac Mimic team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants