test: benchmark end-to-end SuperDex Go2 collector throughput

docs/sphinx/source/adr/ADR-0000-index.md

@@ -21,6 +21,7 @@ orphan: true
 | [ADR-0005 Unified Obs Critic Env And IPC Contract](ADR-0005-unified-obs-critic-env-and-ipc-contract.md) | Observation / IPC | Accepted |
 | [ADR-0006 Community Manager API On NumPy Runtime](ADR-0006-community-manager-api-on-numpy-runtime.md) | Manager API / NumPy runtime | Accepted |
 | [ADR-0007 UniSim Extraction Boundary](ADR-0007-unisim-extraction-boundary.md) | Physics package extraction | Accepted |
+| [ADR-0008 SuperDex Native C++ Scene Batch Executor](ADR-0008-superdex-persistent-cpu-workers.md) | Backend CPU scene execution | Accepted |
 
 ## ADR Governance
 

docs/sphinx/source/adr/ADR-0008-superdex-persistent-cpu-workers.md

@@ -0,0 +1,82 @@
+---
+orphan: true
+---
+
+# ADR-0008 SuperDex Native C++ Scene Batch Executor
+
+- Status: Accepted
+- Date: 2026-09-07
+- Owners: SuperDex fork / UniSim backend / UniLab task-config maintainers
+- Supersedes: None
+- Superseded by: None
+
+## Context
+
+The initial SuperDex adapter creates one independent native scene for every
+environment but advances them serially from Python. `Scene.step()` releases the
+GIL, yet a Python thread pool would still leave generalized-force writes and
+articulated state reads as separate Python-to-C++ calls. A subprocess/shared
+memory design was considered and rejected before implementation because it
+would introduce a new runtime protocol beyond the requested integration scope.
+
+The maintained fork [unilabsim/project_superdex](https://github.com/unilabsim/project_superdex)
+can build an extension against the exact Physics and Robotics sources. This
+removes the wheel/header ABI mismatch that prevents a downstream native shim.
+The roadmap is [UniLab#1533](https://github.com/Motphys/UniLab/issues/1533).
+
+## Decision
+
+1. Add `superdex.physics.SceneBatchExecutor` to the fork's `mochi_physics`
+   pybind extension. It owns persistent C++ worker threads, but does not own
+   scenes or actors.
+2. One executor invocation receives contiguous generalized-force, articulated
+   pose/velocity, link-state, contact and solver-status arrays. It writes each
+   actor's forces, advances each distinct scene, and refreshes every runtime
+   cache before its completion barrier opens.
+3. UniSim owns the executor and creates it after cold-path materialization. It
+   keeps reset, asset conversion, cache-frame conversion and `SimBackend`
+   ownership in the adapter. `close()` joins the executor
+   before destroying bots, scenes, and the process-global runtime.
+4. `superdex_num_workers=0` selects `min(available physical CPU cores,
+   num_envs)`. Explicit worker counts are capped at `num_envs`; the SDK stays
+   single-threaded so the executor is the only physics parallelism layer.
+
+## Consequences
+
+The local integration requires Physics and Robotics bindings built from this
+fork at the same commit. Older qpos/qvel-only executor builds are rejected. It
+neither changes package versions nor publishes a wheel. The executor provides
+CPU scene parallelism; it does not claim GPU physics, native rendering, or
+dynamics equivalence with MuJoCo.
+
+Validation must compare serial and parallel trajectories, selected reset, and
+complete backend throughput using the same scene, actions, batch, and substep
+count. Training evidence must report the actual outer worker count and SDK
+thread count separately.
+
+## Alternatives Considered
+
+- Python `ThreadPoolExecutor`: useful for a narrow step probe, but does not
+  fuse force/state binding calls or give the adapter a durable native barrier.
+- Subprocess IPC and shared memory: rejected because SuperDex does not require
+  process isolation and the protocol is outside this roadmap's approved scope.
+- Linking a downstream extension to the released wheel: rejected because the
+  wheel does not publish a stable extension ABI or matching headers.
+
+## Evidence In Repo
+
+- `src/unilab/conf/ppo/task/go2_joystick_flat/superdex.yaml` selects automatic
+  native workers for the SuperDex owner.
+- `src/unilab/scripts/train_offpolicy.py` and
+  `src/unilab/scripts/train_rsl_rl.py` obtain rank-owned CPU ids from UniRL
+  before environment construction.
+- `tests/algos/test_offpolicy_double_buffer_runner.py` verifies that each rank
+  receives complete physical-core groups, including SMT siblings.
+- `tests/ipc/test_dp_launcher.py` verifies the corresponding resolver contract
+  through the UniRL dependency.
+
+## Related Documents
+
+- {doc}`SuperDex Backend </en/2-user_guide/3-backends/8-superdex>`
+- {doc}`UniSim Extraction Boundary </adr/ADR-0007-unisim-extraction-boundary>`
+- [UniLab#1533](https://github.com/Motphys/UniLab/issues/1533)

docs/sphinx/source/adr/README.md

@@ -20,6 +20,8 @@ orphan: true
 | [ADR-0004 Registry Bootstrap Contract](ADR-0004-registry-bootstrap-contract.md) | Registry bootstrap | Accepted |
 | [ADR-0005 Unified Obs Critic Env And IPC Contract](ADR-0005-unified-obs-critic-env-and-ipc-contract.md) | Observation / IPC | Accepted |
 | [ADR-0006 Community Manager API On NumPy Runtime](ADR-0006-community-manager-api-on-numpy-runtime.md) | Manager API / NumPy runtime | Accepted |
+| [ADR-0007 UniSim Extraction Boundary](ADR-0007-unisim-extraction-boundary.md) | Physics package extraction | Accepted |
+| [ADR-0008 SuperDex Native C++ Scene Batch Executor](ADR-0008-superdex-persistent-cpu-workers.md) | Backend CPU scene execution | Accepted |
 
 ## ADR Governance
 

docs/sphinx/source/conf.py

@@ -277,6 +277,7 @@
 _LANGUAGE_PATH_FORWARD: dict[str, str] = {
     "en/why_unilab": "zh_CN/why_unilab",
     "en/2-user_guide/3-backends/6-drake": "zh_CN/2-user_guide/3-backends/6-drake",
+    "en/2-user_guide/3-backends/8-superdex": "zh_CN/2-user_guide/3-backends/8-superdex",
     "en/1-getting_started/5-faq": "zh_CN/1-getting_started/5-faq",
     "en/4-developer_guide/1-architecture/6-manager_based_api": (
         "zh_CN/4-developer_guide/1-architecture/6-manager_based_api"

docs/sphinx/source/en/2-user_guide/3-backends/0-index.md

@@ -1,7 +1,7 @@
 # Simulation Backends
 
 UniLab exposes backend names through registry/config paths, including `mujoco`,
-`motrix`, `mjwarp`, `drake`, `isaacgym`, `genesis`, `isaacsim`, and `newton`
+`motrix`, `mjwarp`, `drake`, `isaacgym`, `genesis`, `isaacsim`, `newton`, and `superdex`
 where an owner is registered.
 User commands select them with `--sim`, which routes to the matching task owner
 YAML; do not switch a run by overriding `training.sim_backend` alone.
@@ -146,4 +146,5 @@ separately authorized issue.
 5-genesis
 6-drake
 7-newton
+8-superdex
 ```

docs/sphinx/source/en/2-user_guide/3-backends/8-superdex.md

@@ -0,0 +1,156 @@
+# SuperDex Backend
+
+SuperDex is an optional CPU physics adapter owned by `unisim.backend.superdex`.
+The initial UniLab owner is the fixed-base `FR3JointTarget` task:
+`src/unilab/conf/ppo/task/fr3_joint_target/superdex.yaml`. It uses seven torque
+actions, 21 observation values, joint-state resets and the standard NumPy
+manager runtime. Its support level is **Configured**; bounded rollout or short
+training checks do not establish full-training performance or platform support.
+The implementation is tracked in [#1534](https://github.com/Motphys/UniLab/issues/1534)
+under [roadmap #1533](https://github.com/Motphys/UniLab/issues/1533).
+
+## Local Development Setup
+
+This development profile uses locally linked UniSim and UniLab checkouts; it
+does not require a new published version. SuperDex Physics/Robotics 1.0.0 requires
+Python 3.12. CPU physics does not require CUDA. Native rendering and video are
+not part of the FR3 owner; training defaults to `no_play=true`.
+
+In the UniLab checkout, use an existing Python 3.12 virtual environment or create
+one, then install the local packages:
+
+```bash
+uv venv --python 3.12
+export UNILAB_LOCAL_UNISIM=/absolute/path/to/unisim
+uv pip install -e "${UNILAB_LOCAL_UNISIM}[superdex,mujoco]" -e . --group pyproject.toml:dev
+export UV_NO_SYNC=1
+export SUPERDEX_ASSETS_PATH=/absolute/path/to/project_superdex/assets
+```
+
+`UNILAB_LOCAL_UNISIM` enables the repository's local dependency validation:
+tests require an editable installation and check that its metadata and imported
+module point to exactly that checkout. Without this variable, the normal
+indexed-release requirement remains in force. `UV_NO_SYNC=1` preserves the local
+links when running existing Make targets; `uv sync` would re-resolve the locked
+release profile.
+
+Native FR3 assets remain in the upstream SuperDex checkout. The asset hub
+registers `bots/arms/fr3_v2/fr3_v2.superdex_bot`, checking its collision SDF,
+render files, `LICENSE` and `NOTICE` before constructing physics. No native
+robot binaries are bundled or downloaded by UniLab. Set
+`env.superdex_assets_root=/absolute/path/to/project_superdex/assets` to override
+`SUPERDEX_ASSETS_PATH` for a specific owner invocation.
+
+## Run the FR3 Task
+
+```bash
+uv run --no-sync train --algo ppo --task fr3_joint_target --sim superdex \
+  algo.max_iterations=2 algo.num_steps_per_env=16 \
+  algo.algorithm.num_learning_epochs=1
+```
+
+The target joint positions, rewards, reset ranges and action scales live in the
+task's `base.yaml`. The torque bounds `[20,20,20,20,5,5,5]` Nm are an explicit
+research profile, not rated hardware limits. `superdex_effort_limits` declares
+the same bounds at the native backend boundary. The SDK remains single-threaded;
+the native scene executor below owns all supported CPU parallelism.
+
+## Default CPU Environment Parallelism
+
+The backend uses SuperDex's source-built `SceneBatchExecutor`, a persistent C++
+thread pool that owns the barrier across independent scenes. Each substep writes
+batched generalized forces, advances scenes, and returns articulated/link state,
+contact sensors and solver status without per-environment Python binding calls.
+Asset materialization, reset and cache-frame transforms remain owned by the
+UniSim adapter. This is CPU thread parallelism, not GPU physics, and it does
+not change the PPO/APPO collector, learner or policy contracts. The decision is
+recorded in {doc}`/adr/ADR-0008-superdex-persistent-cpu-workers` and
+[unisim#41](https://github.com/unilabsim/unisim/issues/41).
+
+Both task owners select automatic workers by default:
+
+| Owner option | Meaning |
+| --- | --- |
+| `env.superdex_num_workers=0` | Automatic: `min(available physical CPU cores, num_envs)` |
+| `env.superdex_num_workers=1` | One native C++ scene worker |
+| `env.superdex_num_workers=K` | Explicit C++ worker count, capped at `num_envs` |
+
+With 1024 environments on a 16-core/32-thread host, automatic selection yields
+16 workers. Concurrent multi-rank collectors are assigned whole physical-core
+groups, including their logical siblings, so ranks do not split an SMT core.
+`training.dp_collector_cpu_ids` may instead provide one explicit CPU-id list
+per rank. The selected block is applied before SuperDex materializes its native
+worker pool.
+
+For every physics substep, the host runs the pre-step control callback, enters
+the native batch barrier, then publishes the refreshed batch before the next
+callback. Selected reset preserves caller row order and leaves unselected
+environments unchanged. A native worker failure closes the executor and reports
+the failure; it does not silently return stale state or switch to serial.
+Closing an environment joins its C++ workers before scenes are destroyed.
+
+Worker count alone is not evidence of speedup. Throughput comparisons must use
+the same model, control sequence, batch size and substeps, and report complete
+backend/env time, startup, RSS, CPU use and actual worker count. Count environment
+control steps, not physics substeps. Existing contact approximations are unchanged.
+
+The fixed root still has a named entity and readable body state. Reset terms
+write joint state; they do not request a floating-root layout. The task does
+not require contact sensors, cameras, site Jacobians or runtime material DR.
+SuperDex has no native renderer. Its default record playback uses the offline
+MuJoCo renderer with the authored MJCF visual model while SuperDex remains the
+physics backend. `.superdex_bot` scenes must provide `visual_model_file` for
+this path. Selecting playback mode `none` skips playback entirely; it is not
+evidence that a checkpoint has executed a rollout.
+
+`superdex_allow_contact_approximation` defaults to `false`. It is reserved for
+explicitly audited MJCF conversion profiles: enabling it accepts a warning about
+contact/material approximation, including missing torsional/rolling friction
+equivalence. It does not establish arbitrary MJCF task compatibility.
+Contact queries report the last completed solver step. Reset clears this state;
+it does not provide a fresh geometric overlap test until a positive physics step
+has completed. Kinematic body/joint getters are refreshed immediately at reset.
+
+## Validation and Ownership
+
+The `go2_joystick_flat/superdex` PPO owner is a research sim2sim profile. It
+inherits the MuJoCo owner, preserving 49 actor observations, 52 critic
+observations, 12 position-target actions, normalization, network dimensions and
+control timing. It disables runtime PD gain randomization and explicitly opts
+into contact approximation. The adapter's cold-path MJCF conversion is restricted;
+this owner does not imply arbitrary scene support or equivalent walking behavior.
+
+Create a small source checkpoint with the MuJoCo owner, then pass its path to the
+optional checkpoint test:
+
+```bash
+uv run --no-sync train --algo ppo --task go2_joystick_flat --sim mujoco \
+  algo.num_envs=2 algo.max_iterations=2 algo.num_steps_per_env=16 \
+  algo.algorithm.num_mini_batches=1 algo.algorithm.num_learning_epochs=1 \
+  training.device=cpu training.no_play=true
+export UNILAB_SUPERDEX_GO2_CHECKPOINT=/absolute/path/to/source/run/model_1.pt
+uv run --no-sync pytest tests/envs/test_go2_superdex.py -q
+```
+
+This validates the source `run_config.json` before environment construction,
+checks rejection of changed policy action semantics, loads the actual policy
+through the production playback session and executes 64 SuperDex control steps
+without a renderer. It checks finite values and interface compatibility; a
+two-iteration checkpoint is not expected to walk reliably.
+
+```bash
+uv run --no-sync pytest tests/assets/test_superdex_assets.py \
+  tests/envs/test_fr3_superdex.py tests/test_cli_runtime_requirements.py -q
+```
+
+The optional native tests require the SDK and `SUPERDEX_ASSETS_PATH`; they cover
+finite rollout data, selected reset isolation, immediate observation refresh
+and a spawned `EnvFactory`. Missing runtime/assets produce an explicit skip;
+such a run is not native validation. The base asset/config tests need no native
+asset checkout.
+
+Engine conversion and physics live in UniSim; asset registration, Hydra and task
+terms remain in UniLab. See
+{doc}`/adr/ADR-0007-unisim-extraction-boundary`,
+{doc}`/adr/ADR-0006-community-manager-api-on-numpy-runtime` and
+{doc}`/adr/ADR-0002-backend-capability-boundary-for-play-and-snapshot`.

docs/sphinx/source/zh_CN/2-user_guide/3-backends/0-index.md

@@ -2,7 +2,7 @@
 
 UniLab 通过 registry/config 路径暴露后端名称，包括在对应 owner 注册后可用的
 `mujoco`、`motrix`、`mjwarp`、`drake`、`isaacgym`、`genesis`、`isaacsim` 和
-`newton`。用户命令通过
+`newton`、`superdex`。用户命令通过
 `--sim` 选择后端，该选项会路由到对应的 task owner YAML；不要仅靠 override
 `training.sim_backend` 来切换一次运行。
 
@@ -137,4 +137,5 @@ benchmark v1 目前只保留 `BenchmarkCase`、`BenchmarkResult` 和 provenance
 5-genesis
 6-drake
 7-newton
+8-superdex
 ```