Add SimReady Central search in env gen by xyao-nv · Pull Request #1513 · isaac-sim/IsaacLab-Arena

xyao-nv · 2026-10-22T22:18:51Z

Add SimReady Central to agentic environment generation and support searching multiple asset sources. https://huggingface.co/spaces/nvidia/simready-central as a endpoint, coupled with USD search.
The value of this library is a community/OEM contributed & validated industrial assets. See release notes here

Why this change?

Environment generation can have more USDs by accessing to Central’s Hugging Face assets alongside the existing SimReady library. Users should be able to search by object name in env gen.

What changes?

Select search sources with a list. Search each selected backend and merge candidates, deduplicating identical USD paths per phrase.
Keep GA/S3/service search in simready_ga_assets.py, Central REST search in simready_central_assets.py, and shared types and the backend protocol in simready_search_interface.py.
Add tabletop_pick_place_screwdriver_bearing_wrench.yaml: Franka picks and places a ForgeCAD screwdriver on the maple table, alongside an NVIDIA ball bearing and combination wrench. All three props come from Central.
Leave Newton support as a TODO

Usage example

Before:

  --enable_simready_search --simready_source isaac-sim-ga

After:

  --enable_simready_search \
  --simready_sources isaac-sim-ga simready-central

Validation

Tests passed

Example

Franka picks up a spray gun from the maple table and places it on its bench stand. An orange paint jar, paintbrush, paint tube, ball bearing, and combination wrench serve as distractors. All seven objects come from NVIDIA’s Central dataset.

Success requires lifting the gun, then maintaining supported placement on the stand for 10 consecutive steps.

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

greptile-apps · 2026-10-23T04:57:25Z

Confidence Score: 3/5

[Medium impact] The changes affect the optional environment-generation search and use of SimReady assets.

Fix the candidate-name collisions and update the dependency lock before merging.

Findings

Returned assets share one name ▶
Stale lock blocks CI ▶
Asset sources vanish on reopen ▶

Summary

Adds SimReady Central search, pinned asset downloads, and combined searches across selected backends.

Environment generation can search several SimReady sources for each object.
SimReady Central assets can be searched, cached, and used as rigid scene objects.
Asset reports show where searched assets came from and which ones the scene uses.
A Franka scene now shows an egg being placed into a tray among table objects.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Missing object phrases] --> B[Selected search backends]
  B --> C[Existing SimReady libraries]
  B --> D[SimReady Central]
  D --> E[Download pinned package]
  E --> F[Validate body and write wrapper]
  C --> G[Merge distinct USD paths per phrase]
  F --> G
  G --> H[Register catalogue choices]
  H --> I[Generate scene spec]
  G --> J[Asset report]
  I --> J

_{Reviews (1) · Last reviewed commit: "Name SimReady helpers by their operation..." · Reviewed by Greptile}

greptile-apps · 2026-10-23T04:57:28Z

+        for candidate in result.candidates:
+            key = (candidate.search_phrase, candidate.usd_path)
+            if key not in seen:
+                seen.add(key)
+                candidates.append(candidate)


Returned assets share one name

With --simready_sources isaac-sim-ga simready-central, both backends can return different assets for the same phrase, even at the default result limit. This merge keeps both, but register_searched_simready_object gives them the same catalogue name. The agent then overwrites _simready_usd_paths under that name, so the model cannot choose between the assets and the later backend silently wins.

Give each distinct candidate its own catalogue name and keep that name tied to its USD path.

greptile-apps · 2026-10-23T04:57:28Z

    # its own: it adds one wheel, since Isaac Sim already brings everything it depends on, and an
    # extra only bought a way to install the review GUI with its asset search missing.
    "simready-search>=2026.4.2",
+    "huggingface-hub>=0.36",


Stale lock blocks CI

Adding huggingface-hub to pyproject.toml without updating uv.lock breaks CI. The lock still omits this direct dependency from both the dev list and requires-dist. CI runs uv lock --check --no-python-downloads and later uv sync --locked --extra dev, so it rejects the stale lock before running tests.

Regenerate and commit uv.lock with this dependency change.

greptile-apps · 2026-10-23T04:57:28Z

    except Exception:
        return False, traceback.format_exc()

+    st.session_state["simready_asset_report"] = agent.simready_asset_report(spec)


Asset sources vanish on reopen

The GUI keeps the asset report only in st.session_state, while automatically saving the generated YAML. On reopening that YAML in a new session, initialize_state looks for a sibling .assets.json file that the GUI never writes. Central specs contain local wrapper paths, so users also lose the original source URLs unless they downloaded the report separately.

Save the report beside the YAML using the same naming convention as the CLI.

arena-review-bot · 2026-10-23T05:03:19Z

+            _logger.warning("simready backend %s failed: %s", source, exc)
+            continue
+        for candidate in result.candidates:
+            key = (candidate.search_phrase, candidate.usd_path)


🟡 Two hits for one object collapse into one

With two sources selected, "egg" can get a hit from the GA library and one from Central, and both survive this merge. In _extend_catalogue_with_simready, register_searched_simready_object names the asset after the phrase only, so the second call gets the first class back.

As a result:

the LLM sees two identical simready_egg entries;

_simready_usd_paths keeps only the last path;

the spec always gets the asset from the source listed last, so the docs' "up to two candidates per object" never gives the model a choice.

max_results_per_object > 1 already had this problem, but with two sources it now happens at the default setting.

Could we keep only the first candidate per phrase here, in source order? Or give each candidate its own registry name, such as simready_egg_2? An agent-level test with two candidates for one phrase would catch this.

+1. keeping only one is ok, but we should make the choice explicit/documented.

arena-review-bot · 2026-10-23T05:03:20Z

+            ]
+            if not counts or any(float(count) != 1 for count in counts):
+                continue
+            usd_path = _download_and_wrap_central_asset(source["path"])


🟡 Generated specs point at a file in this machine's cache

For Central hits, usd_path is the wrapper under ~/.cache/isaaclab_arena/simready-central/, which in turn points at an absolute Hugging Face cache path. That path goes into the spec, so the spec only loads for the same user on the same machine. GA hits put a remote URL in the spec instead.

SimReadyUsdObject already accepts the pinned URL and builds the wrapper itself, and source_url is that URL. Could the candidate keep the pinned URL as usd_path and still run the checks here? Generated specs would then work anywhere, like the egg-tray YAML. source_url and the cache-path part of the asset report would also no longer be needed.

+1. I think the usd_path can store the pinned remote URL for simready central already, and we can just remove the added source_url param

source_url = source["path"].replace("hf://", "https://", 1) _download_and_wrap_central_asset(source["path"]) # validate and populate the cache candidates.append( SimReadyObjectCandidate( search_phrase=phrase, usd_path=source_url, # ... ) )

arena-review-bot · 2026-10-23T05:03:20Z

    ):
        assert usd_path, "simready_usd_object requires params.usd_path"
+        if usd_path.startswith(("https://huggingface.co/datasets/", "hf://huggingface.co/datasets/")):
+            from isaaclab_arena.agentic_environment_generation.simready_central import _download_and_wrap_central_asset


🟡 Asset class reaches into the env-gen package

SimReadyUsdObject is a core asset used by any spec, but it now imports the private _download_and_wrap_central_asset from agentic_environment_generation. The import is lazy, but asset construction still depends on the agent package. Downloading, checking and wrapping a Central package is asset loading, not agent work.

Could the URL parsing, download and wrap helpers move to an asset-side module, such as isaaclab_arena/assets/simready_central_asset.py, under public names? The search and this class would then both import from there. The prefix check here could also reuse that module's URL parser instead of repeating the prefixes.

arena-review-bot · 2026-10-23T05:03:20Z

+                    source_url=source["path"].replace("hf://", "https://", 1),
+                )
+            )
+        except Exception as exc:


🟡 A broken setup looks like "no asset found"

This catch treats an expected rejection (articulated asset, several bodies, disabled body) the same as a real failure:

snapshot_download failing on network, auth or rate limits;

huggingface_hub not being installed;

a disk error while writing the wrapper.

All of these are logged at INFO for each hit, and the phrase ends up unmatched. Both the CLI and the GUI then tell the user to rephrase the prompt.

Could the checks in _parse_central_asset_url and _download_and_wrap_central_asset raise ValueError instead of using assert, and this block catch only ValueError? Those checks validate downloaded files and URLs that come from the search or from a user's YAML. That is external input, so a real exception fits better, and the checks would still run under -O. Other errors would then reach the WARNING in _search_simready_source.

arena-review-bot · 2026-10-23T05:03:20Z

    # its own: it adds one wheel, since Isaac Sim already brings everything it depends on, and an
    # extra only bought a way to install the review GUI with its asset search missing.
    "simready-search>=2026.4.2",
+    "huggingface-hub>=0.36",


🟡 uv.lock needs updating

Adding huggingface-hub to the dev extra changes the project's requires-dist. However, uv.lock isn't part of this PR, and its isaaclab-arena metadata has no huggingface-hub entry. The native uv job runs uv lock --check before syncing, so it should fail on this. Running uv lock and committing the result should fix it.

arena-review-bot · 2026-10-23T05:03:20Z

+    phrases: list[str], config: SimReadySearchConfig, source: SimReadySourceKind
+) -> SimReadyCandidateCatalogue:
+    """Search one source, retaining the configured number of hits per phrase."""
+    library = (


🔵 Choose the per-phrase search once

This function checks source == SimReadySourceKind.CENTRAL three times and re-runs the import for every phrase. The unmatched_phrases it returns is never read, because search_simready_objects works it out again. Could it choose the search function once and return just the candidates?

if source == SimReadySourceKind.CENTRAL: # Deferred: simready_central imports this module. from isaaclab_arena.agentic_environment_generation.simready_central import search_central_objects search_phrase = search_central_objects else: library = _create_asset_library_for_source(source, s3_url=config.s3_url, service_url=config.service_url) if library is None: return [] search_phrase = functools.partial(_search_library_rigid_body_candidates, library) # ... then, per phrase: hits = search_phrase(phrase, max_results=config.max_results_per_object)

arena-review-bot · 2026-10-23T05:03:20Z

+    assert max_results > 0, "max_results must be positive"
+    response = requests.post(
+        CENTRAL_SEARCH_URL,
+        headers={"x-usdsearch-storage-backend": "usdsearch-dev"},


🔵 Is usdsearch-dev the backend to ship against?

The name suggests a development deployment. If it is renamed or removed, every Central search fails. If it's intended, could it be a named constant next to CENTRAL_SEARCH_URL, with a short note on why this backend is used?

arena-review-bot · 2026-10-23T05:03:20Z

+    )
+    response.raise_for_status()
+    candidates = []
+    for hit in response.json()["hits"]:


🔵 No check that a hit is the object asked for

The library path keeps only hits whose path names the object (_filter_and_rank_path_matches), which guards against a "grey bin" search returning a cabinet. Here, the first single-body hit is accepted whatever it is. Hybrid search usually still returns its top hits when nothing really matches. With two sources, such a hit can also replace a correct GA hit (see the merge comment). Is there a score threshold or a name property we could check?

arena-review-bot · 2026-10-23T05:03:20Z

     - Sets the number of zero-action simulation steps. Default: ``20``.
     - Evaluation smoke test

+With ``--enable_simready_search --simready_source simready-central``, search


🔵 Old flag name

This paragraph still uses --simready_source, while the rest of the page uses --simready_sources. The alias still works, but one name is easier to follow.

Suggested change

With ``--enable_simready_search --simready_source simready-central``, search

With ``--enable_simready_search --simready_sources simready-central``, search

arena-review-bot · 2026-10-23T05:03:23Z

🤖 Isaac Lab-Arena Review Bot

Summary

This PR adds SimReady Central (Hugging Face) as a search backend for agentic environment generation, and lets the runner and GUI search several backends in one run. It also:

downloads and checks Central asset packages before accepting them;
lets SimReadyUsdObject accept pinned Hugging Face URLs;
adds an asset report and an egg-tray example.

The Central download and checking code is careful. The main problems are how results from several backends reach the spec, and where the Central loading code lives.

Design, Boundaries & Scope

Merged results don't reach the model as a choice. When two backends both return a hit for the same object, the two candidates get the same registry name. The spec then always takes the hit from the source listed last, so the docs' "up to two candidates per object" never gives the LLM a real choice. This affects the PR's headline feature (inline on simready_asset_search.py).
Generated specs only work on the machine that ran the search.
- Central hits put a path into the spec that points at a wrapper file in the local cache, which in turn points at an absolute Hugging Face cache path.
- SimReadyUsdObject already accepts the pinned URL. Using that URL would make specs portable and remove the documented "needs the same cache" limitation.
- source_url and the cache-path part of the asset report would then no longer be needed.
Asset loading lives in the agent package. SimReadyUsdObject (core assets) imports the private _download_and_wrap_central_asset from agentic_environment_generation. Downloading and wrapping a Central package is asset loading, so it would sit better in an asset-side module.

Findings

🟡 Warning: simready_asset_search.py:102: two hits for one phrase collapse into one registry entry, and the spec always gets the last backend's path.
🟡 Warning: simready_central.py:54: the spec gets a local cache path instead of the pinned URL.
🟡 Warning: simready_object_library.py:50: the core asset class imports a private helper from the env-gen package.
🟡 Warning: simready_central.py:64: the per-hit except Exception at INFO hides real download and install failures behind "no asset found". The checks on external assets use assert.
🟡 Warning: pyproject.toml:54: uv.lock isn't updated for the new huggingface-hub dev dependency, so the uv lock --check step in native uv CI should fail.
🔵 Improvement: simready_asset_search.py:117: choose the per-phrase search function once. Today the Central check appears three times and the returned unmatched_phrases is never read.
🔵 Improvement: simready_central.py:31: is the usdsearch-dev storage backend meant for shipping?
🔵 Improvement: simready_central.py:43: unlike the library path, Central hits get no check that they are the requested object.
🔵 Improvement: cli_runner.rst:128: the docs use the old --simready_source flag name.

Test Coverage and CI Cost

test_multiple_sources_merge_and_deduplicate_candidates checks that two "egg" candidates come back. No agent-level test passes two candidates for one phrase through _extend_catalogue_with_simready, which is where they collide. One small test in test_environment_generation_agent.py, with two candidates for the same phrase, would catch it. The test should assert that the model is offered distinct entries, or that only one entry per phrase is kept.
CI status is not confirmed. At review time, Docker CI was still queued, and the native uv job had not yet reached its "Validate uv lock" step.

Verdict

Significant concerns

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

qianl-nv

Thanks for initiating this!
It seems we are creating enough special handling logic for simready central that it doesn't fit with the existing AssetLibrary based simready search. Made a proposal to split the two in one of my comments, wdyt?

qianl-nv · 2026-10-23T09:34:54Z

+            ]
+            if not counts or any(float(count) != 1 for count in counts):
+                continue
+            usd_path = _download_and_wrap_central_asset(source["path"])


+1. I think the usd_path can store the pinned remote URL for simready central already, and we can just remove the added source_url param

source_url = source["path"].replace("hf://", "https://", 1) _download_and_wrap_central_asset(source["path"]) # validate and populate the cache candidates.append( SimReadyObjectCandidate( search_phrase=phrase, usd_path=source_url, # ... ) )

qianl-nv · 2026-10-23T14:23:05Z

@@ -0,0 +1,82 @@
+# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).


while the commit message mentions the value of simready-central is "OEM contributed & validated industrial assets", here we are adding another "tabletop/kitchen" example. what's your plan with this example? should we replace with some "industrial looking task" with assets from simready-central (even if it's still pnp)?

qianl-nv · 2026-10-23T14:24:48Z

+            _logger.warning("simready backend %s failed: %s", source, exc)
+            continue
+        for candidate in result.candidates:
+            key = (candidate.search_phrase, candidate.usd_path)


+1. keeping only one is ok, but we should make the choice explicit/documented.

qianl-nv · 2026-10-23T14:27:00Z

@@ -51,6 +51,7 @@ dev = [
    # its own: it adds one wheel, since Isaac Sim already brings everything it depends on, and an
    # extra only bought a way to install the review GUI with its asset search missing.
    "simready-search>=2026.4.2",


which simready version are your local docker using? please make sure if you tested on a newer version, whether this old version is still compatible; if not let's bump the min version here.

qianl-nv · 2026-10-23T15:18:26Z

+_logger = logging.getLogger(__name__)
+
+
+def search_central_objects(phrase: str, *, max_results: int) -> list[SimReadyObjectCandidate]:


As I understand we need this special handling for simready-central because the simready search api still do not support the filter_by_properties_numeric. Let's leave a TODO here explaining the reason?

Alternatively, Central differs enough to justify a backend strategy, separately from GA/S3 ect supported through AssetLibrary.

class SimReadySearchBackend(Protocol): def search( self, phrases: list[str], max_results: int ) -> list[SimReadyObjectCandidate]: ... class AssetLibrarySearchBackend: """Search GA, S3, or generic service sources through AssetLibrary.""" class CentralSearchBackend: """Search Central through its specialized REST API and validate packages."""

Then search_simready_objects() only orchestrates:

Construct selected backends.

Ask each backend for candidates.

Merge/deduplicate candidates.

Compute unmatched phrases.

Separately, move URL parsing, downloading, and wrapper creation into an asset-side module because both Central search and SimReadyUsdObject need it.

qianl-nv · 2026-10-23T15:20:39Z

+
+def _search_simready_source(
+    phrases: list[str], config: SimReadySearchConfig, source: SimReadySourceKind
+) -> SimReadyCandidateCatalogue:


we can return list[SimReadyObjectCandidate]. we don't need a per-source unmathced list, the unmatched phrases is recomputed on line 109 anyway

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

xyao-nv added 3 commits October 22, 2026 13:58

Add single-body SimReady Central asset search

41af2a8

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

Note future Newton support for SimReady Central

847ba2e

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

Add SimReady source lists and Central scene example

d291faf

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

xyao-nv changed the title ~~Add single-body SimReady Central asset search~~ Add SimReady Central search and configurable asset sources in env gen Oct 23, 2026

Define the tabletop egg-tray scene in YAML

1108208

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

xyao-nv changed the title ~~Add SimReady Central search and configurable asset sources in env gen~~ Add SimReady Central search in env gen Oct 23, 2026

xyao-nv added 2 commits October 22, 2026 21:19

Remove pinned asset inputs from environment generation

801fed4

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

Name SimReady helpers by their operations

18b1efc

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

xyao-nv marked this pull request as ready for review October 23, 2026 04:52

xyao-nv requested review from alexmillane, cvolkcvolk, peterd-NV, qianl-nv, viiik-inside and zhx06 as code owners October 23, 2026 04:52

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

arena-review-bot Bot reviewed Oct 23, 2026

View reviewed changes

Remove SimReady report files and CLI doc changes

3cd7a77

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

qianl-nv reviewed Oct 23, 2026

View reviewed changes

xyao-nv added 2 commits October 23, 2026 17:05

Split SimReady backends and update Central example

48da72d

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

Remove Central scene capture from documentation

512bebf

Signed-off-by: Xinjie Yao <xyao@nvidia.com>

	With ``--enable_simready_search --simready_source simready-central``, search
	With ``--enable_simready_search --simready_sources simready-central``, search

		@@ -0,0 +1,82 @@
		# Copyright (c) 2026, The Isaac Lab Arena Project Developers (https://github.com/isaac-sim/IsaacLab-Arena/blob/main/CONTRIBUTORS.md).

		_logger = logging.getLogger(__name__)


		def search_central_objects(phrase: str, *, max_results: int) -> list[SimReadyObjectCandidate]:

Conversation

xyao-nv commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this change?

What changes?

Usage example

Validation

Example

Uh oh!

greptile-apps Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

arena-review-bot Bot commented Oct 23, 2026

🤖 Isaac Lab-Arena Review Bot

Summary

Design, Boundaries & Scope

Findings

Test Coverage and CI Cost

Verdict

Uh oh!

qianl-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

xyao-nv commented Oct 22, 2026 •

edited

Loading

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading