[hujc7]

Summary

• Windows gets the same CI shape as ARM: the locked uv environment runs the windows_ci marker tests through the shared orchestrator (tools/conftest.py), then the shared Cartpole smoke, then camera Cartpole trained on every physics and renderer pairing. Windows has no Docker, so the job syncs the environment on the runner and keeps it there between jobs, and restores the Linux jobs' Warp and shader caches.
• The installation suite (install_ci) gets a Windows job beside x86 and ARM, run natively.
• The checks the workflow used to run as inline scripts are now test files.

Description

Part 3 of the cross-platform CI series (Part 1 #5695 and Part 2 #5698 are merged). Runs on the [self-hosted, gpu-windows] runners.

1. Jobs

#	Job	Mirrors	Runs
1.1	Windows CI / windows-ci	ARM CI / Build & Test	uv sync --frozen --extra test --extra isaacsim --extra ov into an environment kept in the runner's tool cache; restores the Warp kernel, Kit shader and OVRTX shader caches; CI_MARKER=windows_ci pytest tools; install_ci/misc/cartpole_training_smoke.py on Newton, as on ARM; install_ci/misc/cartpole_backend_combinations_smoke.py: Isaac-Cartpole-Camera-Direct on each of the 10 pairings of {Newton MJWarp, Newton Kamino, Isaac Sim PhysX, OvPhysX} x {Newton, OVRTX, Isaac RTX} renderer that can share a process
1.2	Installation Tests / Installation Tests (Windows)	Installation Tests (x86) / (ARM)	tools/run_install_ci.py native on a wheel built with Git's bash: fresh uv sync --locked (base, isaacsim) with Cartpole training on Newton and Isaac Sim PhysX, wheel installs, lock resolution

2. windows_ci tests

#	File	Checks
2.1	isaaclab/test/utils/test_{configclass,dict,episode_data,hdf5_dataset_file_handler}.py	Existing unit tests, now marked

3. Orchestrator on Windows

tools/conftest.py keeps its Linux behavior and gains three Windows paths:

• Output: Windows cannot select() on pipes, so each pipe is peeked (PeekNamedPipe) before a read.
• Kill: Windows has no process groups, so a hung test's tree is killed with taskkill /T.
• Report names: the drive colon is dropped; Windows would read it as an alternate data stream.

4. Design notes

• uv: astral-sh/setup-uv v10.2.0 with uv 0.12.9, the uv the Docker images pin. v6, which the Linux workflows and the install-CI actions use, crashes on exit on these runners once uv is in the tool cache (Node/libuv assertion error on Windows self-hosted GitHub Actions runner astral-sh/setup-uv#686).
• Environment: checkout cleans the workspace, so the environment lives in the runner's tool cache. A reused runner re-syncs only what uv.lock changed and keeps the packages' compiled bytecode; a failed sync rebuilds it once.
• Caches: the Linux test jobs' Warp kernel cache and OVRTX shader caches (Kit's and the kit-less renderer's), each in a Windows collection: pull requests restore the newest snapshot, pushes to protected branches save what changed. The shared actions run bash, which resolves to WSL on these runners, so .github/scripts/windows_ci_caches.ps1 rebuilds the same keys (with the installed Isaac Sim and ovrtx versions) and reports. The caches sit on the runner beside the environment, so a reused runner keeps what earlier pull requests compiled; a save clears them first and publishes only the restored snapshot plus what that push compiled. Pull requests also run in the kept environment and uv cache, so a save reinstalls the environment from a uv cache of its own. uv, the Python it installs and the runner's own files still carry over from pull-request jobs; only runners kept apart from pull requests avoid that. Warp reports its directory with the \\?\ prefix on Windows, which tools/verify_warp_cache.py rejects, so the script checks the path itself.
• Isaac Sim version: the Windows installation tests install isaacsim 6.2.0.0 from pypi.nvidia.com, as the lock gives users.

5. Test plan

• pre-commit, the changelog check, and the orchestrator's unit tests (tools/test_crash_journal.py, tools/test_file_scheduler.py) pass.
• On Linux, CI_MARKER=windows_ci pytest tools selects the four marked files and passes them (62 tests).
• On a Windows 11 VM (L40), the Newton Cartpole smoke cases pass against an environment outside the checkout: 209 s with a cold Warp kernel cache, 69 s warm.
• Camera Cartpole on every pairing, before the caches were wired: 10/10 trained; 19.5 min on a fresh runner (5.5 min sync, first Newton, OVRTX and Isaac RTX runs 132 / 235 / 180 s), 4.3 min on a reused one (17-29 s per pairing).
• windows-ci green with the combinations and caches (run 40006339958): 5.0 min on a reused runner (sync 0 s, combinations 220 s); 17.3 min on a runner with nothing installed or cached, before main has saved a Windows cache (sync 245 s, combinations 680 s, of which the first OVRTX and Isaac RTX pairings compile shaders for 223 and 189 s). The Kit, OVRTX and Warp caches fill to 141, 119 and 75 MB.
• Still green after the push runs began rebuilding the environment (run 40043168812, 23a67be): 19.7 min on another runner with nothing installed or cached (sync 327 s, combinations 646 s); the rebuild itself runs only on pushes.
• Installation Tests (Windows) under 10 min on a reused runner, the budget windows-ci meets at 5.0 min. While iterating, TEMP commits mask jobs with no signal for this change; they are reverted before review.
• Installation Tests (Windows) green, 8/8 in 10.1 min (run 38380247601). The first run failed test_workspace_trains[isaacsim]: pytest's default base temp on the runner put Kit's DLLs past the 260-character path limit, so the job now passes --basetemp under RUNNER_TEMP.

Type of change

• New feature (non-breaking change which adds functionality)

Release backport

• Backport this pull request to the active release branch after it merges into develop

Screenshots

Not applicable.

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with uv run isaaclab --format
• Documentation changes are not required because the documented install path is unchanged
• My changes generate no new warnings
• I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[isaaclab-review-bot]

Review Summary

This PR adds experimental Windows CI jobs for perception smoke testing, building on the uv-based installation path. The approach is well-structured, with incremental jobs that progressively test more functionality.

✅ Strengths

1. Incremental approach: The workflow builds from simple torch/scipy tests → IsaacLab core install → full perception smoke
2. Cross-platform fix in AssetConverterBase: Moving from hardcoded /tmp/IsaacLab to tempfile.gettempdir() is the correct POSIX/Windows-agnostic solution
3. Proper marker recognition in AppLauncher: The windows_ci and arm_ci markers are now correctly filtered from sys.argv
4. Good use of continue-on-error: true: Appropriate for experimental jobs that should not gate merges

Architecture

• ✅ Clean separation of reusable GitHub Actions (windows-instance-state, windows-sim-paths) from the main workflow
• ✅ Consolidated single-job design prevents runner teardown between steps — smart approach for autoscaling runners
• ✅ OS-level watchdogs via Start-Process + WaitForExit instead of relying on GIL-vulnerable Python thread watchdogs
• ✅ Develop-aligned Isaac Sim builds from internal Artifactory index (authenticated)
• ✅ Perception and training smoke tests enabled (WDDM mode confirmed)

⚠️ Optional Suggestions (Non-blocking)

1. run_docker_tests: 'false' in build.yaml: The comment says "TEMP: disable heavy Linux Docker + Tests" — should be reverted before merge if not intentional.
2. Hardcoded Git Bash Path: C:\Program Files\Git\bin\bash.exe assumes standard installation. Consider fallback or documentation.

---

Update (ad92f43): Reviewed incremental diff from bae6977...ad92f43 (rebase on develop + Windows CI refinements):

Windows CI Changes:

• ✅ Perception test fully re-enabled: Both the standalone test-perception step and the perception subcase in training smoke now run. Perception added to $blocking in aggregate.
• ✅ Develop-aligned Isaac Sim install: Setup step now resolves and installs the develop-branch-aligned Isaac Sim build from the internal Artifactory index (authenticated via ISAACSIM_ARTIFACTORY_READONLY_* secrets). This replaces the previous public pip install and ensures CI tests against the same Sim as the Linux/ARM develop container.
• ✅ Wheel install adjusted: The [all] extra (which pins isaacsim==5.1.0) is no longer used during wheel validation, preventing a downgrade of the develop Sim build.
• ✅ Training smoke expanded: Step renamed to "state + perception" with increased timeout (25 min); -k "not perception" filter removed.

Non-CI changes (develop rebase): The diff includes a large number of unrelated develop-branch changes (cloner refactoring, observation pipeline improvements, new tutorials, bugfixes, etc.) that are not part of this PR's Windows CI scope. No new issues found.

Approval stands.

[isaaclab-review-bot]

Updated Review: Windows CI Pipeline (Part 3)

This is a comprehensive and well-structured PR that adds Windows CI support to IsaacLab. The implementation shows strong attention to platform-specific challenges and includes robust error handling. Below is my detailed analysis:

---

✅ Strengths

1. Solid Architecture

• Clean separation of reusable GitHub Actions (windows-instance-state, windows-sim-paths) from the main workflow
• Marker-driven test discovery (windows_ci) allows adding tests without YAML edits
• Consolidated single-job design prevents runner teardown between steps — smart approach for autoscaling runners

2. Robust Process Management

• OS-level watchdogs via Start-Process + WaitForExit instead of relying on GIL-vulnerable Python thread watchdogs
• Per-job timeouts with --timeout-method=thread (correctly avoiding Unix-only SIGALRM)
• Hard timeouts with explicit kill on hung Kit/Vulkan processes

3. Cross-Platform Improvements

• AssetConverterBase now uses tempfile.gettempdir() instead of hardcoded /tmp — correct fix
• wheel_builder/build.sh PYTHON override handles Windows git-bash (which only has python, not python3)
• HDF5 fixture uses ignore_cleanup_errors=True to handle Windows file-lock races — good catch

4. Comprehensive Environment Setup

• EULA acceptance variables properly configured
• DLL search paths correctly prepended for Vulkan ICD discovery
• Instance state reporting helps diagnose runner issues (GPU presence, disk space, cache sizes)

---

⚠️ Suggestions & Minor Issues

1. Temporary Docker/Tests Disable in build.yaml

run_docker_tests: 'false'  # TEMP: disable heavy Linux Docker + Tests

The comment says "revert before final review" — this should be reverted before merge. Consider adding a # TODO: tag that linters/CI can catch.

2. Hardcoded Git Bash Path

$gitBash = "C:\Program Files\Git\bin\bash.exe"
if (-not (Test-Path $gitBash)) { throw "Git Bash not found at $gitBash" }

This assumes a standard Git for Windows installation. Consider:

• Falling back to where.exe bash or checking common alternate paths (C:\Program Files (x86)\Git\bin\bash.exe)
• Or documenting this as a runner requirement

3. Perception Test Gating
The perception test depends on WDDM mode for Vulkan. If runners could be in TCC mode, consider:

• Auto-detecting driver mode via nvidia-smi output
• Adding a conditional skip rather than a CI failure

4. pytestmark Placement in test_configclass.py

from isaaclab.utils.configclass import _field_module_dir, configclass

pytestmark = pytest.mark.windows_ci
from isaaclab.utils.dict import class_to_dict, ...

The pytestmark is sandwiched between imports — while functional, it's unconventional. Consider grouping it with other module-level declarations after all imports.

5. Magic Numbers
Several timeout values are hardcoded (300000ms, 180000ms, 600s, etc.). Consider:

• Documenting why these specific values were chosen
• Using named constants or environment variables for runner-specific tuning

---

📝 Questions

1. Checkpoint Cleanup: The cartpole training smoke tests run two iterations and create checkpoints. Are these cleaned up, or will they accumulate on runners?

2. Change Detection Logic: The changes job patterns don't include .github/actions/windows-* — is this intentional? Changes to these actions wouldn't trigger the workflow for validation.

3. Python 3.12 Assumption: The user site-packages path assumes Python 3.12:

   "user site-pkgs" = "$env:APPDATA\Python\Python312\site-packages"

   Should this be dynamic based on the installed Python version?

---

🔧 Final Notes

Overall, this is high-quality CI infrastructure work. The attention to Windows-specific quirks (DLL search order, file locking, PowerShell encoding, process watchdogs) demonstrates deep platform expertise.

Before merge:

1. Revert the run_docker_tests: 'false' workaround in build.yaml
2. Consider addressing the Git Bash path hardcoding

Other items are suggestions for future improvement rather than blockers.

---

Update (ad92f43): Reviewed the new commits since bae6977.

Changes Observed

1. Perception/Rendering Steps Re-enabled ✅

   • Commit 27def394 re-enables the perception and rendering test steps in the Windows CI workflow
   • The cartpole-camera perception smoke test now exercises the full RTX/Vulkan path

2. Develop-Aligned Isaac Sim Installation ✅

   • The workflow now installs a develop-aligned Isaac Sim build from the internal Artifactory index (instead of the public pip release)
   • Uses authenticated index URLs with ISAACSIM_ARTIFACTORY_READONLY_USERNAME/PASSWORD secrets
   • The tools/resolve_isaacsim_develop.py resolver verifies the build is on the develop branch
   • Good approach to ensure Windows CI tests the same Sim as Linux/ARM develop

3. Authentication with Read-Only Secrets ✅

   • Proper credential handling with URL-encoding for special characters
   • GitHub masks secrets and uv redacts credentials when echoing index URLs

Outstanding Items

1. build.yaml Still Has run_docker_tests: 'false' ⚠️

   • The temporary disable is still in place with the comment "revert before final review"
   • This needs to be reverted before merge

2. Hardcoded Git Bash Path (unchanged from previous review)

   • Still assumes standard Git for Windows installation at C:\Program Files\Git\bin\bash.exe

Summary

The new commits address the perception/rendering testing and add proper develop-aligned Isaac Sim installation with authentication. The Windows CI workflow is now more comprehensive and aligned with the develop branch testing strategy.

Remaining before merge:

• Revert run_docker_tests: 'false' in build.yaml

---

Update (5c04d23): Reviewed incremental changes since ad92f43.

Changes in This Push

1. Artifactory URL Migration ✅
   • All internal Artifactory URLs updated from urm.nvidia.com to artifactory.nvidia.com
   • PyPI index paths updated from sw-isaacsim-pypi/simple → sw-isaacsim-pypi-local/simple and ct-omniverse-pypi/simple → ct-omniverse-pypi-local/simple
   • This affects both the uv pip install index URLs and the resolve_isaacsim_develop.py --index-url argument
   • Standard infrastructure migration from the legacy URM hostname to the canonical Artifactory hostname with corrected local repository paths

Assessment

This is a straightforward infrastructure-only change — no logic modifications, no new features. The URL pattern change (-local/simple suffix) suggests these now point to the correct local Artifactory repositories rather than virtual aggregators.

No new concerns introduced. Previously noted items still apply:

• build.yaml still has run_docker_tests: 'false' that needs reverting before merge
• Hardcoded Git Bash path remains unchanged

---

Update (5941931): Reviewed incremental changes since 5c04d23.

Changes in This Push

1. Fork PR Secret Handling — Graceful Degradation ✅

   • Previously the workflow would hard-fail (throw) when ISAACSIM_ARTIFACTORY_READONLY_* secrets were not set
   • Now emits a GitHub Actions ::warning:: annotation and proceeds unauthenticated
   • This allows fork PRs (which don't receive repository secrets) to still exercise URL/DNS reachability
   • Auth-dependent failures will surface later in in-repo runs where secrets are present
   • Good approach — matches standard CI best practice for fork-safe workflows

2. Artifactory URL Path Correction ✅

   • Removed /api/pypi/ prefix and /simple suffix from all Artifactory index URLs
   • Before: https://...artifactory.nvidia.com/artifactory/api/pypi/sw-isaacsim-pypi-local/simple
   • After: https://...artifactory.nvidia.com/artifactory/sw-isaacsim-pypi-local/
   • Same change applied to ct-omniverse-pypi-local and the resolve_isaacsim_develop.py --index-url argument
   • This likely aligns with the actual repository layout on the new artifactory.nvidia.com host (non-virtual repos don't use the /api/pypi/.../simple PEP 503 endpoint)

Assessment

Both changes are operational improvements with no logic risk:

• The fork PR fix unblocks external contributors from running the workflow
• The URL path fix ensures correct repository resolution on the migrated Artifactory instance

No new concerns introduced. Previously noted items still apply:

• build.yaml still has run_docker_tests: 'false' that needs reverting before merge
• Hardcoded Git Bash path remains unchanged

---

Update (77685d1): Reviewed incremental changes since 5941931.

Changes in This Push

1. Network Reachability Diagnostic Step ✅
   • Added a new "Probe index reachability (diagnostic)" step in the Windows CI workflow
   • Uses if: always() and continue-on-error: true — purely diagnostic, won't block the build
   • Probes three hosts: artifactory.nvidia.com, urm.nvidia.com, pypi.nvidia.com
   • Tests both DNS resolution ([System.Net.Dns]::GetHostAddresses) and TCP 443 connectivity (Test-NetConnection)
   • Logs runner metadata: COMPUTERNAME, RUNNER_NAME, proxy environment variables, DNS server addresses
   • This decouples network/DNS diagnostics from pip failures — useful for debugging transient CI infra issues

Assessment

This is a non-blocking diagnostic addition. It helps isolate whether Windows CI failures stem from network reachability issues vs. package availability, especially given the recent Artifactory URL migrations. The step runs unconditionally (always()) so it will capture state even on prior step failures.

No new concerns introduced. Previously noted items still apply:

• build.yaml still has run_docker_tests: 'false' that needs reverting before merge
• Hardcoded Git Bash path remains unchanged

[hujc7]

Superseded by #6086 — re-opened from a branch in this repo (not a fork) so the Windows CI job receives the ISAACSIM_ARTIFACTORY_READONLY_* org secrets (fork PRs don't get workflow secrets). Closing this one.

[hujc7]

Reopened — #6086 (same-repo variant) is closed. Confirmed the fork rule was not the blocker: even on an in-repo PR the ISAACSIM_ARTIFACTORY_READONLY_* secrets resolve empty, so they aren't scoped to isaac-sim/IsaacLab. Fix is on the secret placement (scope to this repo like NGC_API_KEY) or bake the read-only creds into the Windows self-hosted runner; no change needed in this PR's wiring. This branch also now has develop merged in (no longer conflicting).

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes affect Windows CI and installation checks on GPU runners.

The PR appears safe to merge, with a non-blocking gap in tests for Windows output capture and timeout cleanup.

Findings

1.  Windows timeout paths lack tests ▶

Summary

Adds native Windows installation tests and a Windows CI workflow using the shared test runner.

• Windows CI now runs marked tests and Cartpole smoke checks on GPU runners.
• Windows now builds and tests an Isaac Lab wheel on a native runner.
• Windows CI checks ovrtx rendering with USD and MaterialX loaded in either order.
• Windows CI checks Cartpole camera output through Kit's RTX renderer.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Relevant changes] --> B[Windows CI]
  A --> C[Windows installation tests]
  B --> D[Sync locked environment]
  D --> E[Run windows_ci tests]
  E --> F[Run Cartpole smoke tests]
  C --> G[Build wheel with Git Bash]
  G --> H[Run native installation suite]
  F --> I[Upload reports]
  H --> I

Loading

_{Reviews (1) · Last reviewed commit: "Keep the Windows installation venvs unde..." · Reviewed by Greptile}

[greptile-apps]

Windows timeout paths lack tests

The new Windows pipe and process-kill paths have no focused tests. The existing test that runs a hung child is POSIX-only, so it cannot catch Windows failures in output capture or timeout cleanup. This is a non-blocking coverage gap: ordinary passing tests do not exercise cleanup after a timeout.

Add Windows tests for short output followed by silence, pipe closure, and killing a hung child with descendants, including the taskkill fallback.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[hujc7]

run-ci

[isaaclab-review-bot]

Isaac Lab Review Bot

Needs attention: Design and architecture (1), Style consistency (1). Other reviewed categories look good.

• PR description: Remove the unsupported claim that prior inline scripts became test files; the diff adds one coexistence test and modifies an existing smoke, while cited CI timings include run IDs.
• Design and architecture: Windows skips post-exit process-tree cleanup, allowing orphaned Kit processes to accumulate; launch tests in a Windows job object that terminates descendants on close.
• Scope and over-engineering: No over-engineering issues.
• API: API changes look good.
• Compatibility and deprecation: No breaking changes identified.
• Implementation: Implementation looks good.
• Style consistency: The new workflow uses .yaml while adjacent workflows use .yml; rename it to windows-ci.yml and update its detect-changes pattern.
• Test quality: Test quality looks good.

Minor fixes needed. Posted 2 inline findings.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Style Consistency — Workflow uses .yaml, not .yml

Adjacent workflows use .yml (install-ci.yml, and arm-ci.yml named in this file's header). Rename to windows-ci.yml and update the detect-changes pattern on line 65.

[hujc7]

Keeping .yaml. The workflows directory uses both extensions (10 .yaml, 13 .yml), and build.yaml, which runs the x86 test jobs, is .yaml.

[isaaclab-review-bot]

🟡 Warning · Design Architecture — Windows never reaps orphaned simulator children

On Windows pgid is None, so the post-exit cleanup that kills orphaned Kit processes on Linux is skipped entirely; self-hosted runners accumulate them. Launch the child in a job object that terminates on close.

[hujc7]

Not adding a job object here. A test that times out is killed with taskkill /T while its root is still alive, so its tree goes with it. Children that outlive a root that exited normally are reaped by the runner at job end: the Windows job logs show its "Cleaning up orphan processes" step, and in this PR's runs it has found nothing to terminate.