[CI] Mark kit-less tests with the kitless marker by hujc7 · Pull Request #8609 · isaac-sim/IsaacLab

hujc7 · 2026-10-15T16:28:24Z

Summary

Mark the tests that pass without Isaac Sim as kitless, so a CI run or an external pipeline can select them with -m kitless (or CI_MARKER=kitless through tools/conftest.py).

Description

Adds pytest.mark.kitless to 66 test files in isaaclab_newton, isaaclab_ov and isaaclab_tasks; 79 files carry it after this change.
Test files only: no workflow, runner or library change.

Not complete yet. A run of every test file in the Kit-less image found about 257 more unmarked files that pass without Kit, mostly in isaaclab and isaaclab_tasks. They will be added here after a fresh run at this head, together with fixes for the marked files that do not pass cleanly there.

Type of change

New feature (non-breaking change which adds functionality)

Release backport

Backport this pull request to the active release branch after it merges into develop

Checklist

Docker and GPU tests run on demand. Push the commits you want tested, then
comment run-ci on the pull request.

I have read and understood the contribution guidelines
I have run the pre-commit checks with uv run isaaclab --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
I have added my name to the CONTRIBUTORS.md or my name already exists there

A workflow_dispatch input forwards one registered marker to every test job. A job with no marked file passes with an empty report, and an unregistered marker fails the job before selection.

…er-selection

hujc7 · 2026-10-15T16:28:32Z

run-ci

…er-selection

hujc7 · 2026-10-15T19:09:01Z

run-ci

greptile-apps · 2026-10-15T21:49:08Z

Confidence Score: 4/5

[Medium impact] Adds pytest marker filtering to CI test runs.

The PR appears safe to merge, with non-blocking feedback to test the new session-start outcomes.

Findings

Marker outcomes remain untested ▶

Summary

Adds a manual marker input to select tests across the CI test jobs.

Manual CI runs can select test files by one pytest marker.
Kit starts without pytest’s marker filter in its arguments.

Acknowledged scope from hujc7: empty jobs still pull images and start containers; skipping that setup is deferred. performance-smoke intentionally ignores the marker.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Manual marker input] --> B[Each run-package-tests step]
    B --> C[CI_MARKER]
    C --> D{Registered marker?}
    D -->|No| E[Fail job]
    D -->|Yes| F[Apply job filters and marker scan]
    F --> G{Matching files?}
    G -->|No| H[Write empty report and pass]
    G -->|Yes| I[Run files with pytest marker]
    I --> J[Remove marker arguments before Kit starts]

_{Reviews (1) · Last reviewed commit: "Merge remote-tracking branch 'upstream/d..." · Reviewed by Greptile}

greptile-apps · 2026-10-15T21:49:11Z

+def test_ci_marker_must_be_a_registered_marker_name():
+    """A marker run matches files by name, so an expression or a misspelling must not reach selection."""
+    orchestrator = _load_orchestrator_module()
+
+    registered = orchestrator._registered_markers(str(TOOLS_DIR.parent))
+
+    assert {"kitless", "arm_ci", "isaacsim_ci"} <= registered
+    assert "kitless and not rendering" not in registered
+    assert "kitles" not in registered


Marker outcomes remain untested

The new test checks _registered_markers(), but never checks that pytest_sessionstart() rejects an invalid CI_MARKER. Removing that guard would leave this test passing. The new empty-selection branch also has no regression test.

Add session-start tests that assert an invalid marker exits with code 1 before selection, and a registered marker with no matching files writes an empty report and exits with code 0.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fixed in 44be631: replaced the parser test with two tests that call pytest_sessionstart directly. test_unregistered_ci_marker_fails_before_selection asserts exit code 1 before any file selection, for a misspelling and an expression; it fails when the guard is removed. test_registered_ci_marker_without_marked_files_passes_empty asserts exit code 0 and that the empty report is written.

isaaclab-review-bot

Isaac Lab Review Bot

Needs attention: Design and architecture (1), Style consistency (1), Test quality (1). Other reviewed categories look good.

PR description: The body matches the diff and provides adequate linked CI measurements, but “registered-marker check” overstates validation because the added test never exercises rejection.
Design and architecture: Marker validation duplicates pytest’s resolved registry by reparsing pyproject.toml; use session.config.getini("markers") and remove _registered_markers.
Scope and over-engineering: No over-engineering issues.
API: API changes look good.
Compatibility and deprecation: No breaking changes identified.
Implementation: Implementation looks good.
Style consistency: Shadow Hand reverses the marker order used by sibling rendering files; place arm_ci before kitless for consistent module metadata.
Test quality: The new orchestrator test checks a private parser and copied marker inventory, not unregistered-CI_MARKER rejection; replace it with a boundary test of the failure path.

Minor fixes needed. Posted 3 inline findings.

Automated review; human maintainers own approval decisions.

isaaclab-review-bot · 2026-10-15T22:02:30Z

    assert orchestrator._resolve_startup_deadline("test_sample.py", timeout=10000, is_cold_cache_test=False) == 120
+
+
+def test_ci_marker_must_be_a_registered_marker_name():


🟡 Warning · Test Quality — Test bypasses the added marker guard

This asserts a private parser against a copied marker inventory, and the two negative assertions can never hold a marker name. It passes even if the new unregistered-CI_MARKER rejection is deleted. Assert the rejection itself…

Fixed in 44be631: replaced the parser test with two tests that call pytest_sessionstart directly. test_unregistered_ci_marker_fails_before_selection asserts exit code 1 before any file selection, for a misspelling and an expression; it fails when the guard is removed. test_registered_ci_marker_without_marked_files_passes_empty asserts exit code 0 and that the empty report is written.

isaaclab-review-bot · 2026-10-15T22:02:30Z

 )

-pytestmark = pytest.mark.arm_ci
+pytestmark = [pytest.mark.kitless, pytest.mark.arm_ci]


🟡 Warning · Style Consistency — Marker order reversed versus siblings

The sibling rendering files changed here append the new marker last ([pytest.mark.arm_ci, pytest.mark.kitless]). Use the same order for consistency across the group.

Fixed in 44be631: now [pytest.mark.arm_ci, pytest.mark.kitless], matching the sibling rendering files.

isaaclab-review-bot · 2026-10-15T22:02:30Z

    return grouped


+def _registered_markers(workspace_root: str) -> set[str]:


🟡 Warning · Design Architecture — Reuse pytest's resolved marker registry

_registered_markers re-parses pyproject.toml and hardcodes the [tool.pytest.ini_options] layout, duplicating configuration pytest already loads. Use session.config.getini("markers") in pytest_sessionstart and delete the helper.

Fixed in 44be631: the guard now reads session.config.getini("markers") and _registered_markers is gone. The runner starts pytest from the repository root (.github/actions/run-tests/run_tests.sh), so the root pyproject.toml markers are the ones pytest loads.

Read registered markers from pytest's loaded config instead of re-parsing pyproject.toml, test the rejection and empty-selection exits through pytest_sessionstart, and match the sibling marker order in the Shadow Hand rendering test.

…er-selection

hujc7 · 2026-10-15T23:11:53Z

run-ci

ben-johnston-nv · 2026-10-19T21:36:15Z

For the follow-up in your description ("Jobs with nothing to run still pay image pull and setup"): I put up #8744, stacked on this PR.

Load Config applies the same test file selection as tools/conftest.py to each job's run-package-tests inputs. The selection now lives in a shared module, tools/test_selection.py. Jobs with no marked file are then skipped in their if: before they wait for a runner or pull the image. For kitless it keeps 9 of the 26 jobs, which matches job for job what ran and what passed empty in your run 38775752803.

On runners that pull the image slowly, each skipped job saves roughly 10 minutes. If it's easier, feel free to pull the commit into this PR instead.

…selection # Conflicts: # .github/workflows/build.yaml # source/isaaclab_ov/test/sensors/test_ray_caster.py # source/isaaclab_ov/test/test_ovrtx_compat.py # source/isaaclab_ov/test/test_ovstage_compat.py # source/isaaclab_ov/test/test_runtime_imports.py

hujc7 · 2026-10-23T19:56:16Z

run-ci

hujc7 added 3 commits October 15, 2026 02:22

Mark the test files that run without Kit as kitless

ed2ab7d

Select a whole CI run by pytest marker

c0a249b

A workflow_dispatch input forwards one registered marker to every test job. A job with no marked file passes with an empty report, and an unregistered marker fails the job before selection.

Merge remote-tracking branch 'upstream/develop' into jichuanh/ci-mark…

22ba1d1

…er-selection

github-actions Bot added isaac-lab Related to Isaac Lab team infrastructure labels Oct 15, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 15, 2026

This was referenced Oct 15, 2026

[DO NOT MERGE][CI] Time a kitless marker run for #8609 #8611

Closed

[DO NOT MERGE][CI] Time an OV-only marker run for #8609 #8612

Closed

hujc7 added 2 commits October 15, 2026 10:14

Strip every pytest marker expression from Kit's arguments

a56e01d

Merge remote-tracking branch 'upstream/develop' into jichuanh/ci-mark…

18f2c0f

…er-selection

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 15, 2026

hujc7 marked this pull request as ready for review October 15, 2026 21:45

hujc7 requested a review from a team October 15, 2026 21:45

hujc7 requested review from AntoineRichard, StafaH, fatimaanes, huidongc, kellyguo11, marcodiiga, myurasov-nv, ooctipus, pbarejko and rilei-nvidia as code owners October 15, 2026 21:45

greptile-apps Bot reviewed Oct 15, 2026

View reviewed changes

isaaclab-review-bot Bot reviewed Oct 15, 2026

View reviewed changes

hujc7 added 2 commits October 15, 2026 16:11

Test the CI marker guard through session start

44be631

Read registered markers from pytest's loaded config instead of re-parsing pyproject.toml, test the rejection and empty-selection exits through pytest_sessionstart, and match the sibling marker order in the Shadow Hand rendering test.

Merge remote-tracking branch 'upstream/develop' into jichuanh/ci-mark…

5edc40b

…er-selection

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 15, 2026

ben-johnston-nv mentioned this pull request Oct 19, 2026

Skip marker jobs with no marked test file before they start #8744

Draft

6 of 8 tasks

hujc7 added 2 commits October 23, 2026 10:15

Limit the change to kitless test markers

104eef7

hujc7 changed the title ~~[CI] Select a whole CI run by pytest marker~~ [CI] Mark kit-less tests with the kitless marker Oct 23, 2026

isaaclab-bot Bot added ci:run-docker Trigger the on-demand Docker and GPU CI workflow and removed ci:run-docker Trigger the on-demand Docker and GPU CI workflow labels Oct 23, 2026

		assert orchestrator._resolve_startup_deadline("test_sample.py", timeout=10000, is_cold_cache_test=False) == 120


		def test_ci_marker_must_be_a_registered_marker_name():

		return grouped


		def _registered_markers(workspace_root: str) -> set[str]:

Conversation

hujc7 commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Type of change

Release backport

Checklist

Uh oh!

hujc7 commented Oct 15, 2026

Uh oh!

hujc7 commented Oct 15, 2026

Uh oh!

greptile-apps Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

hujc7 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

Isaac Lab Review Bot

Uh oh!

isaaclab-review-bot Bot Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

hujc7 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

hujc7 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

isaaclab-review-bot Bot Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

hujc7 Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

hujc7 commented Oct 15, 2026

Uh oh!

ben-johnston-nv commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hujc7 commented Oct 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hujc7 commented Oct 15, 2026 •

edited

Loading

greptile-apps Bot commented Oct 15, 2026 •

edited

Loading

ben-johnston-nv commented Oct 19, 2026 •

edited

Loading