build(data_collection): make TORCH_CUDA_ARCH_LIST a --build-arg (unblocks Blackwell / sm_120) - #191
Open
InfraestruturaRobotec wants to merge 1 commit into
Conversation
The dockerfile pins TORCH_CUDA_ARCH_LIST=8.9 (Ada / RTX 4090D) immediately before building cuRobo, so the resulting image cannot run cuRobo on any other architecture without editing the dockerfile. On Blackwell (RTX 50-series, sm_120) this surfaces as CUDA error: no kernel image is available for execution on the device from curobo/curobolib/kinematics.py on the first kinematics call. Confirmed with cuobjdump --list-elf: the shipped extensions contain sm_89 only. Promoting the pin to an ARG keeps the existing default byte-for-byte and lets users override it at build time: docker build --build-arg TORCH_CUDA_ARCH_LIST="8.9;12.0+PTX" ... No behaviour change for anyone who does not pass the flag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Promote
TORCH_CUDA_ARCH_LISTinsource/data_collection/dockerfilefrom ahard-coded
ENVto anARGwith the same default, so it can be overridden atbuild time.
No behaviour change for anyone who does not pass the flag — the default is
byte-for-byte what it was.
Why
The pin sits immediately before cuRobo is built, so the resulting image can only
run cuRobo on Ada. On Blackwell (RTX 50-series,
sm_120) every kinematics callfails at the first launch:
Confirmed with
cuobjdump, the shipped extensions containsm_89only:With this change, a Blackwell user builds with:
docker build --build-arg TORCH_CUDA_ARCH_LIST="8.9;12.0+PTX" \ -f source/data_collection/dockerfile source/data_collectionand all five cuRobo extensions carry
sm_120.Tested on
RTX 5090 (
sm_120), driver 580.126.09, torch 2.7.0+cu128, Isaac Sim 5.1,cuRobo v0.7.6. After rebuilding with the override,
MotionGen.plan_singlesucceeds and a full
geniesim autocollectdata-collection run completes withzero CUDA errors.
Note for 50-series users beyond this PR
This change is necessary but not sufficient on
sm_120. Two further things areneeded, and neither belongs in this PR:
--no-depswhen reinstalling cuRobo, otherwise pip re-resolves itsdependency tree and replaces Isaac Sim's bundled
torch 2.7.0+cu128with aCUDA-13 stack, breaking the simulator.
-maxrregcount=64in cuRobo's own nvcc args — the L-BFGS kernel then hitstoo many resources requested for launchbecausesm_120codegen exceedsthe 64K registers-per-block limit at
threadsPerBlock = v_dim. Filedupstream as L-BFGS kernel: 'too many resources requested for launch' on Blackwell (sm_120) — register pressure, not shared memory NVlabs/curobo#716.
Happy to add a short README note pointing 50-series users at those if that would
be useful.