Skip to content

fix(torch): honor handle streams in generated operators - #880

Open
voltjia wants to merge 3 commits into
masterfrom
fix/torch-backend-stream
Open

fix(torch): honor handle streams in generated operators#880
voltjia wants to merge 3 commits into
masterfrom
fix/torch-backend-stream

Conversation

@voltjia

@voltjia voltjia commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Installs the native stream stored in an InfiniOps Handle as PyTorch's current stream before generated ATen and linked Torch calls, then restores the previous device and stream.
  • Declares the shared C10<kDev> template in src/torch/c10.h and keeps each backend specialization in src/torch/<platform>/c10.h, following the existing platform-oriented C10 design without a separate stream bridge.
  • Centralizes nullable handle-stream translation in C10<kDev>::GetStream, so generated wrappers, linked adapters, and the hand-written FlashAttention wrapper always construct the platform's native PyTorch stream guard.
  • Implements the Hygon bridge with its PyTorch fork's HIP C10 API and keeps the HIP-only implementation isolated from InfiniRT's DTK CUDA compatibility headers.
  • Adds the required vendor Torch headers, libraries, runtime paths, generator assertions, and current-stream versus explicit-handle regression coverage.

Motivation

Generated ATen backends received the InfiniOps handle stream through stream_, but they invoked PyTorch without installing that stream as the framework's current stream. PyTorch work could therefore run on a different stream from the caller-selected InfiniRT stream and violate the caller's ordering assumptions. Linked Torch adapters already modeled device-specific C10 behavior, so the shared implementation should retain that template structure and live under the common Torch backend rather than under linked.

No issue is linked.

Type of Change

  • feat - New feature, operator, or platform.
  • fix - Bug fix.
  • perf - Performance improvement without a behavioral change.
  • refactor - Code restructuring without a behavioral change.
  • test - Adding or fixing tests only.
  • docs - Documentation only.
  • build / ci - Build system or CI configuration.
  • chore - Tooling, formatting, or other non-code changes.
  • Breaking change.

Platforms Affected

  • CPU (WITH_CPU).
  • NVIDIA (WITH_NVIDIA).
  • Iluvatar (WITH_ILUVATAR).
  • Hygon (WITH_HYGON).
  • MetaX (WITH_METAX).
  • Cambricon (WITH_CAMBRICON).
  • Moore (WITH_MOORE).
  • Ascend (WITH_ASCEND).
  • PyTorch C++ bindings (WITH_TORCH).
  • Build system, CMake, or CI.
  • Python bindings or user-facing API.

Smoke Test Result

Final validation SHA: 5d4f0dc1162f1852c47bc585e9a3e9bd88a5eefc.

Final-SHA generator and architecture tests:
45 passed in 3.32s.

Final-SHA NVIDIA main workflow:
121 passed, 28 skipped in 17.50s.

Final-SHA Ascend main workflow:
107 passed, 31 skipped in 23.33s.

Final-SHA pull-request CI matrices:
The main and shadow workflows passed on NVIDIA, Iluvatar, MetaX, Cambricon, Moore, and Ascend.

Final-SHA formatting and documentation checks:
Ruff check and format passed twice, Clang Format passed twice, and the documentation build passed.

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
CPU Yes Covered by generated and vendor smoke suites. The CPU specialization directly defines a no-op StreamGuard and does not introduce a separate alias type.
NVIDIA Yes Main and shadow CI passed on the final SHA. The main workflow reported 121 passed and 28 skipped. The rebase also migrated the newly added moe_wna16_marlin_gemm linked adapter to the shared C10 implementation.
Iluvatar Yes Main and shadow CI passed on the final SHA. The CUDA-compatible C10 specialization uses InfiniRT's native stream type and the PyTorch fork's CUDA C10 stream API.
Hygon Yes The focused Torch object target compiled and linked with --no-undefined in the CI base image. The implementation uses c10::hip::getCurrentHIPStream and c10::hip::getStreamFromExternal from an isolated source file. Hygon hardware runtime testing was not available, and unrelated current InfiniRT/DTK build issues prevent a full smoke result.
MetaX Yes Main and shadow CI passed on the final SHA. The CUDA-compatible specialization uses the PyTorch fork's CUDA C10 stream API.
Cambricon Yes Main and shadow CI passed on the final SHA. The C10 specialization uses the torch_mlu stream API.
Moore Yes Main and shadow CI passed on the final SHA. The C10 specialization uses the torch_musa stream API from the explicitly linked libmusa_python.
Ascend Yes Main and shadow CI passed on the final SHA. The main workflow reported 107 passed and 31 skipped. The current CI torch_npu lacks external-stream support, so this validates the compatibility path that preserves the selected current PyTorch stream.
Validation details.
Ruff 0.15.22 check: passed.
Ruff 0.15.22 format --check: passed.
Clang Format: passed.
Documentation build: passed.
Generator and architecture tests: 45 passed.
NVIDIA main workflow: 121 passed, 28 skipped.
NVIDIA shadow workflow: passed.
Ascend main workflow: 107 passed, 31 skipped.
Ascend shadow workflow: passed.
Iluvatar main and shadow workflows: passed.
MetaX main and shadow workflows: passed.
Cambricon main and shadow workflows: passed.
Moore main and shadow workflows: passed.
Hygon focused Torch object compile and no-undefined link: passed.
NVIDIA pre-rebase generated and linked stream regressions: 4 passed, 2 skipped.

Benchmark / Performance Impact

N/A. This is a stream-ordering correctness fix and an ownership refactor, and no performance benchmark was run.

Notes for Reviewers

InfiniRT remains the source of truth for each backend's native stream type through Runtime<kDev>::Stream. Vendor PyTorch APIs are used only by the corresponding C10<kDev> specialization to translate that native stream into PyTorch's thread-local current-stream state.

src/torch/c10.h contains only the template declaration. Each platform owns its specialization under src/torch/<platform>/c10.h, matching the original C10 organization. The old src/linked/torch/c10.h, backend-local linked C10 headers, and the centralized src/torch/stream_.h bridge have been removed.

Generated and linked call sites construct a native platform StreamGuard unconditionally from C10<kDev>::GetStream. A null handle is translated to the target device's selected current PyTorch stream, while a non-null handle is translated through the vendor's external-stream API. This leaves native C10 guard behavior unchanged and removes repeated null checks from call sites.

The Ascend external-stream API was introduced together with NPUStreamUtils.h, so that header is used as the compile-time capability signal. Older torch_npu releases keep the selected current PyTorch stream for both null and non-null handles; inheriting a non-null InfiniRT stream requires a release that provides c10_npu::getStreamFromExternal.

Hygon is not a no-op. Its PyTorch fork exports HIP C10 stream symbols, but directly combining the HIP headers with InfiniRT's DTK CUDA compatibility headers causes runtime type redefinitions. src/torch/hygon/c10.cc therefore performs only the HIP stream conversion behind a generic c10::Stream declaration, and the normal c10::StreamGuard remains visible in the platform specialization.

@voltjia
voltjia requested a review from a team August 4, 2026 11:36
@voltjia
voltjia force-pushed the fix/torch-backend-stream branch 11 times, most recently from f6e6f27 to bbc7b58 Compare August 10, 2026 06:42
@voltjia
voltjia force-pushed the fix/torch-backend-stream branch from bbc7b58 to ce71ffc Compare August 10, 2026 08:03
@voltjia
voltjia force-pushed the fix/torch-backend-stream branch from ce71ffc to 5d4f0dc Compare August 10, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant