Skip to content

feat(auto-tuing): add a new auto-tuning system - #879

Open
mingdaw689 wants to merge 3 commits into
InfiniTensor:masterfrom
mingdaw689:feat/auto-tuning
Open

feat(auto-tuing): add a new auto-tuning system#879
mingdaw689 wants to merge 3 commits into
InfiniTensor:masterfrom
mingdaw689:feat/auto-tuning

Conversation

@mingdaw689

@mingdaw689 mingdaw689 commented Aug 4, 2026

Copy link
Copy Markdown

Summary

This PR implements runtime auto-tuning for InfiniOps operators, enabling automatic selection of the fastest implementation based on live benchmarking.

  • src/tuning.h / src/tuning.cc — Core tuning infrastructure: TuningSignature extracts shape/dtype/scalars from operator arguments via C++17 fold expressions; TuningManager singleton handles lookup, recording, and persistence to tuning.json with mutex-guarded concurrent access
  • src/tuning_utils.h — Helper utilities: operator name extraction from template types via __PRETTY_FUNCTION__, environment variable parsing for warmup/repeat counts, and first-tensor device type inference
  • src/operator.h — Added ResolveConfigOnline<Key>() called at Operator::Call entry: queries cache, benchmarks candidates via BenchmarkImplementation<Key>() on miss, records winner; added detail::SyncDevice() for accurate GPU timing and detail::ExtractOperatorName<Key>() for runtime operator identification
  • src/config.h — Added auto_select_ member and accessor; set_implementation_index() now sets auto_select_ = false to distinguish user-specified indices from auto-selection
  • src/CMakeLists.txt — Added tuning.cc to build (unconditional compilation)
  • scripts/generate_wrappers.py — Modified Python binding generation: only calls set_implementation_index() when user explicitly passes the parameter, preserving auto_select_=true for tuning; added TuningManager::Instance().LoadTuningCache() call in PYBIND11_MODULE to load existing records at import time
  • First call with a new input shape benchmarks all active implementations (warmup + repeat), selects the fastest, and writes to tuning.json
  • Subsequent calls with the same shape read from cache (zero overhead)
  • Single-implementation operators record their index without benchmarking
  • Explicit implementation_index parameter bypasses auto-tuning entirely

Motivation

In the operator library, the same operator can have different underlying implementations. For instance, vendors can provide multiple interfaces, and there are also various algorithms to choose from for handwriting. Therefore, for the same operator, how to select the backend with the best performance in different situations becomes a problem.

Type of Change

  • [√] feat — new feature / new operator / new platform
  • fix — bug fix
  • perf — performance improvement (no behavioral change)
  • refactor — code restructuring without behavior change
  • test — adding or fixing tests only
  • docs — documentation only
  • build / ci — build system or CI configuration
  • chore — tooling, formatting, or other non-code changes
  • Breaking change (requires a ! in the Conventional Commits prefix or a BREAKING CHANGE: footer)

Platforms Affected

  • CPU (WITH_CPU)
  • [√] NVIDIA (WITH_NVIDIA)
  • Iluvatar (WITH_ILUVATAR)
  • MetaX (WITH_METAX)
  • Cambricon (WITH_CAMBRICON)
  • Moore (WITH_MOORE)
  • Ascend (WITH_ASCEND)
  • PyTorch C++ bindings (WITH_TORCH)
  • Build system / CMake / CI
  • Python bindings / user-facing API

Smoke Test Result

root@master:~/InfiniOps# python -m pytest tests -m smoke -q
Running 75 items in this shard
........ss..ss....ss...............................ssssss..ss........ssssss                                                              [100%]
55 passed, 20 skipped, 18143 deselected in 43.80s

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
NVIDIA smoke passed
Iluvatar
MetaX
Cambricon
Moore
Ascend
Full `pytest` output (optional)

Benchmark / Performance Impact

Notes for Reviewers

@mingdaw689
mingdaw689 requested a review from a team August 4, 2026 11:15
@mingdaw689 mingdaw689 changed the title auto-tuning system feat(auto-tuing): add a new auto-tuning system Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant