Support torchrun-style InfiniTrain multi-process launch - #184
Open
chen2021673 wants to merge 8 commits into
Open
Support torchrun-style InfiniTrain multi-process launch#184chen2021673 wants to merge 8 commits into
chen2021673 wants to merge 8 commits into
Conversation
chen2021673
force-pushed
the
8_proc
branch
2 times, most recently
from
July 16, 2026 07:53
5314e45 to
63aefdc
Compare
Chamberlain0w0
requested changes
Jul 22, 2026
chen2021673
force-pushed
the
8_proc
branch
2 times, most recently
from
July 30, 2026 06:54
350a2fd to
c88f116
Compare
Chamberlain0w0
approved these changes
Jul 31, 2026
kilinchange
requested changes
Aug 5, 2026
chen2021673
force-pushed
the
8_proc
branch
2 times, most recently
from
August 11, 2026 02:26
50c8039 to
307253e
Compare
chen2021673
changed the base branch from
master
to
fix/dataloader-global-batches
August 11, 2026 02:27
Contributor
Author
|
重新测试： 
|
Add a dedicated 8_proc test group containing the 8-process variants of the original basic multi-GPU cases.
Track DataLoader progress by global batches so distributed ranks slice data consistently and can resume/cycle from saved consumption counts. Also scope CCL unique ID files per run, generate NCCL IDs only on the main rank, clean up run-local rendezvous files, and add DataLoader coverage.
- derive parallel state from the global world size - clarify global rank and per-node process semantics - add multi-node rank regression coverage - restore the NCCL-compatible GetUniqueId interface
- add torchrun-style --rdzv_id support - use the shared ID to isolate CCL unique-ID files - preserve automatic run ID generation for single-node runs - document rdzv_id in the multi-node example
- support infini_run with or without the optional -- separator - validate node rank bounds - use infini_run only for the new 8_proc test group - standardize torchrun environment variables and device index mapping - clarify NCCL unique ID filename helpers
- add an _8_proc suffix to multi-process test case IDs - print manual comparison commands when no baseline log directory is set
chen2021673
changed the base branch from
fix/dataloader-global-batches
to
master
August 11, 2026 07:21
kilinchange
approved these changes
Aug 11, 2026
JYMiracle305
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
本 PR 为 InfiniTrain 增加 torchrun 风格的多进程启动能力，使每个本地进程可以绑定独立 GPU，同时保留现有的单进程多线程运行方式。
主要修改
infini_run支持通过nproc_per_node启动多个子进程，并传递 torchrun 兼容的 rank 环境变量。--分隔启动器参数与训练命令。8_proc测例组，以 8 进程、每进程 1 线程运行 GPT-2 和 Llama3 的 DP、TP、TP+SP、PP 及 TP2+PP2+VPP2 配置。兼容性
原有直接启动方式保持支持：
多进程启动支持以下两种写法：
测试