[TRTLLM-17096][feat] modeling_v2: GLM-5.2 NVFP4 low-latency decode target for GB300 by xwang233 · Pull Request #20524 · NVIDIA/TensorRT-LLM

xwang233 · 2026-10-18T05:38:06Z

Dev Engineer Review

Adds GLM-5.2 NVFP4 modeling_v2 targets for SM103: TP8, TEP, and DEP.
Adds extensive CUDA, Triton, CuTe, Torch operator, dispatch, attention, MoE, speculative-decoding, KV-cache, and runtime-hook code.
Adds opt-in RuntimeHooks; default hooks remain disabled. Existing executor paths retain native behavior when hooks are absent or ineligible.
Main correctness risks are cross-rank top-k exchange, Lamport-buffer state, producer-consumer launch ordering, PDL sequencing, timeout handling, and deterministic reductions.
Main performance risks are kernel compilation, dynamic shared memory, collective workspace setup, auxiliary streams, and fallback behavior under unsupported shapes.
The change has broad integration scope. Review coverage and test adequacy remain uncertain because the supplied review assessment omitted or truncated 260 source files.
Review-finding severity counts are unavailable from the supplied evidence.

QA Engineer Review

Adds end-to-end GSM8K accuracy tests in tests/integration/defs/accuracy/modeling_v2/test_glm_moe_dsa.py for TP8, TEP4, and DEP8. Tests are restricted to SM103 and require the applicable GPU and MPI topology.
Adds CI test-list entries in:
- tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml
- tests/integration/test_lists/test-db/l0_gb300_multi_nodes_node2_gpu8.yml
The test-list entries include extended 60–90 minute timeouts and measured runtime notes.
Adds unit coverage for runtime hooks, FMHA library ordering, DSA indexer MTP/top-k helpers, and executor test stubs. These tests cover default-off behavior, fallback paths, staged metadata operations, shutdown events, and import-boundary rules.
The reported test changes replace family unit/module tests with end-to-end GSM8K gates. Engine-side runtime-hook tests remain.
The supplied evidence does not show executed test results. The full GLM suite reportedly exceeds the existing 30-minute CI timeout.
Coverage verdict: needs follow-up.

Per-File QA Perspective

cpp/tensorrt_llm/CMakeLists.txt, cpp/tensorrt_llm/kernels/CMakeLists.txt, cpp/tensorrt_llm/kernels/glmMoeDsa/CMakeLists.txt, cpp/tensorrt_llm/thop/CMakeLists.txt, and cpp/tensorrt_llm/thop/glmMoeDsa/CMakeLists.txt: Verify target inclusion, architecture selection, symbol resolution, and Torch operator linkage.
cpp/tensorrt_llm/kernels/glmMoeDsa/*: Verify SM103 support, validation errors, deterministic reductions, cross-rank synchronization, timeout behavior, PDL ordering, and invalid-input handling.
cpp/tensorrt_llm/thop/glmMoeDsa/*: Verify operator schemas, tensor validation, workspace requirements, stream behavior, fallback errors, and output shapes.
tensorrt_llm/_torch/_experimental/modeling_v2/models/glm_moe_dsa/*: Verify routing, TP8/TEP/DEP target selection, weight loading, fast-path eligibility, native fallback, dispatch installation, speculative decoding, and rank-uniform decisions.
tensorrt_llm/_torch/_experimental/modeling_v2/models/glm_moe_dsa/shared/*: Verify attention, top-k, KV append, sparse MLA, MoE, deferred normalization, draft-loop, workspace, health-check, and host-page-retention contracts.
tensorrt_llm/_torch/pyexecutor/runtime_hooks.py: Verify that all hooks remain disabled by default and that invalid hook declarations fail clearly.
tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py, py_executor.py, cuda_graph_runner.py, kv_cache/kv_cache_manager_v2.py, and speculative-decoding files: Verify native behavior when hooks are absent, staging cleanup after exceptions, CUDA-graph replay, stream synchronization, and sampler-store integration.
tensorrt_llm/_torch/attention/backends/*: Verify staged metadata updates, FMHA library precedence, reused MTP rows, and unchanged native paths.
tensorrt_llm/_torch/_experimental/modeling_v2/README.md, _router_index.py, and package initializers: Verify documented target names, shared-package import rules, and resolved-mode behavior.
tests/integration/defs/accuracy/modeling_v2/test_glm_moe_dsa.py: Covers whole-model GSM8K accuracy for TP8, TEP4, and DEP8. It is listed in the GB300 CI test lists.
tests/unittest/_torch/executor/test_runtime_hooks.py: Covers runtime-hook protocols, default-off isolation, staged/native paths, and executor lifecycle integration. No integration test-list entry is expected for this unit test.
tests/unittest/_torch/attention/sparse/dsa/test_dsa_indexer.py: Covers reused top-k rows, decode filling, and MTP shared-row storage. No integration test-list entry is expected.
tests/unittest/_torch/attention/test_fmha_manager.py: Covers prepended FMHA library ordering and unavailable-library fallback. No integration test-list entry is expected.
tests/unittest/_torch/executor/test_gpu_keepalive.py and tests/unittest/_torch/executor/test_py_executor.py: Update test stubs for new executor-hook fields. No integration test-list entry is expected.
tests/unittest/_torch/modeling_v2/test_modeling_v2_claims.py: Covers family-shared import rules. No integration test-list entry is expected.
tests/integration/test_lists/test-db/*.yml: Adds GB300 CI entries for the GLM GSM8K gates with extended timeouts. Verify that the selected topology matches each test’s required rank count.

Description

Stacked on #20797. Please merge #20797 first. #20797 holds the engine side of this work
(RuntimeHooks and its call sites) so that runtime owners can review it on its own. Until it
merges, this PR's diff still contains those files, byte-identical. Please review them in #20797:

tensorrt_llm/_torch/pyexecutor/: runtime_hooks.py, py_executor.py,
engine/runners/decoder/{runner,inputs}.py

tensorrt_llm/_torch/attention/backends/: trtllm.py, fmha/manager.py,
sparse/dsa/indexer.py

their tests: tests/unittest/_torch/executor/{test_runtime_hooks,test_py_executor,test_gpu_keepalive}.py,
tests/unittest/_torch/attention/test_fmha_manager.py,
tests/unittest/_torch/attention/sparse/dsa/{test_dsa_indexer,test_indexer_gvr_prior}.py

The review scope of this PR is the new target directory, its line in the family's
routing.py, the glmMoeDsa kernels and ops, and the tests. After #20797 merges, main is
merged here and the diff shrinks to those files.

This PR adds a GLM-5.2 NVFP4 low-latency decode target for GB300 (SM103) to modeling_v2:
models/glm_moe_dsa/glm_5_2_nvfp4__sm_103__tp8 (ModelingV2Glm52Nvfp4Sm103Tp8).

The optimization work targets one layout: attention and MoE tensor-parallel over 8 GPUs (TP8),
no attention DP, with one-model MTP-Eagle as a linear chain of five draft tokens. The family's
route() sends only that deployment to this target. Other GLM-5.2 deployments go to the other
glm_moe_dsa targets or to the built-in model:

GB300 TEP4 goes to @erictsai-nv's glm_5_2_nvfp4__sm_103__tep4 from [TRTLLM-16980][feat] modeling_v2: GLM-5.2-NVFP4 sm_103 tep4 target #20662. An earlier
revision of this PR also served TEP and DEP with a subset of its TP8 fast paths; it no longer
does, so the two targets do not overlap.
Rubin TP16 goes to glm_5_2_nvfp4__sm_107__tp16 from [None][feat] Staircase: GLM-5.2 NVFP4 low-latency decode target for Rubin (modeling_v2, TP16) #20517.
Any other deployment runs the built-in model unchanged.

routing.py keeps main's TP16 and TEP4 branches and adds the TP8 one. The NoTarget message
names all three targets.

Layout

The target follows main's target-ownership rules (test_modeling_v2_claims.py): everything it
runs lives in its own directory. Next to modeling.py and weights.py are:

the fast-path plan (dispatch.py);
the attention adoption (attention.py, attention_forward.py);
the layer composition (layers.py);
the speculative worker (spec_worker.py);
the step-loop hooks (runtime.py);
one kernel package per fast path under kernels/.

The fast paths default on inside the target. TRTLLM_STAIRCASE_GLM_LL_DISABLE=1 turns them all
off, and per-path TRTLLM_STAIRCASE_GLM_LL_* variables turn off single paths.

Executor: `RuntimeHooks` (reviewed in #20797)

The target takes over parts of the decode step through RuntimeHooks, a per-model seam that
is off by default:

executor events and an execution-stream loop;
a decode prologue that runs before the model forward.

The target also has a step-input stager and a sampler store writer
(kernels/deferred_input_prep). They are not served: #19989 on main reworked the engine paths
they hooked, so #20797 does not carry those two hooks. The code stays for a follow-up that ports
them onto the new native paths.

The target's side is runtime.py. The engine side, its tests and the note about #20306 are in
#20797.

Kernels and ops

Model-specific CUDA kernels are under cpp/tensorrt_llm/kernels/glmMoeDsa/, and their torch
ops under cpp/tensorrt_llm/thop/glmMoeDsa/: one <stem>Op.cpp per <stem>.cu, registered
as trtllm::glm_moe_dsa_* in the th_common_glm_moe_dsa library next to the TP16 target's ops.
They are used only by this target. They cover:

the post-attention hop and its routing;
the RoPE KV append;
the sparse MLA decode;
the cross-rank indexer top-k;
the dense MLP;
the draft model's routed MoE, live-row hop and tail.

Moving them into the target's own csrc/ under the per-family build of #20896 is a follow-up.

Test Coverage

End-to-end accuracy gate:
tests/integration/defs/accuracy/modeling_v2/test_glm_moe_dsa.py::TestModelingV2Glm52Nvfp4Sm103Tp8::test_gsm8k_tp8.
- It serves GSM8K with modeling_v2="require", so a deployment that misses the target, or a
  step a fast path covers but cannot run, fails instead of silently falling back to the
  built-in model.
- Deployment: TP8, MTP 5, FP8 KV, TRTLLM MoE, batch 1.
- CI stage: l0_gb300_multi_nodes_node2_gpu8.
- Result: 94.35 against the reference of 90, in 34 minutes on GB300. This was measured before
  the latest merges of main; CI re-runs it on the current head.
Engine-seam tests: tests/unittest/_torch/executor/test_runtime_hooks.py and the touched
executor and attention tests cover the hooks both off (the default) and installed.
Accuracy at TP8, compared with the built-in model on the same build:
- GSM8K (1,319 questions, paired per question, two or three passes per arm): +0.34 pp
  [−0.30, +0.99].
- GPQA-diamond: 91.4 % [87.6, 94.9], in line with the model card's 89.4 %.
- MTP acceptance length per request is within ±0.12 % of the built-in model.

PR Checklist

PR description clearly explains what and why.
Follows the TRT-LLM coding guidelines.
Test cases are provided for new code paths.
No API change. The engine seam is internal and off by default; no llm_args change.
No new dependencies.
Documentation updated (modeling_v2 README).

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Add RuntimeHooks (pyexecutor/runtime_hooks.py), a per-model seam through which a modeling_v2 target can take over parts of the decode step without changing the engine's behaviour for other models: - step-input staging, so a target can write the next step's inputs from its own buffers; - the speculative sampler's store writer; - executor events and an execution-stream loop; - a decode prologue that runs before the model forward. The executor, CUDA graph runner, decoder runner, KV cache manager V2, attention backends and the speculative modules read the hooks at their call sites. Every hook is off unless a model installs it, and the default behaviour is unchanged. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Add model-specific CUDA kernels under cpp/tensorrt_llm/kernels/glmMoeDsa/ and their torch ops under cpp/tensorrt_llm/thop/glmMoeDsa/, one <stem>Op.cpp per kernel <stem>.cu, registered as trtllm::glm_moe_dsa_<what>. They cover the decode path of GLM-5.2: the post-attention hop and its routing, the RoPE KV append, the sparse MLA decode, the cross-rank indexer top-k, the dense MLP, and the draft model's routed MoE, live-row hop and tail. The kernels build as one OBJECT library for the SM100 family and are used only by the GLM-5.2 modeling_v2 targets. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Add the glm_moe_dsa family to modeling_v2 with three targets, each claiming its own parallel topology through the family's route(): - glm_5_2_nvfp4__sm_103__tp8: tensor-parallel attention and MoE on 8 GPUs; - glm_5_2_nvfp4__sm_103__tep: tensor-parallel attention with expert-parallel experts on 4 or 8 GPUs; - glm_5_2_nvfp4__sm_103__dep: attention data parallelism with expert-parallel experts on 8 or 16 GPUs. Code the targets share (kernel packages, dispatch, attention, the layer composition, the speculative worker and the runtime-hook wiring) lives in models/glm_moe_dsa/shared/. Each target selects the fast paths it serves through its own plan; the paths default on inside the target and can be turned off with TRTLLM_STAIRCASE_GLM_LL_DISABLE or per-path TRTLLM_STAIRCASE_GLM_LL_* variables. Any configuration that no target claims runs the built-in model. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Unit tests of the glm_moe_dsa family: routing and target claims, isolation from other models, the per-target plans, the adopted layers and the runtime hooks, plus kernel tests against the native implementations. Tests that need the GLM-5.2 checkpoint find it under llm_models_root() and skip when it is not available. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Add the glm_moe_dsa family and its three targets to the modeling_v2 README. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

yuxianq · 2026-10-18T05:46:14Z

@PerkzZheng, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

Assignment explanation

Components	kernels/sparse (50%); kernels/auxiliary (45%); integration/build-config (5%)
Tags	algorithm:sparse (100%); specific-algorithm:dsa (100%); representation:mla (100%); phase:decode (100%); features:rope (40%); features:quantization (40%); features:speculative-decoding (20%); context:llm (100%)
PR effort	13 points — Provisionally 13 for interacting sparse-attention kernels, early dependent launches and cross-rank transport contracts, independently of total file count. Classified as feature: the added weights.py documentation explicitly retains the built-in model's module tree and loader, supporting optimized targets for an existing architecture.
Importance	Important — The visible contracts introduce clustered sparse-MLA softmax reduction, change RoPE/KV producer-consumer launch ordering, and fuse ordered cross-rank attention-output reduction. These require sensitive numerical, synchronization and performance review.
Retained GitHub review requests	litaotju: preserve direct code-owner review request
Affinity (0–1)	0.467 component/tag match; 0.350 after PIC preference
Outstanding points at selection	57 → 70 (+13)
Points already reserved on this PR	0
Effective weight	1
Completed review points (preceding 168 hours)	17
Combined points after assignment	87
Demand-derived target	81.00 points (soft share)
Additional fair share before assignment	7.00 points
Share explanation	Whole PR / qualification or coverage constraint exceeds a soft share; assignment is allowed.
Selection rule	affinity with 0.3-share guard
Assignment scope (PerkzZheng)	all

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

…use the TP8 fast paths Each of the two targets gets a short README: the fast paths were designed and tuned for the TP8 target, these targets run the subset that applies at their topology, and both should be revisited with fast paths of their own. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

clang-format on five C++ files and one test .cu (whitespace only), an unused local and a blank line flagged by ruff, two CuTe tensor names that codespell reads as misspellings, and the prep packet's pinned slots allocated with prefer_pinned(), as the pinned-memory policy requires. The stager declines when prefer_pinned() is False, so a packet is only built when it is True. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Wanli-Jiang · 2026-10-19T09:09:08Z

@@ -0,0 +1,3 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.


do we have ways to combine things into one general target like https://github.com/NVIDIA/TensorRT-LLM/pull/20267/changes did? during target distribution, we can create different targets.

Thanks, done in b209b9f. The tp8, tep and dep targets are now one target, glm_5_2_nvfp4__sm_103__default, following the #20267 convention. route() admits the same layouts as before and returns that class for all of them; the target picks the fast paths of the layout when it is built (target_paths in its modeling.py). The per-layout tables, forward and module adoption are unchanged. Only TP8 is tuned; the target README says TEP and DEP run the applicable subset and should be revisited, and a layout-specific target can be split out again later if one earns its own forward. The three GSM8K gates (tp8, tep4, dep8) pass on GB300 on the current head, after merging main.

…ads as a hardcoded password bandit B105 matches "token" in an assigned name, so it flagged the routed-MoE layout rule ROUTED_MOE_TOKEN_MAJOR_RULE. It is now ROUTED_MOE_UNFINALIZED_LAYOUT_RULE; the text is unchanged. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

nvpohanh · 2026-10-22T04:24:59Z

[by Claude Code] @eopXD Could you review this PR? Thanks!

Add RuntimeHooks, a per-model declaration of engine-side fast paths the model provides itself: a step-input stager, a speculative sampler store writer, a decode prologue, executor events, and running the overlap loop on the execution stream. Every hook defaults to off, and a model without a declaration runs the native path unchanged. Also add two extension points a model can use: FMHA libraries a backend puts ahead of the enabled ones, and the DSA indexer's reused top-k copy as an overridable method. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…side the distributed warmup guard after_warmup may run collectives, so a rank-local failure there must arm the rank-crash watchdog like a warmup failure does. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…ay of a captured graph The step-input window closes right before the replay of an already captured CUDA graph. An eager step, an initial capture or a warmup would run the model before the staged writes are issued, so the decoder runner now offers the window only on such a replay, and StepInputStager documents the restriction. Add behavioral tests with reference hooks: the runner window lifecycle (staged writes issued before the completion event and the replay, discard on a failure while the window is open), steps that are not a replay never open a window, before_replay runs only before the replay of a captured graph, and the speculative sampler's store writer against the native stores on nonconsecutive slots, with its native fallbacks. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Wanli-Jiang · 2026-10-23T05:08:57Z

@xwang233 just to check: I saw many new added cpp kernels, are they all for the modeling v2 specific GLM model target?
I think it should be fine for now,
but I am thinking whether we can put them into the modeling v2 specific model folder with separated compile system, to isolate the model level from stock codes.

xwang233 · 2026-10-23T15:57:04Z

Hi @Wanli-Jiang , yes those are GLM-5.2 modeling_v2 specific kernels. Can you point me the "modeling v2 specific model folder" path, so that I can move those kernels there?

…he sampler store writer Main's one-model speculative decoding work now batches the overlap input gathers and moves the sampler's host copies off the critical path natively, so these two hooks no longer fit the code they replaced. Remove them and their call sites; they can be ported onto the new native paths later. Keep the decode prologue, the executor events, the execution-stream loop and the two extension points (prepended FMHA libraries, the DSA indexer's overridable top-k copy). Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/engine/runners/decoder/runner.py # tensorrt_llm/_torch/pyexecutor/py_executor.py # tests/unittest/_torch/executor/test_py_executor.py

Takes the runtime hooks branch, which now carries main up to 94348c4 and drops the step-input stager and sampler store writer hooks. - glm_moe_dsa: the sm103 target no longer serves deferred_input_prep or sampler_store; make_runtime_hooks declares only the decode prologue, the execution-stream loop and the executor events. - Routing: one route() for GlmMoeDsaForCausalLM covering the sm103 and the sm107 TP16 targets, via the @routes registry. - CMake: the GLM kernel exclude covers only this target's .cu files; its thop ops join th_common_glm_moe_dsa next to the TP16 ones. - Accuracy test module holds both the TP16 and the sm103 classes. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

xwang233 · 2026-10-27T05:05:07Z

@Wanli-Jiang Yes. All the new kernels under cpp/tensorrt_llm/kernels/glmMoeDsa/ and their ops under cpp/tensorrt_llm/thop/glmMoeDsa/ are used only by the GLM-5.2 modeling_v2 targets. Nothing outside models/glm_moe_dsa calls them.

I agree they should live with the model. #20896 gives each family its own libth_modeling_v2_<family>.so, built from models/<family>/csrc and <target>/csrc. Once it lands, we will move these kernels into models/glm_moe_dsa/.../csrc in a follow-up. The TP16 target's kernels from #20517 sit in the same directories, so we can move them in the same follow-up.

For now, I resolved the conflicts with the TP16 target in this PR. Both targets build into the existing th_common_glm_moe_dsa family, and the kernel exclude in kernels/CMakeLists.txt lists only this target's .cu files. That keeps the later move a mechanical change.

…_model main renamed the decoder runner's model attribute to _model, so reading self.model failed at runner construction for every model. Update two tests to main's code: the GVR prior test calls sparse_attn_indexer on a SimpleNamespace, which now needs the indexer seams bound; the forward-step stub needs route_capture, enable_spec_decode and runtime_draft_len. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…-v2-runtime-hooks Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/py_executor.py

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

yuxianq · 2026-10-28T04:15:20Z

@yunruis, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

@PerkzZheng requested an OOTO handoff of unstarted reviews (availability-79077589). The 13 outstanding points have moved from @PerkzZheng to @yunruis. This transfer earns no completed-review credit.

Assignment explanation

Components	kernels/sparse (45%); kernels/auxiliary (50%); integration/build-config (5%)
Tags	algorithm:sparse (100%); representation:mla (100%); phase:decode (100%); features:rope (60%); features:quantization (40%)
PR effort	13 points — The visible additions implement specialized attention execution paths; they do not establish a new model architecture. Clustered sparse MLA, RoPE/KV publication ordering, and distributed post-attention fusion warrant deep, cross-kernel reasoning consistent with 13 points, independently of total PR size. This estimate is provisional because implementation bodies are largely unavailable.
Importance	Important — Visible contracts introduce correctness-sensitive synchronization and numerical behavior: clustered sparse-attention softmax exchange in sparseMlaDecode.cu, early dependent launches in indexerLayerRopeKvAppend.h, and fixed-order distributed reduction in postAttentionHop.h.
Retained GitHub review requests	litaotju: preserve direct code-owner review request
Affinity (0–1)	0.550 component/tag match; 0.413 after PIC preference
Outstanding points at selection	14 → 27 (+13)
Points already reserved on this PR	0
Effective weight	1
Completed review points (preceding 168 hours)	21
Combined points after assignment	48
Demand-derived target	48.00 points (soft share)
Additional fair share before assignment	13.00 points
Share explanation	Within demand-derived share.
Selection rule	affinity with 0.3-share guard
Assignment scope (yunruis)	all

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…deling-v2-runtime-hooks Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/py_executor.py

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

…ownership rules Main now routes sm_103 tep4 to glm_5_2_nvfp4__sm_103__tep4 (NVIDIA#20662), and its claims suite no longer lets a target import a family-level shared package. - The sm_103 low-latency target serves only the layout its fast paths were built for: attention and experts tensor-parallel over eight ranks (tp8). It is renamed glm_5_2_nvfp4__sm_103__tp8 (ModelingV2Glm52Nvfp4Sm103Tp8); the tep and dep layouts, their path sets and the dep cells are removed. - glm_moe_dsa/shared moves into the target directory: its modules beside modeling.py, its kernel packages beside the target's routed-MoE ones under kernels/. Relative imports are rewritten for the new locations. - routing.py keeps main's tp16 and tep4 branches and adds the tp8 branch. - The accuracy gate is TestModelingV2Glm52Nvfp4Sm103Tp8::test_gsm8k_tp8; the tep4 and dep8 cases and their test-list entries are removed. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

xwang233 added 5 commits October 17, 2026 09:10

[None][doc] modeling_v2: list the GLM-5.2 targets

a87e238

Add the glm_moe_dsa family and its three targets to the modeling_v2 README. Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

xwang233 requested review from a team, Wanli-Jiang, WeiHaocheng and litaotju as code owners October 18, 2026 05:38

xwang233 requested review from VALLIS-NERIA, heyuhhh, nv-xtf, reasonsolo, schetlur-nv, thorjohnsen and zhaoyuanh-nvidia October 18, 2026 05:38

github-actions Bot assigned xwang233 Oct 18, 2026

xwang233 requested a review from Tabrizian October 18, 2026 05:40

yuxianq requested review from PerkzZheng and removed request for heyuhhh and schetlur-nv October 18, 2026 05:46

Wanli-Jiang reviewed Oct 19, 2026

View reviewed changes

xwang233 changed the title ~~[TRTLLM-17096][feat] modeling_v2: GLM-5.2 NVFP4 low-latency decode targets for GB300~~ [TRTLLM-17096][feat] modeling_v2: GLM-5.2 NVFP4 low-latency decode target for GB300 Oct 21, 2026

nvpohanh requested a review from eopXD October 22, 2026 04:15

xwang233 mentioned this pull request Oct 22, 2026

[TRTLLM-17096][feat] runtime hooks: model opt-in engine fast paths #20797

Open

5 tasks done

xwang233 added 4 commits October 22, 2026 09:34

Wanli-Jiang mentioned this pull request Oct 23, 2026

[None][feat] modeling_v2: build target C++/CUDA as per-family libraries #20896

Merged

1 task done

xwang233 added 3 commits October 26, 2026 21:49

xwang233 requested a review from dongfengy as a code owner October 27, 2026 05:03

xwang233 added 4 commits October 27, 2026 05:41

Merge commit '6b32265120' into glm52-ll/v3-staircase-pr

b8c77d1

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Merge commit '0a03c3ed057fe8c488e221413640ca87f64e99d2' into modeling…

afc9cce

…-v2-runtime-hooks Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/py_executor.py

Merge commit 'afc9cce9d2' into glm52-ll/v3-staircase-pr

d873539

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

yuxianq requested review from yunruis and removed request for PerkzZheng October 28, 2026 04:15

xwang233 added 5 commits October 27, 2026 21:46

Merge commit '9c5e8e5de3' into modeling-v2-runtime-hooks

bd21d98

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Merge branch 'main' of https://github.com/NVIDIA/TensorRT-LLM into mo…

9b8bb02

…deling-v2-runtime-hooks Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com> # Conflicts: # tensorrt_llm/_torch/pyexecutor/py_executor.py

Merge commit 'b7b797a3aa' into modeling-v2-runtime-hooks

4b4d280

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Merge commit '4b4d280bc6' into glm52-ll/v3-staircase-pr

267280b

Signed-off-by: Xiao Wang <24860335+xwang233@users.noreply.github.com>

Navigation Menu

Repository navigation

[TRTLLM-17096][feat] modeling_v2: GLM-5.2 NVFP4 low-latency decode target for GB300 - #20524

[TRTLLM-17096][feat] modeling_v2: GLM-5.2 NVFP4 low-latency decode target for GB300#20524
xwang233 wants to merge 30 commits into
NVIDIA:mainfrom
xwang233:glm52-staircase-gb300

xwang233 commented Oct 18, 2026 •

edited

Loading

Uh oh!

yuxianq commented Oct 18, 2026 •

edited

Loading

Uh oh!

Wanli-Jiang Oct 19, 2026

Uh oh!

xwang233 Oct 21, 2026

Uh oh!

nvpohanh commented Oct 22, 2026

Uh oh!

Wanli-Jiang commented Oct 23, 2026

Uh oh!

xwang233 commented Oct 23, 2026

Uh oh!

xwang233 commented Oct 27, 2026

Uh oh!

yuxianq commented Oct 28, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1,3 @@
		# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Conversation

xwang233 commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Description

Layout

Executor: RuntimeHooks (reviewed in #20797)

Kernels and ops

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

yuxianq commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Wanli-Jiang Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

xwang233 Oct 21, 2026

Choose a reason for hiding this comment

Uh oh!

nvpohanh commented Oct 22, 2026

Uh oh!

Wanli-Jiang commented Oct 23, 2026

Uh oh!

xwang233 commented Oct 23, 2026

Uh oh!

xwang233 commented Oct 27, 2026

Uh oh!

yuxianq commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xwang233 commented Oct 18, 2026 •

edited

Loading

Executor: `RuntimeHooks` (reviewed in #20797)

yuxianq commented Oct 18, 2026 •

edited

Loading

yuxianq commented Oct 28, 2026 •

edited

Loading