chore(deps): update dependency flashinfer-python to v0.6.18

[renovate]

ℹ️ Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
flashinfer-python	==0.6.16.post3 → ==0.6.18
flashinfer-python	==0.6.14 → ==0.6.18

---

Release Notes

flashinfer-ai/flashinfer (flashinfer-python)

v0.6.18

Compare Source

These highlights are also published at flashinfer.ai/releases.

v0.6.18 Highlights

This release completes NVIDIA Rubin (SM107) support, brings whole-layer expert-parallel MoE to Hopper, adds decode paths for DeepSeek-V4 sparse attention and Kimi K3 linear attention, and broadens low-precision MoE coverage with MXFP4 on Blackwell RTX PRO and DGX Spark and weight-only W4A16 on B200 and B300.

SM107 (Rubin) support

FlashInfer 0.6.18 completes support for NVIDIA Rubin (SM107), begun in 0.6.16. Rubin devices now dispatch through the same unified APIs as Blackwell — attention, GEMM, MoE, and quantization.

Attention gains trtllm-gen FMHA for SM107, including sparse compression and FP16 softmax. The PrimTS attention path accepts Rubin as well. On the GEMM side, batched and low-latency GEMM both run on Rubin, and CUTLASS NVFP4 SVDQuant is enabled. A CuTe-DSL kernel family specialized for Rubin ships alongside them. trtllm-gen MoE now selects valid tactics on Rubin, and router GEMM and topk_varlen accept SM107.

• #​4526
• #​4596
• #​4509
• #​4710
• #​4792

Expert-parallel MoE runs whole-layer on Hopper

The moe_ep mega-kernel stack was Blackwell-only. Two Hopper FP8 backends, Sm90PullFp8MegaMoeConfig and Sm90PushFp8MegaMoeConfig, now let Hopper deployments run dispatch, FC1, SwiGLU, FC2, and combine as one fused layer behind the existing MoEEpLayer, instead of composing an NCCL all-to-all with a local fused-MoE operator. The pull backend reaches 562 TFLOPS/rank at a 384-expert DeepSeek-class geometry. The push backend supports CUDA Graph capture, and its opt-in fused FC1 epilogue drops an approximately 1 GiB per-rank activation buffer at the DeepSeek-V3 EP8 shape. Its grouped GEMM requires CUDA Toolkit 12.8 or newer.

• #​4113
• #​4069
• #​4449

HCA decode backend and top-K selection for DeepSeek-V4

Sparse attention picks the top-K KV positions per request on every decode step, then attends over the compressed cache; both halves now have dedicated paths. FP8 Heavily Compressed Attention (HCA) arrives for SM100/SM103 via trtllm_batch_decode_sparse_mla_dsv4(..., backend="cute-dsl"), taking arbitrary sliding-window row order including ring rotation and wraparound while keeping the compressed cache paged. The new flashinfer.top_k_varlen handles ragged batches through a Blackwell radix kernel, a guess-verify-refine kernel that warm-starts from the previous step's indices, and a CUTLASS fallback for any GPU. SM120/121 also picks up top-k 192 and 256.

• #​3943
• #​3901
• #​4380

Kimi K3 decode fuses into one Blackwell kernel

flashinfer.fused_kda_decode folds Kimi K3's width-four depthwise causal convolution, SiLU, recurrent Kimi Delta Attention update, and gated RMSNorm into a single SM100 launch, covering the production head_dim 128 shapes at 12, 24, 48, and 96 heads and updating the convolution cache and FP32 state in place. On B200 under CUDA Graphs it is 1.33x the vLLM fused kernel at one row (table geomean 1.13x). A T=1 fast path inside recurrent_kda and shared SM100-family recurrent kernels round out the KDA stack.

• #​4243
• #​4417
• #​4562

MXFP4 MoE and video sparse attention on Blackwell RTX PRO

MXFP4 checkpoints run natively on SM120/121: b12x_fused_moe and B12xMoEWrapper accept quant_mode="mxfp4" across the existing fused schedules including CUDA Graph reuse, the b12x dense GEMM gains the matching path, and on an RTX PRO 6000 Blackwell Server Edition MXFP4 tracks NVFP4 against a strict quantized reference. Video Sparse Attention, until now datacenter-Blackwell only, reaches these parts through a vsa_sm120_blk64 backend on BlockSparseAttentionWrapper. Gemma 4 gains asymmetric VO-split NVFP4 paged prefill on SM120/121.

• #​4290
• #​4479
• #​4259
• #​3684

W4A16 MoE and dense GEMM extend to B200 and B300

Weight-only NVFP4 against BF16 activations, which 0.6.14 shipped for SM12x, now runs on the SM100 family: CuteDslMoEWrapper and cute_dsl_fused_moe_nvfp4 accept quant_mode="w4a16", decoding weights to BF16 inside the kernel so no separate activation-quantization or repack launch is needed, and mm_bf16_fp4 gains a dedicated SM100/SM103 kernel. Consuming BF16 directly pays off where MoE decode is memory-bound: at a DeepSeek EP8 shape on B200, W4A16 is 1.50x the W4A4 baseline at one token; W4A4 still wins large-batch prefill. GeGLU-tanh and SiTU are supported.

• #​4048
• #​4466
• #​4394

Unified MoE API adds shared experts, MXINT4, and CUTLASS runners

Shared experts now work through the unified API rather than low-level kernel entry points, via ExpertConfig.num_fused_shared_experts on the block-FP8 and FP4 runners, completing in the unified API what 0.6.15 and 0.6.17 added for FP8 and FP4. The API also gains MXINT4, CUTLASS BF16 and W4A16 runners on SM90, packed per-tensor FP8 routing, BF16 FromLogits routing, and TopKSigmoid.

• #​4326
• #​4320
• #​4328
• #​4227
• #​4404

Fused MNNVL all-reduce tail for tensor-parallel MoE

allreduce_fusion gains a BF16 Blackwell CuTe-DSL backend for the latency-critical tail of tensor-parallel MoE layers, fusing all-reduce, residual add, and RMSNorm, optionally preceded by MoE finalize and shared-expert add, over MNNVL/NVLink multicast. One backend spans decode to prefill by switching protocol with token count, with initial profiles targeting GB300 on TP8 and TP16.

• #​4358

Smaller JIT-cache wheels; SM75 and single-request FA2 are JIT-only

The flashinfer-jit-cache wheels no longer ship precompiled kernels for SM75 (Turing). Those GPUs still run; the kernels compile on first use. CUDA 13 AArch64 wheels also drop native SM121a cubins (DGX Spark keeps running via SM120 family cubins), and the single-request single_decode_with_kv_cache / single_prefill_with_kv_cache FA2 modules are no longer AOT-prebuilt — those APIs still JIT. Fatbins use size-oriented compression.

	0.6.17	0.6.18
cu129 x86_64	1.94 GB	1.02 GB
cu130 x86_64	1.51 GB	1.02 GB
cu130 aarch64	1.69 GB	1.13 GB

• #​4757
• #​4760

What's Changed

• test(moe): add tests for trtllm-gen fused MoE with GeGLU activation by @​Aneureka in #​4265
• perf: optimize trtllm_fmha_v2 fp8 causal attention q-tile scheduling & decoding for uniform seqlen by @​akhilg-nv in #​3575
• docs: improve Ulysses communicator and MoE EP docs by @​kangbintNV in #​4240
• fix(docker): stop pip from swapping the +cuXXX torch in CI image by @​bkryu in #​4284
• fix: skip LogitsTransform on lanes beyond the split-KV chunk boundary in FA2 kernels by @​yichengj0 in #​3890
• fix(test): repair main CI regressions from #​4280 (artifacts Rubin pins + CuTe-DSL MoE device guard) by @​bkryu in #​4301
• fix: preserve DeepSeek no-group sigmoid routing weights by @​alexeldeib in #​3875
• fix(gdn): use block-end decay for SM100 state updates by @​guangyunh-nv in #​4311
• feat(cake_kda): add optimized B200 recurrent prefill backend by @​yyihuang in #​4262
• feat(cake_kda): add optimized B200 recurrent decode backend by @​yyihuang in #​4279
• test: Add sharding support to scripts/task_run_unit_tests.sh by @​righthandabacus in #​4141
• test(msa_ops): fix stale split-K heuristic expectation on high-SM GPUs by @​jimmyzho in #​4303
• perf(moe): sync SM12x NVFP4 fused-MoE kernels to b12x HEAD by @​yichengj0 in #​4285
• feat(moe): sync SM12x W4A16 fused MoE family to b12x HEAD by @​yichengj0 in #​4255
• fix: support fp8 e5m2 output in rmsnorm_quant and fused_add_rmsnorm_quant by @​elwhyjay in #​4202
• feat(moe): add packed per-tensor FP8 and BF16 FromLogits routing to unified MoE API by @​feih-nv in #​4227
• perf: remove dead cudaGetDeviceProperties in sm120 groupwise GEMM by @​aws-jiadingg in #​3523
• fix(quantization): use 64-bit row addressing in per-token NVFP4 quantizer by @​S1ro1 in #​4263
• Fix duplicated words in API documentation by @​cupkk in #​4194
• feat(cake_kda): share recurrent prefill kernels across SM100 family by @​yyihuang in #​4313
• feat(moe): enable BiasType::Mn (LoRA delta) for nvfp4/mxfp4 MoE by @​zetacat in #​3987
• chore: drop unused <nvrtc.h> includes from three launchers by @​aleozlx in #​4293
• perf: remove NVFP4 TMA input padding copy by @​Barry-Delaney in #​4210
• Fix the expert correction bias checking inconsistency in trtllm_mxint4_block_scale_moe by @​b8zhong in #​3898
• Update CODEOWNERS for moe_ep and autotuner sections by @​aleozlx in #​4332
• feat(moe): enable MxInt4 in unified API by @​feih-nv in #​4320
• feat(cake_kda): share recurrent decode kernels across SM100 family by @​yyihuang in #​4314
• [chore] Add jiahanc to gemm code owner by @​jiahanc in #​4327
• feat(kda): add fused Kimi K3 decode kernel by @​djmmoss in #​4243
• feat(cake_tinygemm2): add CAKE-generated SM100/SM103 tinygemm2 variants with bit-identical outputs by @​xslingcn in #​4274
• feat: add sm100 cp delta rule prefill by @​guangyunh-nv in #​4078
• fix(cake_kda): support non-aligned recurrent prefill head counts by @​yyihuang in #​4351
• Revert "test: Add sharding support to scripts/task_run_unit_tests.sh" by @​bkryu in #​4344
• perf(topk): skip output index sort for tie-break selection by @​zianglih in #​4295
• Upgrade CuTe DSL FMHA cubins by @​xrq-phys in #​4291
• fix(moe): handle CuTe DSL finalize output tails by @​S1ro1 in #​4186
• MoE monokernel Bug fix, barrrier remove and kernel rewrite. by @​yugong333 in #​4027
• MSA decode path improvements by @​rmhaskarnvidia in #​4324
• Add CuTe DSL HCA backend for DeepSeek V4 sparse MLA by @​myu-guo in #​3943
• Add @​StudyingShao to CODEOWNERS for multiple sections by @​aleozlx in #​4364
• fix(moe_ep): let the host framework pin the CUDA device via Bootstrap by @​mhoqueanik in #​4348
• [feat] Add Blackwell cutedsl BF16 splitk dense gemm by @​jiahanc in #​4266
• Accept unpacked pre-routed topk in fp8 block-scale and bf16 MoE by @​b8zhong in #​4352
• fix(comm): keep fd_exchange importable on Python 3.10/3.11 by @​nv-guomingz in #​4354
• feat: log git commit hash on import (FLASHINFER_LOGLEVEL >= 1) by @​aleozlx in #​4286
• fix(gemm): gate cuDNN out of SM12x bmm_fp8 auto when override_shape unavailable by @​Saddss in #​4165
• perf: normalize autotuner nearest-profile cache keys by @​wongsingfo in #​3984
• feat(moe): add CuTe-DSL GeGLU-tanh and SiTU activations by @​murphymatt in #​4009
• bump version to 0.6.18 by @​aleozlx in #​4384
• feat(moe): support BF16 activations in CuTe DSL NVFP4 MoE by @​zianglih in #​4048
• docs: document tinygemm2 escape hatch and MonoMoE scratchpad by @​kangbintNV in #​4381
• feat: optimize gated SM12x dynamic NVFP4 MoE by @​EricChen02 in #​4329
• fix(monomoe): restore CUDA 12.0+ compatibility in tma_load_2d by @​aleozlx in #​4406
• feat(moe_ep): SM90 (Hopper) pull-style FP8 mega-MoE backend by @​mhoqueanik in #​4113
• feat(topk): Add top_k_varlen with GVR and radix backends for sparse-attention KV selection by @​dhiraj113 in #​3901
• feat(sm120): consolidate DSV4 sparse MLA top-k 192/256 support by @​lucifer1004 in #​4380
• Fix/quantized allreduce ci harness by @​cindyzxq in #​4330
• perf(moe): persist b12x MoE CuTe-DSL kernels to the disk cache by @​Smallfu666 in #​4331
• feat(topk): support compact page table transforms by @​zianglih in #​4315
• fix(moe_ep): preserve singleton expert TMA modes by @​cherichy in #​4296
• feat(moe): add unified CUTLASS BF16 and W4A16 runners by @​feih-nv in #​4328
• feat(sm120): add native MXFP4 W4A4 fused MoE by @​Yuening-wa in #​4290
• fix(moe): pad BF16 TRTLLM-Gen intermediates to 128 KiB by @​nvpohanh in #​4319
• require explicit ci triggering for all pull requests by @​yongwww in #​4383
• test incidental support for fp32 unpacked expert weights from #​3763 by @​jdebache in #​4448
• fix(moe): prepare MXFP8 MXFP4 profiler inputs by @​ormandj in #​4308
• Feat/deepseek fused moe fp8 blockwise swigluoai by @​vitamin-chaos in #​4405
• feat(comm): add Blackwell MNNVL CuTe DSL all-reduce fusion backend by @​qiangyicheng in #​4358
• perf(gdn): optimize cp host launch overhead for sm90 and sm120 by @​guangyunh-nv in #​4374
• fix(moe): use per-expert Humming residual scales in SM90 CUTLASS MXFP4xFP8 MoE backend by @​StudyingShao in #​4431
• perf(norm): speed up fused add RMSNorm FP4 quantization by @​soodoshll in #​4416
• Add task-scheduled Blackwell attention kernels by @​PerkzZheng in #​4357
• fix(gemm): validate cached CuTeDSL BF16 tactics by @​jiahanc in #​4455
• perf(cake_tinygemm2): add a STAGES=16 kernel tier for single-wave large-K shapes by @​xslingcn in #​4423
• perf(moe): optimize CUTLASS fused MoE mem-bound kernels by @​Aneureka in #​3761
• remove spot instances from pr-test, use on-demand only by @​yongwww in #​4454
• test: Add sharding support to scripts/task_run_unit_tests.sh by @​righthandabacus in #​4359
• feat(gdn): u/d cache spec-decode kernels for replayssm by @​ameynaik-hub in #​4081
• Mirror nightly release to flashinfer-ai/whl repo by @​dierksen in #​4353
• perf(sm120): wave+residue tile-selection for plain groupwise MoE GEMM by @​CarstyYou in #​4318
• refactor(moe): enforce build() for all runners and refine CUTLASS autotuning in unified MoE API by @​feih-nv in #​4376
• Update Docker CI tags to 2026081-e673f7f by @​flashinfer-bot in #​4457
• fix(gdn): convert fp16 decode operands and fix cache identity by @​kahyunnam in #​4219
• fix(fmha): propagate LSE reduction launch errors by @​leejnau in #​4389
• feat: paged KV cache support for the modular CuTe-DSL Blackwell prefill by @​pgera in #​4212
• chore: add KDA CODEOWNERS by @​kahyunnam in #​4482
• Fix HCA Gather4 support for public CuTe DSL by @​myu-guo in #​4368
• feat(moe_ep): add SM90 push FP8 mega-MoE backend for Hopper by @​leonardHONG in #​4069
• fix(comm): align MNNVL two-shot workspace stages by @​aoshen02 in #​4473
• Fused shared experts support via the unified API by @​feih-nv in #​4326
• feat(attention): asymmetric VO-split NVFP4 paged prefill (qk=512/vo=256) for Gemma-4 on SM120/121 by @​jethac in #​3684
• ci: centralize minimum Python version at 3.10 by @​dierksen in #​4464
• perf(gemm): split-K, occupancy, and m=1 stream-GEMV decode tactics for mm_bf16_fp4 by @​yichengj0 in #​4038
• feat(moe): MXFP8 x MXFP4 CuTe-DSL fused MoE for SM100, plus large-batch routing locality by @​vitamin-chaos in #​4440
• Add distribution-aware autotuning for TRT-LLM MoE by @​samuel21119 in #​4106
• feat(quantization): cuTile per-token-group 8bit quant + fused RoPE-FP8 by @​yifeis-nv in #​4019
• feat(gemm): port SM12x MXFP8 dense GEMM from b12x by @​yichengj0 in #​4305
• perf(gdn): reuse pretranspose kernels across pool capacity and stride by @​leonardHONG in #​4444
• chore(codeowners): add @​jimmyzho to tests/attention/ by @​jimmyzho in #​4508
• skip the deliberate-trap test under compute-sanitizer by @​qsang-nv in #​4493
• feat(msa): fp8 index-k and index-q support in the SM12x proxy-score kernels by @​yichengj0 in #​4345
• Fix TRTLLM ragged prefill edge cases by @​alexeldeib in #​3779
• Support non-interleaved KV input to fmha_v2 by @​akhilg-nv in #​4124
• feat(attention): cuDNN paged prefill via direct mixed-form cu_seq_lens by @​egilliam-nv in #​4222
• perf(gdn): reuse MTP decode kernels across cache modes by @​hebo1221 in #​4128
• ci: add PR API and documentation checks by @​kangbintNV in #​3917
• Add SM120 FP8 FMHAv2 self-attention by @​rosenrodt in #​4272
• feat(kda): add packed-input CuTe decode kernel by @​ameynaik-hub in #​4417
• chore: explicitly set mMultiCtasKvMode in ragged attention launcher by @​namgyu-youn in #​3469
• integrate SM120 VSA (Video Sparse Attention) block-sparse backend by @​hsr1234563 in #​4259
• ci: upgrade sccache and retain stats by @​dierksen in #​4505
• fix: correct trtllm-gen reduction indexing for FMHA decode at long q_len by @​jhaotingc in #​4382
• ci: upload GitHub unit-test JUnit reports by @​dierksen in #​4488
• perf(moe): fold the SiTU beta reciprocal in the CuTe DSL MoE epilogue by @​sychen52 in #​4506
• feat: unpacked FP8 per-tensor scaling support for TRTLLM fused MoE by @​jdebache in #​4478
• refactor(moe_ep): taxonomy/provenance restructure; incorporate SM90 push-style FP8 backend; sync CuTe-DSL 4.7 quant-staging fix by @​mhoqueanik in #​4449
• gemm: enable CUTLASS NVFP4 SVDQuant on SM107 by @​Vinnie6167 in #​4509
• ci: grant @​flashinfer-bot access to collaborators who can apply labels by @​yongwww in #​4510
• feat: support FP8 KV NoPE MLA on SM90 by @​JustinTong0323 in #​4373
• fix(ci): correct documentation finding locations and skipped runs by @​kangbintNV in #​4520
• test(gdn): trim GDN decode test matrix to distinct kernel specializations by @​kahyunnam in #​4513
• feat: add BF16Q FP8KV transform-mode selection by @​djmmoss in #​3544
• [prims-ts] Refactor FMHA decode padding-task scheduling by @​IwakuraRein in #​4413
• feat(moe): support GeGLU and SiTU in W4A16 by @​zianglih in #​4394
• feat(cake_kda): add strided prefill state checkpoints and packed decode by @​yyihuang in #​4445
• WIP jit-cache wheel size fix: drop 12.1a from the cu129 aarch64 jit-cache wheel by @​kahyunnam in #​4527
• docs: include top_k in topk API docs by @​kangbintNV in #​4501
• Fix/missing source tree files by @​cindyzxq in #​4517
• ci: install quack-kernels with --no-deps so it cannot downgrade CuTe DSL by @​kahyunnam in #​4556
• test: make heavy attention test files more efficient (bulk JIT precompile + faster references) by @​bkryu in #​4511
• fix(moe): release W4A16 CuTe DSL scratch workspaces by @​zianglih in #​4558
• [fix] disable tileN=192 for trtllmgen per-token NVFP4 MoE by @​jiahanc in #​4563
• feat: sm100 cute_dsl w4a16 gemm by @​IwakuraRein in #​4466
• feat(b12x): support MXFP4 dense GEMM on SM120 by @​ayrnb in #​4479
• feat: Grouped-token MLA support for the TRTLLM-Gen FMHA backend. by @​farazkh80 in #​3849
• fix(build): restore the nixl v1.3.1 submodule pin accidentally rolled back in #​3759 by @​mhoqueanik in #​4530
• Fix/missing source tree files for two more moe_ep test files by @​cindyzxq in #​4580
• fix(xqa): out-of-bounds read of attention sinks in SM90 kernel epilogue by @​bkryu in #​4525
• feat(moe): add TopKSigmoid routing method (TopK -> Sigmoid) by @​EdalatiAli in #​4404
• feat(xqa): support non-MLA head_dim 512 (Gemma-style GQA decode) on SM12x by @​bkryu in #​4570
• perf(moe): fold W4A16 SiTU reciprocals by @​zianglih in #​4540
• fix: guard oversized shared memory driver enums with CUDA >= 13.4 by @​JiaxuanBai in #​4377
• [MoE] Expose zero-copy MegaMoE workspace output view by @​foraxe in #​4341
• refactor(moe): split finalize knobs out of ExecutionConfig into MoEFinalizeConfig by @​aleozlx in #​4385
• feat(cake_kda): add optimized H12 packed decode across SM100 family by @​yyihuang in #​4562
• ci: use the CI image's packages and share a pip cache between jobs by @​yongwww in #​4592
• stabilize pytest node IDs for external scopes to fix issues 4499 and 4500 by @​righthandabacus in #​4538
• fix(gemm,moe): restore bmm_fp8 auto fallback and drop an over-strict SM107 tactic guard by @​Vinnie6167 in #​4645
• fix: align the nvidia-cutlass-dsl floor on release-v0.6.18 with main (>=4.6.2a0) by @​kahyunnam in #​4715
• ci: pin nvidia-cutlass-dsl 4.6.2 on release-v0.6.18 by @​kahyunnam in #​4758
• ci: keep Rubin cute-dsl pins after the 4.6.2 CI pin by @​kahyunnam in #​4762
• Cherry-pick unit, wheel, infra, and Rubin follow-ups onto release-v0.6.18 by @​kahyunnam in #​4764
• fix: skip SM107 low-latency GEMM cubins on Blackwell (#​4773) by @​kahyunnam in #​4786
• fix(moe): filter the trtllm-gen kernel manifest per arch to restore MoE JIT build time by @​jimmyzho in #​4789
• [release-v0.6.18] Rubin (SM107) open-issue fixes by @​Vinnie6167 in #​4790
• [release-v0.6.18] Five SM107 (Rubin) fixes: 133 CI failures by @​Vinnie6167 in #​4792

New Contributors

• @​alexeldeib made their first contribution in #​3875
• @​righthandabacus made their first contribution in #​4141
• @​S1ro1 made their first contribution in #​4263
• @​cupkk made their first contribution in #​4194
• @​Barry-Delaney made their first contribution in #​4210
• @​rmhaskarnvidia made their first contribution in #​4324
• @​myu-guo made their first contribution in #​3943
• @​Saddss made their first contribution in #​4165
• @​wongsingfo made their first contribution in #​3984
• @​EricChen02 made their first contribution in #​4329
• @​Smallfu666 made their first contribution in #​4331
• @​cherichy made their first contribution in #​4296
• @​ormandj made their first contribution in #​4308
• @​qiangyicheng made their first contribution in #​4358
• @​aoshen02 made their first contribution in #​4473
• @​jethac made their first contribution in #​3684
• @​samuel21119 made their first contribution in #​4106
• @​ayrnb made their first contribution in #​4479
• @​farazkh80 made their first contribution in #​3849
• @​JiaxuanBai made their first contribution in #​4377
• @​foraxe made their first contribution in #​4341

Full Changelog: flashinfer-ai/flashinfer@v0.6.17rc5...v0.6.18

v0.6.17

Compare Source

These highlights are also published at flashinfer.ai/releases.

v0.6.17 Highlights

This release brings MoE expert parallelism into serving engines, refreshes the Blackwell SM12x fused-MoE kernels with an FP4 accuracy fix, extends the unified MoE API to MXFP4 W4A8/W4A16 and shared experts, and adds decode coverage for Kimi K3 MLA and MiniMax-M3 sparse attention on vLLM.

MoE expert parallelism production-ready in vLLM

The MegaMoE path in flashinfer.moe_ep is ready for serving engines: full CUDA-graph capture and replay, a fused single-launch quantize-and-stage hot path, prequantized weight packs, symmetric-buffer workspaces pooled across layers, and a persistent knob cache that resolves tuned knobs by lookup, so production sessions run with no in-engine autotuning. An opt-in fault-tolerance rank mask, available over both NCCL-EP and NIXL-EP, masks and skips a peer that times out during dispatch or combine, keeping the job alive when a rank dies. Deployment gets simpler too: the CuTe-DSL runtime floor returns to 4.5.2, the version vLLM 0.25.1 pins, and a new BootstrapConfig.device lets the host framework pin each worker's CUDA device.

• #​4079
• #​4183
• #​4101
• #​4348

Blackwell SM12x fused MoE refreshed, with an FP4 accuracy fix

W4A4 serving on DGX Spark and RTX PRO parts (GB10, SM120/SM121) now delivers the output quality its benchmark scores imply, with two NVFP4 quantization bugs fixed and a new input_global_scale that lets integrators pass a checkpoint's weight scale directly. The SM12x fused-MoE families are synced to current b12x: the NVFP4 W4A4 backend reaches kernel parity across decode and prefill, and the W4A16 family adds cooperative persistent launches, a tensor-core decode path for small batches, and shape-stable route packing that keeps decode batch-size changes recompile-free.

• #​3932
• #​4285
• #​4255
• #​4253
• #​4130

Unified MoE API adds MXFP4 W4A8 and W4A16, shared experts, and SiTU

TRTLLM-gen MXFP4 weights now run through the unified MoELayer API against both MXFP8 activations (W4A8) and BF16 activations (W4A16), completing the FP8 series begun in 0.6.16, alongside per-tensor routed FP8. Routing adds an unpacked pre-routed FP4 mode that accepts topk_ids and topk_weights as separate contiguous tensors. TRTLLM-gen FP4 MoE also gains shared-expert fusion, extending to FP4 what 0.6.15 added for FP8, plus SiTU activation for MXFP4 x MXFP8 and NVFP4 x NVFP4.

• #​4159
• #​4088
• #​4104
• #​4239
• #​4180

Kimi K3 MLA decode, and MiniMax-M3 sparse attention under vLLM

Blackwell decode now covers Kimi K3's MLA geometry — 96 global query heads against one KV head, TP-local head counts down to 6, speculative query lengths up to 8, and context parallelism for long contexts — by packing query-token and query-head rows into shared CuTe-DSL tiles, adding compact variable-length Q, and extending TRTLLM-gen dense and sparse MLA to non-power-of-two head counts. Sparse MLA decode also serves the no-rotary-tail shape (kv_lora_rank=512, qk_rope_head_dim=0) natively. MiniMax Sparse Attention accepts vLLM's packed paged KV layout for MiniMax-M3, unblocking the vLLM integration on SM120/SM121, with lower per-call overhead on paged decode.

• #​4178
• #​4108
• #​4039
• #​4324

Ulysses sequence parallelism for long-context and video diffusion

Ulysses sequence parallelism gets its head-scatter / sequence-gather all-to-all as a public API, for video diffusion transformers and other long-sequence attention workloads. A fused NVLink-P2P kernel folds the layout permutation directly into the cross-GPU writes over CUDA IPC for a single coalesced push, with automatic NCCL fallback when P2P is unavailable.

• #​3820
• #​4240

What's Changed

• feat: close feature gap by wiring up per-tensor routed FP8 fused-moe by @​jdebache in #​4088
• Revert PR 4122 by @​jimmyzho in #​4171
• [GDN] improve sm100 GDN performance by @​Observer007 in #​4133
• fix(gdn): support WY decode on SM121 by @​kahyunnam in #​4117
• fix(norm): convert float2 to e4m3 directly in packed cast by @​elwhyjay in #​4167
• perf(test): bulk-precompile XQA decode kernels to cut test wall time ~4x by @​bkryu in #​4119
• [perf] Optimize TRT-LLM routing for high-expert, high-top-k workloads by @​jiahanc in #​4152
• feat(xqa): ragged Q and per-row sliding-window masking for speculative decode by @​yichengj0 in #​4137
• test(jit): assert BMM export symlink under GEN_SRC_DIR by @​kahyunnam in #​4187
• Feat/ulysses p2p a2a by @​forrestl111 in #​3820
• feat(moe_ep): MegaMoE framework integration ready: CUDA graph support, fused quant+stage launch, persistent knob cache, and prequantized weight packs by @​mhoqueanik in #​4079
• docs: document CuTe prefill scheduling override by @​kangbintNV in #​4162
• docs: add missing trtllm_fp8_per_tensor_scale_routed_moe API entry by @​kangbintNV in #​4175
• docs(mamba): document checkpointing varlen arguments by @​hebo1221 in #​4129
• Yanqinz/fix-gemm-and-grouped-mm-test-issue by @​yanqinz2 in #​4185
• feat(comm): extend trtllm_allreduce to SM12x and fix lamport buffer pointer packing by @​yichengj0 in #​3903
• feat(moe): add unified unpacked pre-routed FP4 mode by @​feih-nv in #​4104
• feat(mla): support packed low-head and variable-Q decode by @​PerkzZheng in #​4178
• bump version to 0.6.16 by @​jimmyzho in #​4142
• [fix]fix xqa flaky test on spark by @​qsang-nv in #​4161
• fix: make mxfp8 gemm test pass by having it quantize along the correct dimension by @​jdebache in #​3882
• fix(moe): serialize CuTe DSL autotune replay by @​zianglih in #​4192
• feat(moe_ep): fault-tolerance rank mask (NCCL-EP + NIXL-EP) by @​Anerudhan in #​4183
• fix(xqa): fix PDL load ordering and SM90 fp8 draft-mask dispatch by @​yichengj0 in #​4199
• feat(msa): accept K/V views split from a packed paged KV cache by @​yichengj0 in #​4039
• feat(moe): add TRTLLM MXFP4 W4A8 and W4A16 unified API support by @​feih-nv in #​4159
• [cli] add CLI helper for flashinfer-jit-cache and flashinfer-cubin wheel installs by @​dierksen in #​3142
• [feat] Add SITU trtllmgen MOE by @​jiahanc in #​4180
• feat(sm120): fused MoE (SwiGLU) via moe_gemm is_gated for cute SM120 group

✂ Note

PR body was truncated to here.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.