Pick gather_qmm_rhs tile size from rows per expert - #3918
Open
spokvulcan wants to merge 1 commit into
Open
Conversation
With sorted rhs indices each expert's rows form one contiguous run, and a tile that straddles two runs pays a full K-loop per run. The stock 16x32 tile is right while runs are short, but once they reach 32 rows a 32x64 tile is outright faster: +12-14% kernel-level at rows-per-expert 32 on Qwen3.6-35B-A3B MoE prefill shapes (E=256, N=512/K=2048 and N=2048/K=512), +19-22% at 64-128, and 0.6-0.8x below 24, hence the threshold at 32. The expert count comes from the caller, which already derives it for its own routing, and the new geometry is instantiated ahead of time next to the stock one. Bitwise: per-element K-accumulation order is tile-geometry-independent, verified elementwise-exact at every config and rows-per-expert in the sweep, and token-identical end to end over 34/34 A/B pairs on two models. End-to-end effect: +6.2% MoE prefill at 32K context (M3 Max, 4-bit, group size 128).
spokvulcan
force-pushed
the
perf/gather-qmm-rpe-tiles
branch
from
July 31, 2026 18:44
20f51b2 to
f853cd8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During MoE prefill, gather_qmm_rhs runs with sorted expert indices, so each expert's rows form one contiguous run. The kernel currently uses a single tile geometry (16x32) with a "TODO: Tune the block sizes" above it. The 16-row tile is the right call while runs are short, since a wider tile would straddle two experts and pay for two K-loops. But once runs reach 32 rows, a 32x64 tile is outright faster: single-run tiles plus denser MMA.
So this picks the geometry from the measured rows per expert: 32x64 when M/E >= 32, the stock 16x32 below.
From an isolated kernel sweep at Qwen3.6-35B-A3B prefill shapes (E=256, N=512/K=2048 and N=2048/K=512, 4-bit, group size 128, f16 activations, M3 Max):
With the wide tile the kernel reaches 96% of a dense 4-bit qmm doing the same FLOPs, so most of the gather overhead is gone at large batch. End to end this is +6.2% MoE prefill at 32K context on that model.
On numerics: per-element K-accumulation order is tile-geometry-independent, and we verified that empirically rather than just arguing it. Elementwise bit-exact at every config and every rows-per-expert in the sweep, and token-identical over 34 of 34 interleaved A/B pairs on two models end to end.
Two notes for review. The 32x64 geometry is instantiated ahead of time next to the stock one, which doubles the gather_qmm_rhs variant count in the prebuilt metallib; that seemed better than leaving the win JIT-only, but it is easy to change if you would rather not grow the library. And everything here was measured on M3 Max; if you would prefer the wide tile gated by device class the way get_qmv_batch_limit does it, that is a one-line change.
There is more headroom below the threshold (small runs still sit far under the dense anchor, an occupancy problem rather than bandwidth), but that needs a real kernel change instead of tile selection, so this PR takes the cheap reliable half.