Skip to content

Pick gather_qmm_rhs tile size from rows per expert - #3918

Open
spokvulcan wants to merge 1 commit into
ml-explore:mainfrom
spokvulcan:perf/gather-qmm-rpe-tiles
Open

Pick gather_qmm_rhs tile size from rows per expert#3918
spokvulcan wants to merge 1 commit into
ml-explore:mainfrom
spokvulcan:perf/gather-qmm-rpe-tiles

Conversation

@spokvulcan

@spokvulcan spokvulcan commented Jul 25, 2026

Copy link
Copy Markdown

During MoE prefill, gather_qmm_rhs runs with sorted expert indices, so each expert's rows form one contiguous run. The kernel currently uses a single tile geometry (16x32) with a "TODO: Tune the block sizes" above it. The 16-row tile is the right call while runs are short, since a wider tile would straddle two experts and pay for two K-loops. But once runs reach 32 rows, a 32x64 tile is outright faster: single-run tiles plus denser MMA.

So this picks the geometry from the measured rows per expert: 32x64 when M/E >= 32, the stock 16x32 below.

From an isolated kernel sweep at Qwen3.6-35B-A3B prefill shapes (E=256, N=512/K=2048 and N=2048/K=512, 4-bit, group size 128, f16 activations, M3 Max):

  • rows per expert 32: +13.6% (gate/up shape), +12.5% (down shape)
  • rows per expert 64 to 128: +19 to 22%
  • rows per expert <= 24: 0.6 to 0.8x, the straddle cliff, which is why the threshold sits at 32

With the wide tile the kernel reaches 96% of a dense 4-bit qmm doing the same FLOPs, so most of the gather overhead is gone at large batch. End to end this is +6.2% MoE prefill at 32K context on that model.

On numerics: per-element K-accumulation order is tile-geometry-independent, and we verified that empirically rather than just arguing it. Elementwise bit-exact at every config and every rows-per-expert in the sweep, and token-identical over 34 of 34 interleaved A/B pairs on two models end to end.

Two notes for review. The 32x64 geometry is instantiated ahead of time next to the stock one, which doubles the gather_qmm_rhs variant count in the prebuilt metallib; that seemed better than leaving the win JIT-only, but it is easy to change if you would rather not grow the library. And everything here was measured on M3 Max; if you would prefer the wide tile gated by device class the way get_qmv_batch_limit does it, that is a one-line change.

There is more headroom below the threshold (small runs still sit far under the dense anchor, an occupancy problem rather than bandwidth), but that needs a real kernel change instead of tile selection, so this PR takes the cheap reliable half.

With sorted rhs indices each expert's rows form one contiguous run, and
a tile that straddles two runs pays a full K-loop per run. The stock
16x32 tile is right while runs are short, but once they reach 32 rows a
32x64 tile is outright faster: +12-14% kernel-level at rows-per-expert
32 on Qwen3.6-35B-A3B MoE prefill shapes (E=256, N=512/K=2048 and
N=2048/K=512), +19-22% at 64-128, and 0.6-0.8x below 24, hence the
threshold at 32. The expert count comes from the caller, which already
derives it for its own routing, and the new geometry is instantiated
ahead of time next to the stock one.

Bitwise: per-element K-accumulation order is tile-geometry-independent,
verified elementwise-exact at every config and rows-per-expert in the
sweep, and token-identical end to end over 34/34 A/B pairs on two
models. End-to-end effect: +6.2% MoE prefill at 32K context (M3 Max,
4-bit, group size 128).
@spokvulcan
spokvulcan force-pushed the perf/gather-qmm-rpe-tiles branch from 20f51b2 to f853cd8 Compare July 31, 2026 18:44
@spokvulcan spokvulcan changed the title [Metal] Rows-per-expert-aware gather_qmm_rhs tile geometry Pick gather_qmm_rhs tile size from rows per expert Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant