Optimize large NVFP4 QMV on M5 Max

[dhiltgen]

Proposed changes

Use a narrower output tile for large NVFP4 matrix-vector products on M5 Max GPUs. Reducing the results per SIMD-group from four to two lowers register pressure for output dimensions of 4096 or greater.

Keep the existing kernel configuration for other architectures, quantization modes, and smaller output dimensions.

Add coverage for batched and unbatched large-output QMV across the supported floating-point types.

Performance

Apple M5 Max, 40-core GPU, 128 GB, macOS 26.5.2.

The optimized path is intentionally limited to M5 Max (applegpu_g17s), which is the only M5 configuration I was able to test. The narrower tile may also benefit other M5 GPUs, but I do not have the hardware to validate that tradeoff without risking regressions.

python -m mlx_lm.benchmark \
  --model mlx-community/Qwen3.5-4B-nvfp4 \
  -p 2048 -g 128 -n 9

After a warmup, baseline and candidate were run in B/A/A/B order. Results are the midpoint of the paired nine-trial medians.

Model	Prompt tok/s	Generation tok/s
Qwen3.5 4B NVFP4	5394.5 → 5416.0 (+0.4%)	128.0 → 131.5 (+2.7%)

Ollama MLX generation improved by 2.8% on Qwen3.5 4B NVFP4 and 3.9% on Qwen3.5 9B NVFP4; prompt throughput was effectively unchanged.

Checklist

Put an x in the boxes that apply.

• I have read the CONTRIBUTING document
• I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
• I have added tests that prove my fix is effective or that my feature works
• I have updated the necessary documentation (if needed)