[zheyuf]

Summary

Adds an InferenceMAX-style GSM8K accuracy eval mode so MiniMax-M3 accuracy can be validated under the exact protocol published at inferencex.semianalysis.com/evaluation (SemiAnalysisAI/InferenceX), directly comparable to the vLLM reference numbers there.

The in-repo completion-format gsm8k task measures chat/thinking-tuned models several points below their leaderboard numbers (M3-NVFP4: ~90.5 vs ~95.5 on the same engine) because it applies no chat template, runs no thinking, and caps output at 256 tokens. This PR adds the chat-protocol measurement without touching the existing one.

What the InferenceMAX protocol is (verified against their repo)

lm-eval local-chat-completions --apply_chat_template with their modified task yaml (utils/evals/gsm8k.yaml): chat template applied (thinking in template default), 5-shot as multiturn chat, doc_to_text appends "End your response with the answer on the last line, formatted as: #### [number]" (lm-eval#3411), 12288-token generation budget, full 1319 samples, greedy, em_strict (#### N) as primary score.

Changes

• tensorrt_llm/evaluate/lm_eval_tasks/gsm8k_inferencemax/gsm8k_inferencemax.yaml — their task yaml nearly verbatim (renamed task; max_gen_toks: 12288 pinned so thinking CoT is not truncated at lm-eval's 256-token default).
• GSM8KInferenceMax evaluator in evaluate/lm_eval.py (follows the AIME2026 local-task pattern), exported and registered as trtllm-eval gsm8k_inferencemax.
• GSM8KInferenceMax AccuracyTask in accuracy_core.py (MAX_OUTPUT_LEN=12288, apply_chat_template=True, fewshot_as_multiturn=True, gated on exact_match,strict-match via the standard hypothesis-test machinery).
• references/gsm8k_inferencemax.yaml — thresholds measured on 4x B200 (TP4/EP4), annotated with the published vLLM B200 FP4 band.
• test_nvfp4 / test_nvfp4_eagle3 gain eval_mode (default | inferencemax). Default mode is unchanged; inferencemax swaps MMLU+GSM8K for the chat-format task and raises max_seq_len to 16384.

Validation (4x B200, TP4/EP4, NVFP4)

Cross-framework reference (from InferenceX's public data API, MiniMax-M3 B200 FP4): vLLM Eagle3 mean 95.49 [95.15-96.06] n=8; vLLM spec-off mean 95.46 [94.84-95.91] n=13. Their identical-config repeats scatter up to ~1 point (median per-config SD 0.31 across 54 repeated configs), so single-run deltas under ~1 point are noise.

Test ID	strict-match	Status
test_nvfp4_eagle3[...overlap_scheduler=True-use_msa=True-cuda_graph=True-eval_mode=inferencemax] (3 runs)	95.53 / 95.53 / 95.00 (mean 95.35)	PASSED x3 (acceptance 0.835-0.841 / 3.50-3.52)
test_nvfp4[use_msa=False-eval_mode=inferencemax]	95.53	PASSED — matches the serve-based measurement of the same config exactly
test_nvfp4[use_msa=True-eval_mode=inferencemax]	95.68	PASSED

Standalone serve-based replication of the literal InferenceMAX command (pinned lm-eval 0.4.9.2 @ b315ef3b, their sitecustomize patch, /v1/chat/completions, concurrency 64) on the same stack: full-stack 95.38/95.45/95.60, MSA spec-off 95.91, triton spec-off 95.53 — the in-process test and the genuine serve pipeline now agree within run-to-run noise.

Protocol fidelity (how "genuine" was verified)

The in-process evaluator was audited byte-for-byte against the serve/CLI path InferenceMAX uses:

1. Chat-template rendering: identical strings for identical messages (in-process tokenizer.apply_chat_template vs the serve chat handler's inputs/utils.apply_chat_template).
2. Tokenization: identical (M3's tokenizer adds no BOS on encode; the template supplies its own prefix).
3. Few-shot exemplars: initially DIFFERENT — LmEvalEvaluator shuffles every dataset split and seeds the few-shot sampler with random_seed, while the lm-eval CLI samples shots from the unshuffled train split with sampler seed 1234. This produced a systematic ~0.45-point deficit at every engine config (measured across 18 runs, spec-on and spec-off alike). Fixed via new shuffle_dataset / fewshot_random_seed evaluator params (defaults preserve existing behavior for all other tasks); the built request messages are now byte-identical to what the lm-eval CLI sends.
4. Batching regime: inferencemax test mode caps max_batch_size at 64, matching EVAL_CONCURRENT_REQUESTS=64 and keeping decode inside CUDA-graph coverage as in real serving. (Controlled experiment note: batch size alone did not measurably shift scores; the exemplar fix was the load-bearing change.)

A 10-run fresh-engine stability battery (run under the pre-fix exemplars) showed SD 0.26 / spread 0.91 with acceptance rate 0.835-0.845 and length 3.505-3.534 — per-run noise tighter than InferenceMAX's own repeats, and no flake risk on the acceptance floors (0.78 / 3.3).

Gates

Gate	Value	Basis
GSM8K strict-match (all 3 rows)	>= 94.0, uniform explicit floor (threshold: in the references yaml)	references 95.53 / 95.68 / 95.35; measured run-to-run SD 0.26, lm-eval per-run stderr 0.58; InferenceMAX gates the same task at 90.0
Acceptance rate	> 0.78	drafter card 0.839; measured 0.835-0.845 (no ADP) / 0.819-0.846 (ADP)
Acceptance length	> 3.3	drafter card 3.518; consistent with the rate floor (AL ~= 1 + 3*rate at draft_len=3)

The test log prints the measured values next to their references (including the published vLLM numbers via reference_note), and assert messages carry the references so failures are self-explanatory. A small harness addition (threshold override on reference rows) supports the explicit floor without touching the hypothesis-test path used by every other task.

Notes for reviewers

• Runtime: the inferencemax mode is dominated by thinking output — ~5 min eval with Eagle3+CUDA graphs, tens of minutes eager. Not added to any CI test list here; the eval_mode IDs exist for opt-in runs.
• scores_filter="exact_match,strict-match" also sidesteps the known lm-eval>0.4.10 np.mean-over-metrics crash.
• The evaluator's enable_thinking=False injection heuristic does not trigger for M3 (its template uses thinking_mode), so template-default adaptive thinking is preserved — matching how InferenceMAX drives the model.

🤖 Generated with Claude Code

[zheyuf]

Gate validation: 4 consecutive fresh-engine runs against the live 94.0 floor (test_nvfp4_eagle3[...overlap_scheduler=True-use_msa=True-cuda_graph=True-eval_mode=inferencemax], final PR head):

Run	strict-match	Acceptance rate / length	Verdict
1	95.15	0.840 / 3.521	PASSED
2	95.68	0.844 / 3.533	PASSED
3	95.53	0.843 / 3.529	PASSED
4	95.68	0.841 / 3.522	PASSED

Post-exemplar-fix distribution for this config is now n=7: 95.00–95.68, mean 95.44, SD 0.26 — the 94.0 floor sits a full point below the observed minimum (~5.5 sigma below the mean). Acceptance metrics remain in the 0.835–0.845 / 3.50–3.53 band, comfortably above the 0.78 / 3.3 floors.

(A fifth run in this battery hit a CUDA OOM at engine load caused by leaked mpi4py.futures.server orphans from an earlier crashed run squatting GPU memory — an environment issue, not a gate event; noting it here for anyone who sees a similar failure mode on shared nodes.)

🤖 Generated with Claude Code

[zheyuf]

Review moved to the NVIDIA repo now that feat/m3_with_msa lives there: NVIDIA#16711 (same branch/commits). Keeping this draft open for reference until that one lands.

🤖 Generated with Claude Code