Skip to content

[None][test] Add InferenceMAX-style GSM8K accuracy eval mode - #16711

Merged
brb-nv merged 11 commits into
NVIDIA:feat/m3_with_msafrom
zheyuf:feat/m3-inferencemax-eval
Jul 22, 2026
Merged

[None][test] Add InferenceMAX-style GSM8K accuracy eval mode#16711
brb-nv merged 11 commits into
NVIDIA:feat/m3_with_msafrom
zheyuf:feat/m3-inferencemax-eval

Conversation

@zheyuf

@zheyuf zheyuf commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Adds an InferenceMAX-style GSM8K accuracy eval mode so MiniMax-M3 accuracy can be validated under the exact protocol published at inferencex.semianalysis.com/evaluation (SemiAnalysisAI/InferenceX), directly comparable to the vLLM reference numbers there.

The existing in-repo gsm8k task measures several points below InferenceMax's leaderboard numbers (M3-NVFP4: ~90.5 vs ~95.5) because it applies no chat template, runs no thinking, and caps output at 256 tokens.

This new InferenceMAX-style mode is model-agnostic — easy to extend to other InferenceMAX-tracked models (DSR1, GPT-OSS, Kimi, GLM…)

Accuracy Thresholds for TestMiniMaxM3

InferenceMAX mode

Reference scores are from published vLLM score in inferencex.semianalysis.com, 2026-07.
We use strict threshold to ensure model quality during bringup/perf improvement.

Test Eval Reference (source) Threshold
test_nvfp4[...] GSM8K 95.46 — vLLM B200 FP4 spec-off mean [94.84–95.91] n=13 94.0
test_nvfp4_eagle3[…] GSM8K 95.49 — vLLM B200 FP4 Eagle3 mean [95.15–96.06] n=8 94.0

In my local testing, current accuracy of test_nvfp4_eagle3[tp_size=4-ep_size=4-attention_dp=True-overlap_scheduler=True-use_msa=True-cuda_graph=True-eval_mode=inferencemax] (95.35, n=3) is on par with vLLM reference (95.46).

To reviewers

@brb-nv I changed minimax-M3 Tests in QA list to InferenceMax mode. Not sure if it is a good choice.

zheyuf added 10 commits July 21, 2026 07:35
@zheyuf
[TRTLLM-14093][test] Add InferenceMAX-style GSM8K accuracy eval mode …
388db81 
…for MiniMax-M3

Add a gsm8k_inferencemax lm-eval task mirroring the public InferenceMAX
(SemiAnalysisAI/InferenceX) GSM8K protocol: chat template + 5-shot
multiturn context, an explicit '#### [number]' answer-format instruction,
a 12288-token generation budget for thinking output, and strict '#### N'
extraction. Scores are directly comparable to
inferencex.semianalysis.com/evaluation, unlike the completion-format
gsm8k task which measures chat/thinking-tuned models several points
lower (M3-NVFP4: ~90.5 vs ~95.5 on the same engine).

- tensorrt_llm/evaluate/lm_eval_tasks/gsm8k_inferencemax/: task yaml
  (InferenceX utils/evals/gsm8k.yaml nearly verbatim; renamed task,
  max_gen_toks pinned so CoT is not truncated at lm-eval's 256 default)
- GSM8KInferenceMax evaluator (AIME2026 local-task pattern), exported
  and registered as a trtllm-eval subcommand
- GSM8KInferenceMax AccuracyTask gated on em_strict via the standard
  hypothesis-test machinery; references measured on 4x B200 and
  annotated with the published vLLM B200 FP4 band (94.84-96.06)
- test_nvfp4 / test_nvfp4_eagle3 gain an eval_mode parameter
  (default | inferencemax); default mode is unchanged, inferencemax
  runs the chat-format task with max_seq_len=16384

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] True up gsm8k_inferencemax references to in-proc…
544d33c 
…ess measurements

All three inferencemax test variants pass on 4x B200: triton spec-off
95.45 (15m16s), MSA/FP8-KV spec-off 95.30 (7m06s), MSA+Eagle3+graphs
95.22 (10m24s, acceptance 0.842/3.525). The FP8-KV row is now measured
under the config the test actually runs (the provisional value came from
a BF16-KV serve run).

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Set Eagle3 gsm8k_inferencemax reference to 10-ru…
474dcf1 
…n mean

10 consecutive fresh-engine runs of test_nvfp4_eagle3[...eval_mode=
inferencemax]: all passed, strict-match 94.62-95.53, mean 95.01, SD 0.26
(tighter than InferenceMAX's own median per-config repeat SD of 0.31).
Acceptance rate 0.835-0.845 / length 3.505-3.534 across all runs.

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Reproduce InferenceMAX few-shot exemplars exactl…
4fe6c80 
…y in gsm8k_inferencemax

A byte-level audit of the in-process evaluator vs the lm-eval CLI path
(which InferenceMAX uses against the serving endpoint) showed identical
chat-template rendering and tokenization but DIFFERENT few-shot
exemplars: LmEvalEvaluator shuffles every dataset split with random_seed
and seeds the few-shot sampler with it, while the CLI samples shots from
the unshuffled train split with lm-eval's default sampler seed (1234).
The exemplar difference produced a systematic ~0.45-point offset at
every engine config, spec-on or off.

Add shuffle_dataset / fewshot_random_seed evaluator params (defaults
preserve existing behavior for all other tasks); GSM8KInferenceMax
defaults to the CLI-faithful combination. The built request messages are
now byte-identical to what the lm-eval CLI sends. Also cap
max_batch_size at 64 in inferencemax test mode to match the InferenceMAX
eval regime (client concurrency 64) and keep decode inside CUDA-graph
coverage.

Revalidated on 4x B200 under the fixed protocol: triton spec-off 95.53
(exactly matches the serve-based measurement), MSA/FP8-KV spec-off
95.68, Eagle3+graphs+overlap 95.53/95.53/95.00 (mean 95.35, vs published
vLLM B200 FP4 Eagle3 mean 95.49). References updated accordingly.

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Fix test-list IDs for eval_mode param; print ext…
cf5010c 
…ernal reference in accuracy report

- Update QA list, l0_dgx_b200, and waives.txt entries for TestMiniMaxM3
  test_nvfp4 / test_nvfp4_eagle3 to the new parametrize IDs
  (-eval_mode=default), keeping current CI behavior unchanged. The
  waives.txt entries are updated too so the B200/B300/GB300 waivers keep
  matching (the AST validator does not check waives).
- Restructure the spots pre-commit's yapf mangled (pre-computed
  batch_kwargs / max_batch_size / fewshot_seed variables).
- Add an optional reference_note field to accuracy reference rows,
  printed in the hypothesis-testing report; gsm8k_inferencemax rows now
  carry the published InferenceMAX vLLM B200 FP4 scores for direct
  comparison in test logs.

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Uniform explicit 94.0 accuracy floor; print acce…
c64ee1a 
…ptance references

- Add an optional per-row 'threshold' override to the accuracy reference
  yaml (HypothesisTestingParams.threshold_override): when set it replaces
  the sigma-derived hypothesis-test threshold with an explicit
  benchmark-style floor. All three gsm8k_inferencemax rows gate at a
  uniform 94.0 (InferenceMAX gates the same task at 90.0; our measured
  run-to-run SD is 0.26 and lm-eval per-run stderr 0.58, so 94.0 keeps
  >2-sigma headroom below the weakest reference, 95.35).
- Acceptance gates stay at their original values (rate > 0.78,
  length > 3.3 — mutually consistent: at draft_len=3, AL ~= 1 + 3*rate),
  and the test now prints the drafter-card references
  (Inferact/MiniMax-M3-EAGLE3: 0.839 / 3.518) after the checks and in
  the assert messages, so failures are self-explanatory.
- Validated live: fresh-engine runs pass the 94.0 gate (95.15 strict
  on the first post-change run; more repeats in flight).

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Trim acceptance-gate comment to reference and th…
e5e0cab 
…resholds

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Report explicit-floor gates without hypothesis-t…
1e5be7e 
…est statistics

With a threshold override active, printing Alpha/Beta/Sigma/Theta
suggests a sigma-derived threshold that differs from the printed one;
show only the fields that apply.

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Use published InferenceMAX scores as gsm8k_infer…
093357a 
…encemax references

Reference accuracies are now the published vLLM B200 FP4 means (the
directly comparable numbers under this protocol), with the source
rendered inline on the Reference line instead of a separate External
reference row. Our own TRT-LLM measurements move to per-row comments;
gating is unchanged (explicit 94.0 floor).

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf
[TRTLLM-14093][test] Include threshold and reference source in accept…
7167e1a 
…ance assert messages

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf zheyuf mentioned this pull request Jul 22, 2026
Closed
@zheyuf
zheyuf force-pushed the feat/m3-inferencemax-eval branch from b68dca1 to 7167e1a Compare July 22, 2026 05:50
@zheyuf
[TRTLLM-14093][test] Switch MiniMax-M3 QA accuracy entries to the Inf…
6da0761 
…erenceMAX protocol

All four QA rows now gate on eval_mode=inferencemax, directly comparable
to the published InferenceMAX scores. Measured on 4x B200 (all under the
existing TIMEOUT 180): test_nvfp4 triton 24:56 (95.53), test_nvfp4 MSA
11:27 (95.68), eagle3 adp=False 7:01 (95.35 mean of 3), eagle3 adp=True
7:16 (95.91, acceptance 0.829/3.488 — first-run validated). The
use_msa=False platform waivers (B200/B300/GB300 nvbugs) move to the
inferencemax ID with the entry; l0_dgx_b200 keeps its default-mode
use_msa=True entry, so that ID's coverage is unchanged.

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf zheyuf changed the title [TRTLLM-14093][test] Add InferenceMAX-style GSM8K accuracy eval mode for MiniMax-M3 [None][test] Add InferenceMAX-style GSM8K accuracy eval mode for MiniMax-M3 Jul 22, 2026
@zheyuf zheyuf changed the title [None][test] Add InferenceMAX-style GSM8K accuracy eval mode for MiniMax-M3 [None][test] Add InferenceMAX-style GSM8K accuracy eval mode Jul 22, 2026
@zheyuf
zheyuf requested review from brb-nv and pcicotti July 22, 2026 06:50
@zheyuf
zheyuf marked this pull request as ready for review July 22, 2026 06:53
@zheyuf
zheyuf requested review from a team as code owners July 22, 2026 06:53
@brb-nv

brb-nv commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

@brb-nv I changed minimax-M3 Tests in QA list to InferenceMax mode. Not sure if it is a good choice.

I like this at least for the side branch.

brb-nv
brb-nv approved these changes Jul 22, 2026
View reviewed changes

@brb-nv brb-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me!

@brb-nv
brb-nv merged commit 7f7365a into NVIDIA:feat/m3_with_msa Jul 22, 2026
55 checks passed
brb-nv pushed a commit to brb-nv/TensorRT-LLM that referenced this pull request Jul 23, 2026
@zheyuf @brb-nv
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode (NVIDIA#…
d030341 
…16711)

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
ZhanruiSunCh pushed a commit that referenced this pull request Jul 29, 2026
@zheyuf @brb-nv
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode (#16711)
cd4dae9 
Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
brb-nv pushed a commit to brb-nv/TensorRT-LLM that referenced this pull request Aug 25, 2026
@zheyuf @brb-nv
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode (NVIDIA#…
dcaf83a 
…16711)

Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com>
@zheyuf zheyuf mentioned this pull request Sep 4, 2026
Merged
1 task done
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 10, 2026
@zheyuf
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode
e9b9ae4 
Adds a `gsm8k_inferencemax` lm-eval task and evaluator that reproduce the
public InferenceMAX (SemiAnalysisAI/InferenceX) GSM8K protocol: chat
template with 5-shot multiturn exemplars, an explicit "#### [number]"
answer-format instruction, a 12288-token generation budget for thinking
models, strict "#### N" extraction, and the lm-eval CLI's exemplar
selection (unshuffled train split, sampler seed 1234). Scores are directly
comparable to inferencex.semianalysis.com/evaluation, unlike the
completion-format `gsm8k` task, which runs several points lower on
chat/thinking-tuned models.

The accuracy harness gains a matching `GSM8KInferenceMax` task plus
optional `threshold` / `reference_note` fields on reference rows, so a
benchmark-style explicit floor with published provenance can replace the
hypothesis-test threshold. `TestMiniMaxM3::test_nvfp4` is parametrized by
`eval_mode` (default = MMLU + GSM8K, inferencemax = chat GSM8K at 16k
context); the QA list runs the inferencemax variants, pre-merge keeps the
default variant.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711). On main the
`gsm8k_inferencemax` command also carries the `--num_fewshot` and
`--stop_strings` knobs that `gsm8k` gained since, the task is listed in the
trtllm-eval documentation, and CPU unit tests pin the exemplar selection
and the CLI parity with `gsm8k`.

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 10, 2026
@zheyuf
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode
904062d 
Adds a `gsm8k_inferencemax` lm-eval task and evaluator that reproduce the
public InferenceMAX (SemiAnalysisAI/InferenceX) GSM8K protocol: chat
template with 5-shot multiturn exemplars, an explicit "#### [number]"
answer-format instruction, a 12288-token generation budget for thinking
models, strict "#### N" extraction, and the lm-eval CLI's exemplar
selection (unshuffled train split, sampler seed 1234). Scores are directly
comparable to inferencex.semianalysis.com/evaluation, unlike the
completion-format `gsm8k` task, which runs several points lower on
chat/thinking-tuned models. Exposed as `trtllm-eval gsm8k_inferencemax`
with the same override knobs as `gsm8k`, documented in the trtllm-eval
page, and pinned by CPU unit tests (exemplar selection, CLI parity).

The accuracy harness gains a matching `GSM8KInferenceMax` task plus
optional `threshold` / `reference_note` fields on reference rows, so a
benchmark-style explicit floor with published provenance can replace the
hypothesis-test threshold; the MiniMax-M3-NVFP4 reference rows carry the
published InferenceMAX vLLM numbers. The accuracy-test arm that consumes
them (the MiniMax-M3 Eagle3 test) follows once that test is on main.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 10, 2026
@zheyuf
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode
4bacc3f 
Adds a `gsm8k_inferencemax` lm-eval task and evaluator that reproduce the
public InferenceMAX (SemiAnalysisAI/InferenceX) GSM8K protocol: chat
template with 5-shot multiturn exemplars, an explicit "#### [number]"
answer-format instruction, a 12288-token generation budget for thinking
models, strict "#### N" extraction, and the lm-eval CLI's exemplar
selection (unshuffled train split, sampler seed 1234). Scores are directly
comparable to inferencex.semianalysis.com/evaluation, unlike the
completion-format `gsm8k` task, which runs several points lower on
chat/thinking-tuned models. Exposed as `trtllm-eval gsm8k_inferencemax`
with the same override knobs as `gsm8k` and documented in the trtllm-eval
page.

The accuracy harness gains a matching `GSM8KInferenceMax` task plus
optional `threshold` / `reference_note` fields on reference rows, so a
benchmark-style explicit floor with published provenance can replace the
hypothesis-test threshold; the MiniMax-M3-NVFP4 reference rows carry the
published InferenceMAX vLLM numbers. The accuracy-test arm that consumes
them (the MiniMax-M3 Eagle3 test) follows once that test is on main.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 10, 2026
@zheyuf
[None][test] Add InferenceMAX-style GSM8K accuracy eval mode
409e62c 
Adds a `gsm8k_inferencemax` lm-eval task and evaluator that reproduce the
public InferenceMAX (SemiAnalysisAI/InferenceX) GSM8K protocol: chat
template with 5-shot multiturn exemplars, an explicit "#### [number]"
answer-format instruction, a 12288-token generation budget for thinking
models, strict "#### N" extraction, and the lm-eval CLI's exemplar
selection (unshuffled train split, sampler seed 1234). Scores are directly
comparable to inferencex.semianalysis.com/evaluation, unlike the
completion-format `gsm8k` task, which runs several points lower on
chat/thinking-tuned models. Exposed as `trtllm-eval gsm8k_inferencemax`
with the same override knobs as `gsm8k` and documented in the trtllm-eval
page.

The accuracy harness gains a matching `GSM8KInferenceMax` task plus
optional `threshold` / `reference_note` fields on reference rows, so a
benchmark-style explicit floor with published provenance can replace the
hypothesis-test threshold; the MiniMax-M3-NVFP4 reference rows carry the
published InferenceMAX vLLM numbers. The accuracy-test arm that consumes
them (the MiniMax-M3 Eagle3 test) follows once that test is on main.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 14, 2026
@zheyuf @claude
[None][test] Add InferenceX-style GSM8K accuracy eval mode
e1687a2 
Adds a `gsm8k_inferencex` lm-eval task and evaluator that reproduce the
public InferenceX (formerly InferenceMAX; SemiAnalysisAI/InferenceX)
GSM8K protocol: chat template with 5-shot multiturn exemplars, an
explicit "#### [number]" answer-format instruction, a 12288-token
generation budget for thinking models, strict "#### N" extraction, and
the lm-eval CLI's exemplar selection (unshuffled train split, sampler
seed 1234). Scores are directly comparable to
inferencex.semianalysis.com/evaluation, unlike the completion-format
`gsm8k` task, which runs several points lower on chat/thinking-tuned
models. Exposed as `trtllm-eval gsm8k_inferencex` with the same override
knobs as `gsm8k` and documented in the trtllm-eval page.

The accuracy harness gains a matching `GSM8KInferenceX` task plus
optional `threshold` / `reference_note` fields on reference rows, so a
benchmark-style explicit floor with published provenance can replace the
hypothesis-test threshold; the MiniMax-M3-NVFP4 reference rows carry the
published InferenceX vLLM numbers.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 15, 2026
@zheyuf @claude
[None][test] Add InferenceX-style GSM8K accuracy eval mode
da7ba6d 
Adds a `gsm8k_inferencex` lm-eval task and evaluator reproducing the
InferenceX (formerly InferenceMAX) GSM8K protocol: chat template, 5-shot
multiturn, a 12288-token generation budget, strict "#### N" extraction and
lm-eval's default exemplar selection, so scores are comparable to
inferencex.semianalysis.com/evaluation. Exposed as `trtllm-eval
gsm8k_inferencex`. The accuracy harness gains a matching `GSM8KInferenceX`
task and an optional explicit `threshold` on reference rows.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
zheyuf added a commit to zheyuf/TensorRT-LLM that referenced this pull request Sep 15, 2026
@zheyuf @claude
[None][test] Add InferenceX-style GSM8K accuracy eval mode
8f45fd1 
Adds a `gsm8k_inferencex` lm-eval task and evaluator reproducing the
InferenceX (formerly InferenceMAX) GSM8K protocol: chat template, 5-shot
multiturn, a 12288-token generation budget, strict "#### N" extraction and
lm-eval's default exemplar selection, so scores are comparable to
inferencex.semianalysis.com/evaluation. Exposed as `trtllm-eval
gsm8k_inferencex`. The accuracy harness gains a matching `GSM8KInferenceX`
task and an optional explicit `threshold` on reference rows.

Ported from the feat/m3_with_msa side branch (NVIDIA#16711).

Signed-off-by: Zheyu Fu <zheyuf@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcicotti pcicotti pcicotti approved these changes
@brb-nv brb-nv brb-nv approved these changes

Assignees

@zheyuf zheyuf

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zheyuf @brb-nv @pcicotti