[https://nvbugs/6982380][fix] Open the Harmony final channel in JSON mode eval prompts by trtllm-agent · Pull Request #21048 · NVIDIA/TensorRT-LLM

trtllm-agent · 2026-10-26T23:20:24Z

Summary

Root cause: JsonModeEval constrained the gpt-oss Harmony message header instead of the final-channel answer, because the generation prompt stops at <|start|>assistant.
Fix: Open the Harmony final channel in JSON mode eval prompts. JsonModeEval.do_apply_chat_template appends <|channel|>final<|message|> after a <|start|>assistant generation prompt, so the unchanged JSON-schema guide constrains the final message body.
Original test: bash .repair-bot/reproduce/6982380-f188d625273e-1793045351165755849/tot/driver.sh --reps 1
Automated fix generated by repair-bot

Test plan

Verify fix on the same GPU type as the original failure: the accepted reproduction passed on b200 at 83ff2187e7fc
Related regression tests were not run by repair-bot. The added CPU tests and broader regression coverage are pending PR CI.

Links

Bug: https://nvbugs/6982380

Reproduction comparison

Failed commit: reproduce failed — could not resolve the failed commit: Only failed candidate a334c95 is a release/1.3-only commit: Jenkins LLM_FUNCTION_CLUSTER_TEST/2003 (B200, param branch=release/1.3) built TensorRT-LLM at git-3 (git@g…
ToT (main) f188d62: reproduced (3 runs) — AssertionError: Reference accuracy is 62.000, threshold is 50.369. Expected accuracy >= threshold, but got 44.000. Please see hypothesis testing report:

Only the ToT run produced a result, so the two cannot be compared.

Dev Engineer Review

JsonModeEval.do_apply_chat_template now appends <|channel|>final<|message|> when the rendered prompt ends exactly with <|start|>assistant. Other prompts remain unchanged. The change does not alter the JSON schema or scoring logic. The exact-suffix condition is narrow; verify it against the intended Harmony generation prompts. Current review finding counts are unavailable.

QA Engineer Review

Added tests/unittest/evaluate/test_json_mode_eval.py and listed it in tests/integration/test_lists/test-db/l0_cpu.yml. The tests cover final-channel insertion and unchanged behavior for two other generation-prompt formats. They do not cover suffix variations or schema-constrained generation. Test execution results were not supplied. Coverage verdict: needs follow-up.

Per-File QA Perspective

tensorrt_llm/evaluate/json_mode_eval.py — Verify that the exact assistant-start suffix occurs for the intended prompts and that JSON guidance applies to the final-channel answer body.
tests/unittest/evaluate/test_json_mode_eval.py — Covers Harmony final-channel insertion and unchanged handling of two other prompt formats. Listed in the CPU test database.
tests/integration/test_lists/test-db/l0_cpu.yml — Adds the unit test to the CPU test list.

coderabbitai · 2026-10-26T23:25:27Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: NVIDIA/TensorRT-LLM
Branch: main
File: .coderabbit.yaml
Commit: 4a40ab1
Layers: Repository YAML
Review profile: CHILL
Plan: Enterprise
Run ID: 2460aa2d-8cec-4f8a-a0ee-bae762af777f

Commits

Reviewing files that changed from the base of the PR and between 83ff218 and 7791ca9.

Files selected for processing: 1

tests/integration/test_lists/test-db/l0_cpu.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

JsonModeEval appends a Harmony final-channel prefix when the templated prompt ends with the assistant-start prefix. CPU tests cover this case and confirm that two other prompt formats remain unchanged.

Changes

JSON Mode Prompt Formatting

Layer / File(s)	Summary
Add and validate Harmony prompt formatting `tensorrt_llm/evaluate/json_mode_eval.py`, `tests/unittest/evaluate/test_json_mode_eval.py`, `tests/integration/test_lists/test-db/l0_cpu.yml`	`JsonModeEval` appends the Harmony final-message prefix when the templated prompt ends with the assistant-start prefix. CPU tests cover matching and non-matching prompts. The CPU test list includes the new test.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: bowenfu

Merge Risk: Minimal · up to 7791c

No actionable merge-blocking risk is established; the new CPU test is correctly selected and the Harmony formatting branch matches the intended prompt contract.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly identifies the fix and matches the main change: opening the Harmony final channel in JSON mode evaluation prompts.
Description check		The description explains the root cause, fix, test plan, bug reference, and reproduction results. It does not reproduce the template headings exactly or explicitly complete the PR checklist, but it pr…
Docstring Coverage		Docstring coverage is inconclusive: files from earlier reviews still need to be rechecked within the check's analysis limits. Not blocking.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains the root cause, fix, test plan, bug reference, and reproduction results. It does not reproduce the template headings exactly or explicitly complete the PR checklist, but it provides the required core information.

Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-27T07:09:27Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

mzweilz

Approved from infra perspective.

…mode eval JsonModeEval constrains the whole completion with the sample's JSON schema. For gpt-oss, the chat template's generation prompt stops at `<|start|>assistant`, so the first generated tokens are the Harmony message header, where the model writes `<|channel|>...<|message|>`. The grammar forced JSON into the header position instead, which degraded the answers (JSON Mode Eval 44.00 against a 50.37 threshold for TestGPTOSS::test_eagle3_guided_decoding_4gpus). Append `<|channel|>final<|message|>` after a Harmony generation prompt so that the unchanged JSON-schema guide binds the final message body. Other chat templates and the scoring are unchanged. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>

dongfengy · 2026-10-28T15:41:06Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-28T15:48:27Z

PR_Github #81197 [ run ] triggered by Bot. Commit: 7791ca9 Link to invocation

tensorrt-cicd · 2026-10-28T22:50:02Z

PR_Github #81197 [ run ] completed with state SUCCESS. Commit: 7791ca9
/LLM/main/L0_MergeRequest_PR pipeline #66889 completed with status: 'FAILURE'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #9595: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-10-28T23:12:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-28T23:20:43Z

PR_Github #81273 [ run ] triggered by Bot. Commit: 7791ca9 Link to invocation

tensorrt-cicd · 2026-10-29T03:54:17Z

PR_Github #81273 [ run ] completed with state SUCCESS. Commit: 7791ca9
/LLM/main/L0_MergeRequest_PR pipeline #66962 completed with status: 'FAILURE'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #4142: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-10-29T04:30:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-29T04:37:57Z

PR_Github #81339 [ run ] triggered by Bot. Commit: 7791ca9 Link to invocation

tensorrt-cicd · 2026-10-29T05:20:05Z

PR_Github #81339 [ run ] completed with state FAILURE. Commit: 7791ca9
/LLM/main/L0_MergeRequest_PR pipeline #67022 completed with status: 'FAILURE'

CI Report

⚠️ Resource Maintenance — Test Coverage Reduced:

Resource maintenance in LLM/main/L0_Test-x86_64-Single-GPU #10115: L40S-PyTorch-2 is temporarily paused due to Kubernetes agent provisioning and node issues.
Affected stages:
L40S-PyTorch-2

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

dongfengy · 2026-10-29T07:51:03Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-29T07:58:32Z

PR_Github #81393 [ run ] triggered by Bot. Commit: 7791ca9 Link to invocation

tensorrt-cicd · 2026-10-29T09:51:28Z

PR_Github #81393 [ run ] completed with state SUCCESS. Commit: 7791ca9
/LLM/main/L0_MergeRequest_PR pipeline #67068 completed with status: 'SUCCESS'

CI Report

Link to invocation

trtllm-agent requested review from a team as code owners October 26, 2026 23:20

trtllm-agent requested review from BowenFu and QiJune October 26, 2026 23:20

trtllm-agent assigned dongfengy Oct 26, 2026

trtllm-agent requested review from mlefeb01 and mzweilz October 26, 2026 23:20

github-actions Bot assigned trtllm-agent Oct 26, 2026

BowenFu approved these changes Oct 27, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 27, 2026

mzweilz approved these changes Oct 28, 2026

View reviewed changes

dongfengy force-pushed the repair-bot-bug6982380 branch from 83ff218 to 7791ca9 Compare October 28, 2026 15:37

dongfengy merged commit 25e2aeb into NVIDIA:main Oct 29, 2026
11 checks passed

Conversation

trtllm-agent commented Oct 26, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Links

Reproduction comparison

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

coderabbitai Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

mzweilz left a comment

Choose a reason for hiding this comment

Uh oh!

dongfengy commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

dongfengy commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

dongfengy commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

dongfengy commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

tensorrt-cicd commented Oct 29, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

trtllm-agent commented Oct 26, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 26, 2026 •

edited

Loading