[chienchunhung]

Summary

Carry optional KV reuse hints through Python and HTTP requests while ignoring untrusted, unsupported, oversized or malformed envelopes. Freeze admitted metadata so later mutation cannot alter queued requests.

Scope

• HTTP workers and the disaggregated proxy strip untrusted hints before schema validation; TRTLLM_TRUST_KV_HINTS=1 opts in but does not authenticate clients.
• Envelope admission checks the complete envelope against the 64 KiB limit, the snapshot JSON decoder and MessagePack encoding. It ignores and counts invalid optional hints, including out-of-range integers and excessive nesting, while ordinary request validation, required P/D checks and direct typed-envelope construction remain strict.
• Immutable snapshots retain the checked JSON bytes and preserve opaque payloads through worker conversion, single-round and TP/PP broadcasting, and child requests. The three HTTP request models omit kv_hints from repr() and str() without removing it from serialized requests.
• The private candidate collector retains supported fetch actions in producer order, with unordered supporting-family sets and separate ignored-action counts; consumer integration must select and authorize one binding.
• Documentation defines ingress trust, admission limits, named-fetch boundaries, LoRA exclusion and authentication/endpoint checks required before consumer activation. The HTTP test capture endpoints now use async handlers to avoid an unnecessary AnyIO worker thread.

This PR carries metadata; it does not execute actions or activate a backend. Required P/D handoff stays separate, and gRPC hint transport is not included. Explicitly logging serialized hints still requires redaction.

Earlier verification also identified CPU test-harness defects fixed in this PR: missing subprocess timeouts, a BOLT test import that left sys.path modified, and conflicting capture fixtures in two perf-client tests.

Verification

Current source: 4a38ad70b0409159f0d1bf9b92a4c472795ff01b, verified before commit from the frozen worktree snapshot on one H200 NVL, with normal TensorRT-LLM/native imports and repository pytest plugins, including pytest-threadleak.

Target	Passed	Failed / errors / skipped
tests/unittest/llmapi/test_kv_hints.py	186	0 / 0 / 0
tests/unittest/disaggregated/test_kv_hint_selection.py	10	0 / 0 / 0
HTTP ingress matrix, additional run 1	48	0 / 0 / 0
HTTP ingress matrix, additional run 2	48	0 / 0 / 0
HTTP ingress matrix, additional run 3	48	0 / 0 / 0

196 distinct test cases passed; 340 successful executions include the three additional ingress runs. The hint suite includes 42 integer-boundary cases, 24 nesting cases and 6 request-representation/transport cases added in this revision.

python3 -m pytest -c tests/unittest/pytest.ini -v -ra \
  tests/unittest/llmapi/test_kv_hints.py \
  tests/unittest/disaggregated/test_kv_hint_selection.py

# Three additional runs with the same repository configuration and plugins:
for repeat in 1 2 3; do
  python3 -m pytest -c tests/unittest/pytest.ini -v -ra \
    tests/unittest/llmapi/test_kv_hints.py \
    -k test_http_ingress_policy_before_nested_validation
done

Full-diff self-review found no unresolved issues, and all required commit hooks passed. The signed commit has the same source content as the verified snapshot; hooks made no file changes. All 2,138 package Python files matched the frozen source before and after execution. The prebuilt native artifact came from CI 67130, source f6b302aa60a44569b319ded1d2769f6f184281ba; its full native build inputs match the PR base. This verification did not rebuild native code or run model inference, GPU-kernel correctness tests, multi-rank serving or full public CI.

Full CI for 4a38ad70b0409159f0d1bf9b92a4c472795ff01b was requested with /bot run and started as L0_MergeRequest_PR #67284. The trigger workflow, security scan and release checks passed; full CI results are pending.

The previous-head CI report, build 67157 at 1389ebfd6d65d4538c90f85a74b8e22d5cc15351, finished with 100,344 passed, 0 failed and 14,385 skipped after retries. Its initial HTTP ingress failure reported an AnyIO worker thread; the file passed on retry. The async test-handler change targets that fixture issue. Multi-GPU stages did not run because the required approval label was absent when this run started; the label was added afterward.

Earlier verification — not rerun on the current revision

At 1389ebfd6d65d4538c90f85a74b8e22d5cc15351, 1,031 tests passed, with no failures, errors or skips, in 248.28 seconds on one H200. This broader run used normal TensorRT-LLM imports, native bindings and repository pytest plugins, without import stubs or a marker filter. It included the child-request hint inheritance regression test.

python3 -m pytest -c tests/unittest/pytest.ini -v -ra \
  tests/unittest/llmapi/test_kv_hints.py \
  tests/unittest/disaggregated/test_kv_hint_selection.py \
  tests/unittest/api_stability/ \
  tests/unittest/_torch/visual_gen/test_attention_importable.py \
  tests/unittest/llmapi/test_trtllm_serve_config_schemas.py \
  tests/unittest/others/test_maintain_prims_ts.py \
  tests/unittest/others/test_vendor_bot.py \
  tests/unittest/scripts/test_bolt.py \
  tests/unittest/scripts/test_perf_agentx_aiperf_probe.py \
  tests/unittest/scripts/test_perf_agentx_client.py \
  tests/unittest/_torch/executor/test_request_utils.py \
  tests/unittest/executor/test_generation_request_array_prompt.py

All 2,138 tracked package Python files matched that source snapshot before and after the run. It used the same compatible CI 67130 native artifact described above. Full-diff self-review before and after that rebase, diff checks and required hooks passed for that revision.

CI build 66524 at 9f2fad264b50a70519764a8d2d4dc4cb0e0848a2 recorded 88,975 passed, 0 failed and 12,577 skipped. It was marked unstable because both multi-GPU dispatches were blocked by the missing approval label.

Other earlier evidence: 10 candidate tests passed at efeb4088; 157 selected tests passed at 9f2fad2; nine timeout/recovery tests passed at ee1790eb; and 694 selected tests passed on B300 at ee8242a, including the earlier subprocess-timeout and Blackwell numerical cases. No numerical threshold was changed.

Request-broadcast measurements (before rebase)

These tables are historical evidence for efeb4088ae5bf64f7472fcdac0c863ec05b7a00b. They have not been rerun on the current head. These measurements exclude admission, including the new JSON and MessagePack checks. The rebase adopted upstream request-field collection and child-request construction changes; it retained full-envelope transport.

A standalone benchmark exercised the real RequestBroadcaster with native requests and MPIDist on one AMD EPYC 7313P node with two H800 NVL GPUs. The tested source was 9f2fad264b50a70519764a8d2d4dc4cb0e0848a2 plus candidate-helper, test, and documentation changes now committed as efeb4088ae5bf64f7472fcdac0c863ec05b7a00b (patch SHA256 229e3c0157c9289471dc5f19e5856c265c31683d91cf71489ee5a2f1342ed32b). The measured broadcast code matches that pre-rebase revision. A compatible prebuilt native wheel was used; this was not a full native rebuild.

For TP2, batch size 1, and 256-token requests, each trial offered 10,000 requests/s for 5 seconds. Values below are medians across three trials.

Hint envelope	Root broadcast-call p50	Completed requests/s
None	52 µs	10,000
1 KiB	61 µs	10,000
64 KiB	97 µs	9,985

The 64 KiB case completed 9,631–10,000 requests/s across trials and accumulated backlog in some trials. Its worst trial had 189 ms p99 lateness against the synthetic arrival schedule. At batch size 32 and the same offered rate, all sizes completed approximately 10,000 requests/s. In separate saturated TP2 trials with batch size 32 (200 broadcasts per trial, three trials), component throughput fell from 78,514 requests/s without hints to 29,554 with 64 KiB (62.4% lower).

The wider matrix also covered synchronous PP2, long prompts, and batches that crossed the MPI chunk boundary. All seven runs completed, and cross-rank payload checks passed. Prebuilt requests isolate metadata collection, serialization, communication, and receiver deserialization. These results exclude admission, model execution, provider access, and serving TTFT; they do not qualify TorchDist/NCCL or multiple nodes. Large hints with short inputs are transport stress cases, not verified reuse workloads.

Full-envelope broadcast remains unchanged. The measured cost is material at the maximum envelope size. Selecting on rank 0 remains a possible optimization that needs a defined selection owner and a common decision for all ranks. The benchmark is outside the PR diff; these measurements do not establish production performance acceptance.

Complete aggregate results — all 57 measured configurations

The tables contain every non-smoke configuration from the six measurement runs. Each configuration has three trials; 57 × 3 trials plus six smoke checks give 177 cases per rank, or 354 rank-trials across two ranks. All runs completed, with no missing cases or cross-rank payload mismatches.

• Values are medians across three trials. The p50 and p99 columns each report the median of that per-trial percentile.
• Hint size 0 is the same-source no-hint baseline. Throughput changes compare the same topology, input length, batch size, and arrival rate.
• Root call duration includes metadata collection and serialization. Completed throughput includes a final barrier that waits for both ranks.
• Root CPU is process CPU time per request over the timed loop, before the final drain barrier. Root lag is lateness against the synthetic arrival schedule; it is not serving queue delay.
• Serialized batch size is the pickle payload size, not measured network wire bytes. The 64 KiB, batch-128 case crosses the 4 MiB MPI chunk threshold.
• Initial singleton trials at 10,000 offered requests/s have short windows: approximately 20 ms for TP2 and 10 ms for PP2. Use the added five-second TP2 trials for the longer high-rate comparison.
• The first batch is submitted at time zero. Finite trials can therefore report completed rates slightly above the offered rate; this does not establish extra capacity.

Smoke check

Passed six smoke cases per rank. These checks validate execution and payloads; their timings are not performance results.

TP2, 256-token requests — initial trials (200 broadcasts per trial)

Batch	Offered req/s	Hint KiB	Serialized batch KiB	Root p50/p99 µs	Completed req/s (Δ)	Root CPU µs/req	Root lag p99 ms
1	saturated	0	1.5	53.8/68.0	18196.5 (+0.0%)	54.87	—
1	saturated	1	2.6	63.3/75.1	15514.0 (-14.7%)	64.36	—
1	saturated	64	65.6	93.8/108.0	10499.7 (-42.3%)	95.11	—
1	1000.0	0	1.5	48.1/55.8	1004.4 (+0.0%)	56.48	0.058
1	1000.0	1	2.6	53.0/62.0	1004.3 (-0.0%)	60.93	0.058
1	1000.0	64	65.6	75.6/84.0	1004.2 (-0.0%)	84.07	0.057
1	10000.0	0	1.5	54.0/71.4	10005.7 (+0.0%)	57.10	0.057
1	10000.0	1	2.6	60.6/76.5	9985.8 (-0.2%)	62.75	0.057
1	10000.0	64	65.6	95.2/113.5	9954.1 (-0.5%)	98.46	0.098
32	saturated	0	36.7	406.9/420.4	78513.9 (+0.0%)	12.73	—
32	saturated	1	69.4	518.7/540.5	61378.2 (-21.8%)	16.27	—
32	saturated	64	2085.6	1080.7/1136.9	29554.4 (-62.4%)	33.82	—
32	1000.0	0	36.7	479.2/734.1	1004.9 (+0.0%)	18.67	0.096
32	1000.0	1	69.4	480.9/505.4	1004.9 (-0.0%)	15.85	0.080
32	1000.0	64	2085.6	1446.4/1499.4	1004.7 (-0.0%)	43.77	0.097
32	10000.0	0	36.7	594.1/623.4	10037.9 (+0.0%)	17.51	0.068
32	10000.0	1	69.4	534.4/566.1	10038.2 (+0.0%)	16.72	0.065
32	10000.0	64	2085.6	1053.1/1205.2	10028.4 (-0.1%)	33.09	0.068

Synchronous PP2, 256-token requests (100 broadcasts per trial)

Batch	Offered req/s	Hint KiB	Serialized batch KiB	Root p50/p99 µs	Completed req/s (Δ)	Root CPU µs/req	Root lag p99 ms
1	saturated	0	1.5	48.1/58.1	20297.1 (+0.0%)	49.12	—
1	saturated	1	2.6	53.4/62.2	18301.5 (-9.8%)	54.40	—
1	saturated	64	65.6	67.8/77.4	14449.9 (-28.8%)	69.03	—
1	1000.0	0	1.5	59.1/64.0	1008.6 (+0.0%)	67.18	0.058
1	1000.0	1	2.6	63.8/69.3	1008.6 (-0.0%)	72.09	0.058
1	1000.0	64	65.6	77.3/83.3	1008.4 (-0.0%)	85.20	0.058
1	10000.0	0	1.5	57.0/64.9	9961.3 (+0.0%)	59.87	0.058
1	10000.0	1	2.6	61.1/66.0	9953.5 (-0.1%)	63.80	0.057
1	10000.0	64	65.6	70.9/81.6	9937.5 (-0.2%)	76.19	0.057
32	saturated	0	36.7	288.0/298.3	109980.6 (+0.0%)	9.05	—
32	saturated	1	69.4	347.5/355.6	91233.9 (-17.0%)	10.91	—
32	saturated	64	2085.6	966.2/995.1	33062.3 (-69.9%)	30.18	—
32	1000.0	0	36.7	432.8/612.6	1009.9 (+0.0%)	14.20	0.094
32	1000.0	1	69.4	506.6/788.9	1009.9 (-0.0%)	17.48	0.096
32	1000.0	64	2085.6	1125.9/1421.0	1009.7 (-0.0%)	36.95	0.093
32	10000.0	0	36.7	364.8/563.7	10083.4 (+0.0%)	12.12	0.067
32	10000.0	1	69.4	551.2/574.0	10081.7 (-0.0%)	16.95	0.068
32	10000.0	64	2085.6	1058.9/1208.4	10057.6 (-0.3%)	33.85	0.065

TP2, 131,072-token requests (100 broadcasts per trial)

Batch	Offered req/s	Hint KiB	Serialized batch KiB	Root p50/p99 µs	Completed req/s (Δ)	Root CPU µs/req	Root lag p99 ms
1	saturated	0	512.5	386.7/499.5	2506.9 (+0.0%)	398.13	—
1	saturated	1	513.6	460.5/572.4	2149.4 (-14.3%)	464.15	—
1	saturated	64	576.6	467.1/556.5	2198.0 (-12.3%)	453.99	—
1	1000.0	0	512.5	247.8/268.0	1005.7 (+0.0%)	257.93	0.058
1	1000.0	1	513.6	214.4/227.4	1006.0 (+0.0%)	222.68	0.058
1	1000.0	64	576.6	233.8/242.2	1005.7 (-0.0%)	242.25	0.058
32	saturated	0	16388.9	18061.2/21056.6	1801.2 (+0.0%)	553.72	—
32	saturated	1	16421.6	18170.9/21179.4	1777.5 (-1.3%)	561.90	—
32	saturated	64	18437.9	19930.2/23110.0	1627.2 (-9.7%)	613.90	—
32	1000.0	0	16388.9	13751.1/15242.0	1004.8 (+0.0%)	433.14	0.065
32	1000.0	1	16421.6	13901.1/15231.4	1004.3 (-0.1%)	436.47	0.065
32	1000.0	64	18437.9	15345.1/16751.9	1002.7 (-0.2%)	479.61	0.065

TP2, batch 128 — MPI chunk stress (100 broadcasts per trial)

Batch	Offered req/s	Hint KiB	Serialized batch KiB	Root p50/p99 µs	Completed req/s (Δ)	Root CPU µs/req	Root lag p99 ms
128	saturated	0	145.6	1529.8/1600.5	83589.9 (+0.0%)	11.93	—
128	saturated	1	276.3	1993.3/2054.9	64072.9 (-23.3%)	15.55	—
128	saturated	64	8341.4	7906.4/8706.2	16193.7 (-80.6%)	61.74	—

TP2, batch 1 — five-second arrival windows (50,000 broadcasts per trial)

Batch	Offered req/s	Hint KiB	Serialized batch KiB	Root p50/p99 µs	Completed req/s (Δ)	Root CPU µs/req	Root lag p99 ms
1	10000.0	0	1.5	52.1/65.0	10000.0 (+0.0%)	55.19	0.057
1	10000.0	1	2.6	61.3/70.3	10000.0 (-0.0%)	62.16	0.057
1	10000.0	64	65.6	96.8/111.7	9985.0 (-0.2%)	98.35	7.430

TP2, batch 32 — 6.4-second arrival windows (2,000 broadcasts per trial)

Batch	Offered req/s	Hint KiB	Serialized batch KiB	Root p50/p99 µs	Completed req/s (Δ)	Root CPU µs/req	Root lag p99 ms
32	10000.0	0	36.7	369.6/468.1	10004.1 (+0.0%)	12.08	0.065
32	10000.0	1	69.4	442.4/556.0	10003.9 (-0.0%)	14.34	0.062
32	10000.0	64	2085.6	1073.4/1182.3	10002.9 (-0.0%)	33.76	0.062

Notes

Related to #18151 and #18153. One signed implementation commit based on main at 7d8d94684c4b3b03c835c82c75be290f051a29f6, with no dependency on the backend or lender integration PRs. The rebase preserves upstream request-field collection and carries the immutable hint snapshot into child requests.

• Contribution checklist reviewed: scoped tests and documentation included, additive APIs labeled api-compatible, and no new dependencies or LLM configuration fields.

[chienchunhung]

/bot run

[tensorrt-cicd]

PR_Github #77705 [ run ] triggered by Bot. Commit: b549b2d Link to invocation

[tensorrt-cicd]

PR_Github #77705 [ run ] completed with state SUCCESS. Commit: b549b2d
/LLM/main/L0_MergeRequest_PR pipeline #64055 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #77927 [ run ] triggered by Bot. Commit: b549b2d Link to invocation

[tensorrt-cicd]

PR_Github #77927 [ run ] completed with state FAILURE. Commit: b549b2d
/LLM/main/L0_MergeRequest_PR pipeline #64238 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

• Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
• If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
  Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #78251 [ run ] triggered by Bot. Commit: 8645ea4 Link to invocation

[chienchunhung]

@Shixiaowei02 Thanks for the review. I addressed all of your comments as part of the design update.

• Servers now ignore and count untrusted hints by default. Unknown versions and hints above 64 KiB are also ignored and counted. Invalid data in a supported version still fails validation.
• The docs clarify content naming, batch message IDs and supported entry points. The hint types are also available from llmapi.
• The docs now require endpoint authorization before any connection. The consumer integration must implement and test this check before activation. This PR only transports hints.

Please take a look; thanks!

[tensorrt-cicd]

PR_Github #78251 [ run ] completed with state FAILURE. Commit: 8645ea4
/LLM/main/L0_MergeRequest_PR pipeline #64498 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

• Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
• If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
  Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[tensorrt-cicd]

PR_Github #80778 [ run ] triggered by Bot. Commit: ee1790e Link to invocation

[tensorrt-cicd]

PR_Github #80716 [ run ] completed with state ABORTED. Commit: ee8242a

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #80786 [ run ] triggered by Bot. Commit: 9f2fad2 Link to invocation

[tensorrt-cicd]

PR_Github #80778 [ run ] completed with state ABORTED. Commit: ee1790e

Link to invocation

[tensorrt-cicd]

PR_Github #80786 [ run ] completed with state FAILURE. Commit: 9f2fad2
/LLM/main/L0_MergeRequest_PR pipeline #66524 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

• Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
• If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
  Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #81487 [ run ] triggered by Bot. Commit: 1389ebf Link to invocation

[tensorrt-cicd]

PR_Github #81487 [ run ] completed with state SUCCESS. Commit: 1389ebf
/LLM/main/L0_MergeRequest_PR pipeline #67157 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

• Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
• If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
  Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[chienchunhung]

/bot run

[tensorrt-cicd]

PR_Github #81617 [ run ] triggered by Bot. Commit: 4a38ad7 Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #81622 [ run ] triggered by Bot. Commit: 4a38ad7 Link to invocation

[tensorrt-cicd]

PR_Github #81617 [ run ] completed with state ABORTED. Commit: 4a38ad7

Link to invocation

[tensorrt-cicd]

PR_Github #81622 [ run ] completed with state FAILURE. Commit: 4a38ad7
/LLM/main/L0_MergeRequest_PR pipeline #67288 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[chienchunhung]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #81749 [ run ] triggered by Bot. Commit: 4a38ad7 Link to invocation

[chang-l]

Approved from the VisualGen side.

[tensorrt-cicd]

PR_Github #81749 [ run ] completed with state SUCCESS. Commit: 4a38ad7
/LLM/main/L0_MergeRequest_PR pipeline #67406 completed with status: 'SUCCESS'

CI Report

Link to invocation