[TRTLLM-17239][feat] Locality domain support for the C++ KV cache manager v2 and two-stream attention by farazkh80 · Pull Request #20229 · NVIDIA/TensorRT-LLM

farazkh80 · 2026-10-13T14:36:55Z

Locality domain (LD) support for the KV cache on the C++ KVCacheManagerV2, plus two-stream attention. Supersedes #18919 and #19336.

Description

A locality domain is one half of a Rubin GPU with its own HBM. With enable_locality_domains (default off), a request's KV cache lives in one domain and its attention runs on that domain's SMs.

KVCacheManagerV2: one physical pool per domain, mapped into one VA reservation per pool, so slot ids and base + slotSize * id stay global. Allocation, eviction, migration, resize and quota are per domain; a page's domain follows from its slot id, and the domain id joins the reuse key. Disaggregated serving transfers each domain's pool.
Attention: one attention call per domain on its own stream with sliced metadata; MLA decode sizes its KV split for the domain's SMs.
Placement: a new request goes to the domain that already holds its longest cached prefix, unless that domain is overloaded; otherwise round-robin. On a multi-turn chat benchmark (Qwen3-4B, 64 conversations x 4 turns) this raises reused prompt tokens on later turns from 65.8% to 94.8% and cuts their time 2.1x.
docs/source/features/locality-domains.md says when to enable it.
TRT_LLM_MOCK_LOCALIZATION_SUPPORT=1 splits the KV cache on any GPU without forking streams.

Limits. Prefix reuse is per domain; a request whose prefix sits on a busy domain recomputes it. Partitioned attention rejects FlashMLA, beam search, cross attention, helix, custom masks, sparse indices and MRoPE outside the mixed input type. Sparse attention models (DSA, DeepSeek-V4, GLM k-pool) are rejected when LD is on; supporting them is a follow-up. LD weight sharding is #19272.

Test Coverage

kvCacheManagerV2LocalityTest.cpp (real and mock), localizationTest, test_kv_cache_localization{,_wrapper}.py, test_locality_domain_{partitions,dispatch,request}.py, test_cache_transceiver_locality_domains, and, Rubin only, test_locality_domain_attention_gpu.py (LD on matches LD off, GQA and MLA) and test_stream_localization_e2e.py (Qwen3-4B). All pass on Rubin at the current head; there is no Rubin CI list yet.

Performance

One decode attention layer on one GPU with a real KV cache manager and no model weights; no model was run. Heads are per GPU (the model's heads split into shards; source model in brackets). CUDA graph, 106+106 SM split, 4.75 GHz HBM.

Where the gain comes from. With LD off the KV cache is spread over both HBM halves, so each half of the GPU reads half its KV from the other half over the die-to-die link. A quarter of all reads crosses the link in each direction, which caps LD off at 4 x the per-direction link rate. LD on keeps every read in the reading half's own HBM. Raw streaming reads:

	read bandwidth
HBM peak, all SMs	13.3 TB/s
die-to-die link, per direction	3.56 TB/s
LD-off cap, 4 x 3.56	14.2 TB/s
LD off: both halves reading spread memory	14.1 TB/s, at the cap
LD on: both halves reading their own HBM	17.5 TB/s, +24%

attention shape	batch x context	LD off	LD on	delta	KV read, off -> on
GQA, 96 Q / 8 KV heads (1 shard), bf16 KV (GLM-4.7)	1024 x 32k	10031 us	7567 us	-24.6%	13.7 -> 18.2 TB/s
GQA, 48 Q / 4 KV heads (96/8 in 2 shards), bf16 KV (GLM-4.7)	256 x 16k	653 us	500 us	-23.5%	13.2 -> 17.2 TB/s
GQA, 12 Q / 1 KV head (96/8 in 8 shards), bf16 KV (GLM-4.7)	1024 x 32k	1271 us	974 us	-23.3%	13.5 -> 17.6 TB/s
MLA, 64 heads (128 in 2 shards), fp8 KV (DeepSeek-R1)	1024 x 32k	1958 us	1788 us	-8.7%	9.9 -> 10.8 TB/s
MLA, 64 heads (128 in 2 shards), fp8 KV (DeepSeek-R1)	8 x 4k	21 us	32 us	+52%	0.9 -> 0.6 TB/s

GQA decode is bandwidth-bound, so it gets nearly the whole +24%: its KV reads go from the link cap to 17-18 TB/s.
MLA decode reads less KV per step (one shared latent per token), so the link cap binds less; it gains 8-9% at high batch.
The gain is locality: the same two-kernel split with the KV in ordinary memory is no faster than LD off (e.g. GQA 96/8 at 1024 x 32k: 10503 us split vs 10031 us off).
Small batches lose: below batch 16-128, depending on the shape, splitting a small batch over two SM halves costs more than locality saves. The next step is to skip the split below a batch threshold. Default-off.

Full sweep, batch 1-1024

Every shape at batch 1-1024 x context 1k-32k, 2 repeats each. Cells are LD on / LD off: 🟩 faster in every repeat, 🟥 slower in every repeat, 🟨 tie. Geomean of LD on / LD off:

attention shape	batch 1-16	batch 32-128	batch 256-1024
MLA, 64 heads (128 in 2 shards), fp8 KV (DeepSeek-R1)	1.21	1.03	0.95
GQA, 96 Q / 8 KV heads (1 shard), bf16 KV (GLM-4.7)	1.11	0.86	0.78
GQA, 48 Q / 4 KV heads (96/8 in 2 shards), bf16 KV (GLM-4.7)	1.19	0.95	0.79
GQA, 12 Q / 1 KV head (96/8 in 8 shards), bf16 KV (GLM-4.7)	1.22	1.13	0.88

MLA, 64 heads (128 in 2 shards), fp8 KV (DeepSeek-R1)

batch	1k	2k	4k	8k	16k	32k
1	🟥 1.12	🟥 1.16	🟥 1.09	🟥 1.15	🟥 1.20	🟥 1.16
2	🟥 1.39	🟥 1.34	🟥 1.36	🟥 1.28	🟥 1.20	🟥 1.06
4	🟥 1.24	🟥 1.36	🟥 1.42	🟥 1.29	🟥 1.12	🟥 1.04
8	🟥 1.24	🟥 1.38	🟥 1.52	🟥 1.13	🟥 1.06	🟨 1.01
16	🟥 1.56	🟥 1.58	🟥 1.11	🟥 1.09	🟥 1.05	🟨 1.00
32	🟥 1.17	🟥 1.12	🟥 1.07	🟥 1.02	🟨 1.00	🟩 0.98
64	🟥 1.12	🟥 1.08	🟥 1.05	🟨 1.00	🟩 0.98	🟩 0.95
128	🟥 1.10	🟥 1.05	🟥 1.00	🟩 0.97	🟩 0.94	🟩 0.93
256	🟥 1.08	🟥 1.01	🟩 0.97	🟩 0.93	🟩 0.91	🟩 0.90
512	🟥 1.00	🟩 0.97	🟩 0.94	🟩 0.92	🟩 0.92	🟩 0.91
1024	🟩 0.98	🟩 0.96	🟩 0.94	🟩 0.93	🟩 0.92	🟩 0.91

GQA, 96 Q / 8 KV heads (1 shard), bf16 KV (GLM-4.7)

batch	1k	2k	4k	8k	16k	32k
1	🟥 1.16	🟥 1.15	🟥 1.17	🟥 1.15	🟥 1.11	🟥 1.10
2	🟥 1.21	🟥 1.18	🟥 1.12	🟥 1.04	🟩 0.99	🟩 0.99
4	🟥 1.16	🟥 1.09	🟥 1.19	🟥 1.16	🟥 1.18	🟥 1.10
8	🟥 1.27	🟥 1.17	🟥 1.24	🟥 1.22	🟥 1.07	🟩 0.94
16	🟥 1.37	🟥 1.20	🟥 1.10	🟥 1.04	🟩 0.87	🟩 0.81
32	🟥 1.03	🟨 1.07	🟩 0.93	🟩 0.88	🟩 0.82	🟩 0.81
64	🟩 0.98	🟩 0.91	🟩 0.87	🟩 0.83	🟩 0.78	🟩 0.77
128	🟩 0.91	🟩 0.85	🟩 0.81	🟩 0.79	🟩 0.77	🟩 0.76
256	🟩 0.84	🟩 0.81	🟩 0.79	🟩 0.76	🟩 0.75	🟩 0.74
512	🟩 0.82	🟩 0.80	🟩 0.78	🟩 0.76	🟩 0.76	🟩 0.75
1024	🟩 0.80	🟩 0.78	🟩 0.77	🟩 0.76	🟩 0.76	🟩 0.75

GQA, 48 Q / 4 KV heads (96/8 in 2 shards), bf16 KV (GLM-4.7)

batch	1k	2k	4k	8k	16k	32k
1	🟥 1.25	🟥 1.29	🟥 1.21	🟥 1.16	🟥 1.08	🟥 1.07
2	🟥 1.41	🟥 1.34	🟥 1.36	🟥 1.22	🟥 1.15	🟥 1.05
4	🟥 1.48	🟥 1.30	🟥 1.22	🟥 1.11	🟥 1.09	🟨 1.02
8	🟥 1.28	🟥 1.17	🟥 1.26	🟥 1.22	🟥 1.21	🟥 1.15
16	🟥 1.24	🟥 1.17	🟥 1.18	🟥 1.11	🟥 1.05	🟩 0.95
32	🟥 1.41	🟥 1.25	🟥 1.17	🟨 1.00	🟩 0.89	🟩 0.82
64	🟥 1.08	🟥 1.02	🟩 0.97	🟩 0.88	🟩 0.83	🟩 0.82
128	🟥 1.03	🟩 0.96	🟩 0.88	🟩 0.82	🟩 0.79	🟩 0.77
256	🟩 0.94	🟩 0.87	🟩 0.82	🟩 0.79	🟩 0.77	🟩 0.75
512	🟩 0.87	🟩 0.82	🟩 0.78	🟩 0.76	🟩 0.75	🟩 0.73
1024	🟩 0.82	🟩 0.81	🟩 0.78	🟩 0.76	🟩 0.76	🟩 0.75

GQA, 12 Q / 1 KV head (96/8 in 8 shards), bf16 KV (GLM-4.7)

batch	1k	2k	4k	8k	16k	32k
1	🟥 1.23	🟥 1.20	🟥 1.19	🟥 1.17	🟥 1.11	🟥 1.04
2	🟥 1.39	🟥 1.34	🟥 1.36	🟥 1.35	🟥 1.22	🟥 1.11
4	🟥 1.35	🟥 1.33	🟥 1.30	🟥 1.24	🟥 1.18	🟥 1.08
8	🟥 1.34	🟥 1.32	🟥 1.31	🟥 1.22	🟥 1.17	🟥 1.07
16	🟥 1.39	🟥 1.29	🟥 1.24	🟥 1.09	🟥 1.04	🟥 1.05
32	🟥 1.22	🟥 1.13	🟥 1.29	🟥 1.22	🟥 1.20	🟥 1.15
64	🟥 1.28	🟥 1.20	🟥 1.19	🟥 1.16	🟥 1.05	🟩 0.96
128	🟥 1.41	🟥 1.23	🟥 1.14	🟥 1.02	🟩 0.90	🟩 0.82
256	🟥 1.12	🟥 1.04	🟩 0.97	🟩 0.89	🟩 0.83	🟩 0.83
512	🟥 1.02	🟩 0.96	🟩 0.89	🟩 0.83	🟩 0.79	🟩 0.78
1024	🟩 0.93	🟩 0.86	🟩 0.84	🟩 0.79	🟩 0.77	🟩 0.77

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

yuxianq · 2026-10-15T19:37:05Z

@yuxianq, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

Assignment explanation

Components	runtime/kv-cache/layout-io (30%); runtime/kv-cache/lifecycle (25%); backends/trtllm (20%); native-ops/attention (9%); native-ops/bindings (4%); backends/shared (2%); fmha/fallback (1%); integration/attention-tests (6%); integration/build-config (1%); unmapped (2%)
Tags	features:paged-kv (100%)
PR effort	13 points — Physical KV placement, slot ownership, reuse-key compatibility, eviction/resume contracts and attention-op isolation create a broad architectural review burden. Their interacting correctness requirements justify 13 points; the estimate remains provisional because most partition execution code is truncated.
Importance	Important — Per-domain physical pools, mandatory domain validation, domain-specific reuse hashes and domain-keyed attention-op caching change allocation/reuse interfaces and sensitive concurrent attention-state ownership.
Retained GitHub review requests	schetlur-nv: preserve other-group coverage: NVIDIA/trt-llm-kv-cache-manager-devs, NVIDIA/trt-llm-disagg-devs, NVIDIA/trt-llm-qa-function, NVIDIA/trt-llm-kernels-devs, NVIDIA/trt-llm-runtime-devs, NVIDIA/trt-llm-devs
Affinity (0–1)	0.992 component/tag match; 0.994 after PIC preference
Outstanding points at selection	108 → 121 (+13)
Points already reserved on this PR	0
Effective weight	1.5
Completed review points (preceding 168 hours)	94
Combined points after assignment	215
Demand-derived target	141.00 points (soft share)
Additional fair share before assignment	0.00 points
Share explanation	Whole PR / qualification or coverage constraint exceeds a soft share; assignment is allowed.
Selection rule	affinity with half-share guard

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

farazkh80 · 2026-10-26T19:53:05Z

/bot run

tensorrt-cicd · 2026-10-26T19:59:24Z

PR_Github #80686 [ run ] triggered by Bot. Commit: 00f2d69 Link to invocation

tensorrt-cicd · 2026-10-26T22:46:43Z

PR_Github #80686 [ run ] completed with state SUCCESS. Commit: 00f2d69
/LLM/main/L0_MergeRequest_PR pipeline #66434 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

BowenFu · 2026-10-28T00:52:26Z

+                    runtime.fork()
+                try:
+                    with _locality_domain_metadata(metadata, ld_slice,
+                                                   locality_domain_id):


Could we give each locality-domain call its own CuTe DSL MLA split-plan tensor and cache key, or keep that backend on a single stream? On Rubin the default selector can choose CuteDslMlaFmha, for example H=96 FP8 decode. Its _split_kv_plan() keeps one _cute_dsl_mla_split_plan on this shared metadata. Both iterations here launch plan writers and MLA readers against it on separate streams before runtime.join(). With different KV lengths, domain 1 can overwrite split counts while domain 0's attention/reduction still reads them, producing incorrect output. The per-domain workspace overrides do not isolate that tensor. Please cover unequal per-domain KV lengths with the CuTe DSL backend and concurrent streams.

Thanks, Bowen! Fixed in 79391d2: each domain now keeps its own split plan and key. test_cute_dsl_mla_decode_with_unequal_domain_lengths covers it and passes on Rubin.

yuxianq · 2026-10-28T09:53:10Z

+
+    def pick_locality_domain(
+        self,
+        request_id: int,


Why we still pass request_id? It is unused after you switch to arrival-order round-robin method.

Good catch, removed in 83d406e.

…on and resize in the C++ KV cache manager v2 On a GPU with locality domains (Rubin), each domain's KV cache lives in its own physical memory. The C++ KVCacheManagerV2 now keeps one physical chunk pool per domain, mapped at explicit offsets of one VA reservation per pool, so slot ids stay global and base + slotSize * id is unchanged for every consumer. - Allocation, eviction, migration, resize and quota are per domain; a domain's share of the quota is taken from the raw quota on init and resize. - ReuseScope carries the domain, so prefix reuse stays within a domain. - LocalityDomainId is a StrongIndex like SlotId. - A domain chunk must start on a physical-page boundary in every pool; when the slot sizes need an oversized chunk the manager keeps one domain. - Whole-pool consumers get the mapped ranges; the per-domain free-slot count is an introspection hook for tests. - TRT_LLM_MOCK_LOCALIZATION_SUPPORT=1 splits the cache on any GPU for testing. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

…est path enable_locality_domains builds a localized KVCacheManagerV2. Each new request is placed on a domain round-robin over arrival order, carries it through every KV cache creation, probe and prefetch, and child requests inherit it. Sparse attention models are rejected at manager creation when the flag is set, and mock localization keeps attention on one stream. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

…stream With locality domains, attention forks one call per domain onto the domain's Green Context stream, with metadata sliced to that domain's requests, and joins afterwards. The decoder runner groups generation rows by domain (keeping the overlap scheduler's host-token rows first), splits by ownership when the rows are domain-sorted and by position otherwise, and builds per-domain warmup and CUDA graph dummies. Per-domain MLA decode state uses one counter allocation and reuses domain 0's sequence offsets. Speculative decoding slices its per-domain offsets. Partitioned attention rejects FlashMLA, beam search, cross attention, helix, custom masks, sparse indices and MRoPE outside the mixed input type. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

… locality domain A kernel launched on a locality-domain stream runs only on that Green Context's SMs, but MLA decode picked the trtllm-gen kernel and sized its KV split for the whole device, launching about twice the CTAs one wave could hold. The localization handle now records the SM count of every stream it creates, and MLA generation passes it to the FMHA runner in place of the device count. The lookup is a table read, so it is safe during CUDA graph capture. GQA decode keeps the device count: at batch 1-16 its kernels ran 5-10% slower with it. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

A localized pool is backed only on its domains' mapped ranges, so the transceiver registers each backed range of a pool rather than the whole extent, and the page table carries the ranges to the peer. The single-process transceiver test transfers between localized and non-localized pools, MHA and MLA, at TP 1->1, 2->1 and 1->2, with a head size that leaves a hole in the pool. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

Explains what a locality domain is, when enabling it helps (dense GQA and MLA decode attention at high batch or concurrency) and when to leave it off (low concurrency, small decode batches, sparse attention), with the config snippet. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

…KVCM2 Each Slot stored its locality domain next to its slot id, and every page rebuild and release copied or checked it, so the two could disagree. The slot id already decides the domain: each chunk of the shared slot-id space is mapped from exactly one domain's memory. GpuPoolGroup now keeps a per-chunk owner table, updated when a chunk is assigned to or returned from a domain, and answers localityDomainOf(slotId). Page::localityDomainId() and release use that lookup; Slot no longer carries a domain, and release no longer needs it to find the issuing allocator. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

…ir cached prefix Prefixes are reused only within a locality domain, so round-robin placement sent about half of a conversation's follow-up turns to the domain that had not cached its history, and those turns recomputed it. pick_locality_domain now probes each domain's radix tree with the prompt and picks the domain with the longest cached prefix when the match covers at least one block and that domain does not carry clearly more live KV caches than the others (more than max(8, a quarter of the total)). Otherwise it falls back to round-robin. On a 64-conversation, 4-turn chat with Qwen3-4B and locality domains on, the share of prompt tokens served from cache on turns 2-4 goes from 65.8% to 94.8%, and those turns run 2.1x faster. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

…n its domain count differs from the target's The target and a separate draft KV cache manager each fall back to one domain on their own slot sizes and quota, so their domain counts can differ. The target places the request, and the draft manager now maps that placement into its own domains: a one-domain draft ignores the id, and a two-domain draft under a one-domain target picks a stable domain from the request id. The draft sub-forward runs unpartitioned in both cases. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

CuTe DSL MLA decode caches its per-request split-KV plan on the attention metadata and relaunches the plan writer when its key changes. The per-domain calls slice the KV lengths, so each domain's key differs and domain 1's writer rewrote the shared plan on its stream while domain 0's kernels could still read it. Each domain now keeps its own plan and key on the metadata, swapped in for its call and reset with the shared key. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

Placement is round-robin over arrival order or follows the cached prefix, so the request id is no longer an input. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

farazkh80 · 2026-10-28T14:24:28Z

/bot run

yuxianq · 2026-10-28T14:32:41Z

Could we make matching target/draft locality-domain counts an initialization invariant?

This is not required for this PR; we can also address it in a follow-up PR if that makes sense.

Currently, each StorageManager independently applies the chunkBytes > maxChunkBytes fallback. Different quotas and coalesced slot sizes can therefore produce different domain counts on the same GPU, leading to the draft’s request-ID modulo fallback.

Suggested approach:

Resolve the domain count once from the device and shared locality configuration, then pass it to both managers.
Treat that count as immutable during manager construction; remove the independent chunk-size-based downgrade.
Validate actual backing requirements, VA reservation, and page-index bounds. If either manager cannot support the required count, fail initialization explicitly. Any desired fallback to one domain should be decided jointly before construction.
Have the draft always use the target-owned request domain ID, retaining it consistently across probing, prefetching, and allocation. Remove the mismatch-handling and modulo branches.

The allocation code suggests that removing the chunk-size fallback may not require slot padding: GpuSlotPool::resize() commits backing incrementally, and GpuPoolGroup already reserves extra VA for partial chunks. A large chunk extent therefore does not imply equally large stranded physical memory.

For the configuration-derived DeepSeek example—61 target layers, one draft layer, 61 GiB/1 GiB quotas—the existing formulas give 14,563 slots per domain for both managers with two domains. Target backing is 30.5 GiB per domain despite its 17,568 MiB chunk extent. This arithmetic has not been validated in an end-to-end GPU run.

Regression coverage should include this asymmetric layout, multiple pools with small scale buffers, growth/shrink across chunk boundaries, and request IDs 0, 2, 4, 6 verifying that both managers retain the target’s placement.

tensorrt-cicd · 2026-10-28T14:32:57Z

PR_Github #81190 [ run ] triggered by Bot. Commit: 83d406e Link to invocation

tensorrt-cicd · 2026-10-28T14:52:30Z

PR_Github #81190 [ run ] completed with state FAILURE. Commit: 83d406e
/LLM/main/L0_MergeRequest_PR pipeline #66883 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned farazkh80 Oct 13, 2026

farazkh80 added the api-compatible Accepted LLM API contract change that is backwards-compatible label Oct 13, 2026

farazkh80 force-pushed the faraz/ld-cpp-kvcm branch 3 times, most recently from 8dc54c0 to 3d44d3a Compare October 15, 2026 17:50

farazkh80 marked this pull request as ready for review October 15, 2026 19:25

farazkh80 requested review from a team as code owners October 15, 2026 19:25

farazkh80 requested review from BowenFu, PerkzZheng, VALLIS-NERIA, cascade812, dpitman-nvda, mikeiovine, nv-xtf, pcastonguay, rosong11, schetlur-nv, tongyuantongyu, weiminwang-nv, yizhang-nv and yunruis October 15, 2026 19:25

yuxianq self-requested a review October 15, 2026 19:37

farazkh80 requested review from arysef and laikhtewari and removed request for arysef and laikhtewari October 26, 2026 14:58

farazkh80 force-pushed the faraz/ld-cpp-kvcm branch from 55b71f3 to 00f2d69 Compare October 26, 2026 19:49

farazkh80 requested a review from lowsfer October 27, 2026 01:01

BowenFu reviewed Oct 27, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/kv_cache/kv_cache_manager_v2.py

farazkh80 force-pushed the faraz/ld-cpp-kvcm branch from 00f2d69 to 17b79ba Compare October 27, 2026 16:34

BowenFu reviewed Oct 28, 2026

View reviewed changes

yuxianq reviewed Oct 28, 2026

View reviewed changes

farazkh80 added 11 commits October 28, 2026 13:09

[None][chore] Drop the unused request id from pick_locality_domain

83d406e

Placement is round-robin over arrival order or follows the cached prefix, so the request id is no longer an input. Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>

farazkh80 force-pushed the faraz/ld-cpp-kvcm branch from 17b79ba to 83d406e Compare October 28, 2026 14:23

Conversation

farazkh80 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

Performance

Full sweep, batch 1-1024

PR Checklist

GitHub Bot Help

Uh oh!

yuxianq commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

farazkh80 commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

tensorrt-cicd commented Oct 26, 2026

Uh oh!

Uh oh!

BowenFu Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

farazkh80 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

yuxianq Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

farazkh80 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

farazkh80 commented Oct 28, 2026

Uh oh!

yuxianq commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

tensorrt-cicd commented Oct 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

farazkh80 commented Oct 13, 2026 •

edited

Loading

yuxianq commented Oct 15, 2026 •

edited

Loading

BowenFu Oct 28, 2026 •

edited

Loading