[None][feat] Add shared KV transfer ownership and progress by chienchunhung · Pull Request #20268 · NVIDIA/TensorRT-LLM

chienchunhung · 2026-10-14T05:59:38Z

Summary

Add an acquired-lease holder that separates transfer completion from permission to consume fetched KV. Return writes only after fixed delivery results, ended access, and orchestration authorization.

Scope

TransferManager retains leases, backend attempts, and the originating lender's bound return endpoint; it reports operation facts and returned bounds.
Preserve sticky abandonment, release reads/discarded writes after access ends, and retain attempts until late results are observed.
Never poll a ready lease again. Stop the lender domain after an operational return failure, retain unresolved resources, and never retry or release the failed return twice.
Keep serialized provider shutdown, owner-thread finalizers, Lifecycle arbitration, and optional executor/coordinator progress hooks.

Request lending, lender progress, distributed return agreement, readiness, and recovery remain with orchestration. This foundation does not activate a provider; DEVICE support and qualified nonfatal containment remain separate integration work.

Verification

Revision: 58b8c9be9b5e4307481d419b2c89da4c965dee15.

1,222 passed, 0 remaining failures, 4 inherited hardware skips, across the full suite and focused rerun on NVIDIA A30 with normal pytest imports and plugins.
The full suite at 1acbe240 passed 1,216 cases; six GPU cases failed at a stale test-fixture constructor. The corrected six-case module passed at this revision. Production and all other test files are byte-identical between those revisions.
Ran python3 -m pytest -c tests/unittest/pytest.ini -v -ra over holder/coordinator/loop, real-manager transfer, backend/lender, Lifecycle, executor cleanup, generation/capacity, and forward-error suites. Four inherited foundation cases require Blackwell. No provider or distributed qualification is claimed.
Used the native wheel built for [TRTLLM-16186][feat] Lend KV cache pages and recurrent state in place #20534 at bc07138a5, with matching candidate native/build sources. Repository hooks, GitHub pre-commit/DCO/metadata checks, and a fresh independent review passed. Full CI launched; results are pending.

Notes

Stacked directly on #20266 at dce241c32ab9b8b61fc40589274680d5ab734dbc, which uses #20534 at bc07138a5f8ca79088a9a6c1aece3eebdd9d95ed. Lifecycle #19822 and backend contract #20117 are already merged in that foundation.

One implementation commit follows the assembly prerequisite. Implementation-only diff.

Contribution checklist reviewed: private interfaces, focused tests, no new dependencies or public configuration fields.

chienchunhung · 2026-10-14T06:18:47Z

/bot run

tensorrt-cicd · 2026-10-14T06:26:35Z

PR_Github #77704 [ run ] triggered by Bot. Commit: b44c0bc Link to invocation

tensorrt-cicd · 2026-10-14T08:49:30Z

PR_Github #77704 [ run ] completed with state SUCCESS. Commit: b44c0bc
/LLM/main/L0_MergeRequest_PR pipeline #64058 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chienchunhung · 2026-10-14T23:39:50Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-15T00:05:06Z

PR_Github #77929 [ run ] triggered by Bot. Commit: 73f380f Link to invocation

chienchunhung · 2026-10-15T00:28:27Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-15T00:35:22Z

PR_Github #77936 [ run ] triggered by Bot. Commit: 4da6155 Link to invocation

tensorrt-cicd · 2026-10-15T00:40:30Z

PR_Github #77929 [ run ] completed with state ABORTED. Commit: 73f380f

Link to invocation

Shixiaowei02 · 2026-10-15T05:57:11Z

+
+    Backend shutdown alone runs on a daemon worker. Backend methods must satisfy
+    their nonblocking contract, while the supplied watchdog runs independently.
+    The initial policy remains conservative executor-fatal containment.


For host staging, suggest isolating the binding rather than failing the executor: keep lease, hold and registration, mark nothing delivered, stop submitting past a retention threshold, and recompute locally.

Shixiaowei02 · 2026-10-15T05:57:11Z

+            return "backend does not support the requested direction"
+        return None
+
+    def _lend(


The lender requires lending outside scheduling and batch preparation. Running lend, mark and cancel from advance() at the loop head would make that hold by construction.

Shixiaowei02 · 2026-10-15T05:57:11Z

+        *,
+        key: RequestKey,
+    ) -> StartResult:
+        """Publish committed state; a granted read may outlive the producer's exit.


A request with an in-place write lease or a lent state row must not run until release. Could the holder publish such blocking leases so scheduling can hold those requests?

Shixiaowei02 · 2026-10-15T05:57:12Z

+            deadline if self._shutdown_deadline is None else min(self._shutdown_deadline, deadline)
+        )
+        for key in tuple(self.requests):
+            self.cancel(key, CancelReason.REQUEST_EXIT)


Shutdown reuses REQUEST_EXIT; a separate reason would let parked requests get RECOMPUTE or REJECTED. On request exit, could not-yet-submitted in-place reads be abandoned, since their locks pin scheduling?

Shixiaowei02 · 2026-10-15T05:57:12Z

+        if threading.get_ident() != expected or self._shutdown_deadline is not None:
+            raise RuntimeError("backend assembly is unavailable on this thread or after shutdown")
+
+    def assemble(


Instead of a separate thread and finalize, could the loop call shutdown(0) each round and spend the remaining budget on the shutdown path, with holds returned on the manager thread?

Shixiaowei02 · 2026-10-15T05:57:12Z

+        coordinator = self.__dict__.get("_disagg_coordinator")
+        if coordinator is not None and not coordinator.shared_shutdown_complete(
+        ):
+            raise RuntimeError(


Raising here skips the KV connector and manager shutdowns, where the lender decides retention (it keeps pools while holds or locks remain). Suggest logging the incomplete cleanup and continuing.

tensorrt-cicd · 2026-10-15T06:29:21Z

PR_Github #77936 [ run ] completed with state FAILURE. Commit: 4da6155
/LLM/main/L0_MergeRequest_PR pipeline #64241 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Shixiaowei02 · 2026-10-15T08:58:00Z

+                # A violating provider cannot convert a cancellation error into ended access.
+                self._stop_binding(record.binding, f"backend cancel failed: {error}")
+
+    def advance(self, now: float) -> tuple[RuntimeUpdate, ...]:


Admission would need a per-iteration view of holding costs: requests a lease keeps from running, pages leases keep after a request is freed (per pool group), and leases retained per binding.

Shixiaowei02 · 2026-10-15T09:09:12Z

+        manager = self._shared_transfer_manager
+        return manager is None or manager.shutdown_status().complete
+
+    def _advance_shared_transfers(self) -> None:


With no shared manager attached this path should cost nothing: no collectives, no threads, no events. Suggest a test that pins that, and a trtllm-bench comparison (main, feature off, on but idle) before a profile is enabled; on but idle should stay within about 1% step time.

Shixiaowei02 · 2026-10-15T09:09:12Z

+        manager = self._shared_transfer_manager
+        return manager is not None and manager.pending()
+
+    def take_shared_updates(self) -> Tuple[RuntimeUpdate, ...]:


Cross-rank agreement (resume intervals, release, admission numbers) could travel as one message on the per-iteration allgather the transceiver already does, choosing the group as native does (pp_allgather under attention DP). Every rank joins each iteration, including one whose lend failed; no rank joins when the feature is off.

Shixiaowei02 · 2026-10-15T09:09:12Z

+            )
+        except ValueError as error:
+            return Rejected(str(error))
+        except Exception as error:


Suggest one error ladder across layers: lender argument errors are bugs (fail the request and log an error); refusals that depend on state arrive as leases that fail at the call (request level); only undocumented lender exceptions disable the domain. Request- or binding-level problems should never stop the domain or the executor.

Shixiaowei02 · 2026-10-15T09:09:12Z

+            raise RuntimeError("a shared transfer manager is already attached")
+        self._shared_transfer_manager = manager
+
+    def has_pending_shared_transfers(self) -> bool:


While both paths exist, suggest one writer per request at a time: a context request may fetch a prefix through the shared path before it computes, and the native send starts only after that fetch has returned its leases; a generation request receives through native only.

Shixiaowei02 · 2026-10-15T09:09:12Z

@@ -0,0 +1,919 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Owner-loop transfer ledger, independent of request scheduling and backend progress."""


Suggest a minimal metrics set: per binding, submissions, deliveries, recompute fallbacks, retained leases and their age, late releases and isolations; per domain, whether it is disabled; transfer bytes and latency; and transfer end kinds in the logs.

chienchunhung · 2026-10-15T23:41:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-20T20:25:36Z

PR_Github #79382 [ run ] triggered by Bot. Commit: ca16f04 Link to invocation

tensorrt-cicd · 2026-10-21T01:10:28Z

PR_Github #79382 [ run ] completed with state SUCCESS. Commit: ca16f04
/LLM/main/L0_MergeRequest_PR pipeline #65377 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Shixiaowei02 · 2026-10-22T10:51:18Z

+
+
+@dataclass(frozen=True)
+class LifecycleBinding:


RetirementWatchdog makes executor-fatal containment the only policy: one expired grace closes admission for every binding. Suggest not using the #19822 lifecycle package: deadlines belong to orchestration, access-end evidence and peer isolation to backends, lease holding and binding stop to the holder, and only local contradictions trip the executor.

Shixiaowei02 · 2026-10-22T10:51:21Z

+        if record.abandoned:
+            self._mark(record, consume=False)
+        if record.access_ended:
+            self._release(record)


Each rank returns a lease once its own evidence allows, so ranks can return in different rounds, while #20534 requires distributed transfers to return in the same iteration. Suggest collect() reporting local facts, orchestration agreeing on them over the per-round allgather, and settle(tickets) returning the same leases on every rank.

Shixiaowei02 · 2026-10-22T10:51:24Z

+        binding.lifecycle = lifecycle.watchdog.create_owner(-1, "receive", lifecycle.timeout_s)
+        self.bindings[binding_id] = binding
+        try:
+            binding.backend = factory(namespace.regions, self.notify, config, cleanup.post)


Here the holder builds backends, receiving bindings one at a time. Suggest construction in the assembly (_assembly.py): it calls each factory with data only, never the lender, resolves optional protocols once, and builds the holder once with every binding and lender, so the holder never calls a factory.

Shixiaowei02 · 2026-10-22T10:51:27Z

+            self._stop_binding(binding_id, f"backend construction failed: {error}")
+            raise
+
+    def submit(self, lease: Lease, intent: TransferIntent) -> SubmitResult:


Deferred and Rejected hand the lease back, so ownership depends on the result. Suggest take(lease, mode, hint, timeout_s=None) -> LeaseTicket, which always takes the lease: a binding not accepting, shutdown or a stopped lender becomes a fact, only TypeError and ValueError raise, and no call scans every operation.

Shixiaowei02 · 2026-10-22T10:51:33Z

+            except Exception as error:
+                self._stop_binding(record.intent.binding_id, f"backend cancel failed: {error}")
+
+    def advance(self, now: float) -> tuple[OperationObservation, ...]:


advance(now) touches every operation every round and polls leases after their view was returned, which the Lease contract rules out. Suggest four calls per round: collect(), settle(tickets) after cross-rank agreement, advance(unsettled_lenders) as the only Lender.progress() caller, and finish(), so idle rounds cost O(1) and busy rounds O(changes).

Shixiaowei02 · 2026-10-22T10:51:36Z

+        self._binding_deadline(binding, fault)
+        self._launch_shutdown(binding, self._clock() + binding.lifecycle_binding.timeout_s)
+
+    def begin_shutdown(self, deadline: float) -> None:


Suggest replacing the per-binding shutdown threads with shutdown(budget_s) -> HolderShutdown on the calling thread, repeatable and never raising: each call drives Backend.shutdown(budget), returns leases on evidence, and calls release_hold() once a binding's shutdown() returns empty with no lease unreturned. A binding stopped mid-run retires likewise, via shutdown(0) each round.

Shixiaowei02 · 2026-10-22T10:51:38Z

+
+
+@dataclass(frozen=True)
+class TransferIntent:


TransferIntent is built per operation, so each submission allocates, re-checks profile_id and memory_kind, and projects the view. Suggest a shared SubmitMode built once per kind (binding, direction, Urgency.URGENT or RECLAIM, reuse bit), with the hint and relative timeout_s passed to take, and the lease view handed to the backend unchanged.

Shixiaowei02 · 2026-10-22T10:51:41Z

+
+
+@dataclass(frozen=True)
+class OperationObservation:


One OperationObservation per changed operation allocates on every change. Suggest one columnar LeaseFacts batch per call: ticket and Fact bit arrays, bounds of returned write leases, and sparse FactDetails only for failures and attribution, each with a top-level FailureClass that orchestration decides on; DomainFailure becomes Fact.LENDER_STOPPED.

Shixiaowei02 · 2026-10-22T10:51:44Z

+    def _progress_operation(self, record: OperationRecord) -> None:
+        """Observe delivery and access independently without driving provider I/O."""
+        binding = self.bindings[record.intent.binding_id]
+        if record.timeout.is_set():


Here the holder decides expiry, after which late rows never count. Superseding the earlier suggestion to mark nothing delivered, orchestration keeps the only deadline, per request, and on expiry resumes the request at the settled bounds if access ended, otherwise waits, moves it to a new cache, or fails it.

Shixiaowei02 · 2026-10-22T10:51:52Z

+            record.submission = Submission.REJECTED
+            self._abandon(record, str(error))
+        except Exception as error:
+            record.submission = Submission.UNCERTAIN


Holding the lease on an uncertain submission is right; always stopping the whole binding is not. Suggest marking the ticket Fact.UNCERTAIN and taking the binding effect from a per-binding containment level: stop the binding, or isolate the failed peer. A stopped binding retires once its per-round shutdown(0) returns empty.

Shixiaowei02 · 2026-10-22T15:30:20Z

+    fault: str | None = None
+
+
+class TransferManager:


Suggest one lease holder shared by peer, store and third-party backends: it only takes leases, reports facts, and returns each lease exactly once on access-end evidence. Reason: two holders mean two state machines for the same ownership rules, so every safety fix lands twice and the two can drift apart.

Shixiaowei02 · 2026-10-22T15:30:23Z

+        self._shared_transfer_manager: Optional["TransferManager"] = None
+        self._shared_updates: List["OperationObservation"] = []
+
+    def attach_shared_transfer_manager(self, manager: "TransferManager") -> None:


Suggest leaving DisaggTransferCoordinator and the no-op coordinator unchanged, and adding the new path behind an executor-facing interface whose null implementation does nothing. Reason: mixing new and old logic in one class means neither path can be switched off or deleted on its own, and the null object stops being null.

Shixiaowei02 · 2026-10-22T15:30:26Z

-            len(self.waiting_queue) == 0 and not self._has_pending_connector_transfers()
+            len(self.waiting_queue) == 0 and not self._has_pending_transfers()
+
+    def _has_pending_transfers(self) -> bool:


Suggest gating stopping, draining and idle waits on state all ranks agreed on in the same round, not on rank-local holder state. Reason: the loop assumes every rank sees the stop condition together; a rank-local gate lets ranks exit in different rounds and leaves the others waiting in collectives.

Shixiaowei02 · 2026-10-22T15:30:29Z

+        """Increment one payload-free owner-thread operation counter."""
+        self.counters[key] = self.counters.get(key, 0) + 1
+
+    def notify(self) -> None:


Suggest one wake signal per backend, raised only after the backend has listed what changed, so that each round reads only those changes and an idle round does nothing. Reason: a single shared notify cannot say what changed, so per-round cost grows with transfers in flight rather than with events.

Shixiaowei02 · 2026-10-22T15:30:32Z

+class TransferManager:
+    """Hold operations on one manager thread while backend I/O progresses independently.
+
+    There is no production activation here: request admission, loop-head lending,


Suggest lending at the executor event that makes KV available or needed, admission for receives and step completion for sends, and passing that step's ready event along. Reason: lending at the loop head delays every transfer by up to a round and cannot tell when the KV was actually written.

Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung · 2026-10-23T23:33:28Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-23T23:40:07Z

PR_Github #80287 [ run ] triggered by Bot. Commit: 58b8c9b Link to invocation

tensorrt-cicd · 2026-10-24T04:28:29Z

PR_Github #80287 [ run ] completed with state FAILURE. Commit: 58b8c9b
/LLM/main/L0_MergeRequest_PR pipeline #66079 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned chienchunhung Oct 14, 2026

chienchunhung mentioned this pull request Oct 14, 2026

[#18151][feat] Admit trusted KV hints and carry immutable request metadata #20269

Open

1 task done

chienchunhung force-pushed the dev/ri-02-transfer-manager branch from d4d98fe to b44c0bc Compare October 14, 2026 06:04

chienchunhung force-pushed the dev/ri-02-transfer-manager branch from b44c0bc to 73f380f Compare October 14, 2026 23:29

chienchunhung mentioned this pull request Oct 14, 2026

[None][feat] Add shared KV runtime assembly #20266

Draft

1 task done

chienchunhung force-pushed the dev/ri-02-transfer-manager branch 2 times, most recently from e26a87d to 4da6155 Compare October 15, 2026 00:25

chienchunhung requested review from Shixiaowei02, chuangz0 and nv-xtf October 15, 2026 00:30

Shixiaowei02 reviewed Oct 15, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/disaggregation/transfer_manager.py Outdated

Comment thread tensorrt_llm/_torch/disaggregation/transfer_manager.py Outdated

Shixiaowei02 reviewed Oct 15, 2026

View reviewed changes

Shixiaowei02 requested a review from bo-nv October 15, 2026 08:54

Shixiaowei02 reviewed Oct 15, 2026

View reviewed changes

chienchunhung requested review from pcastonguay and removed request for bo-nv October 15, 2026 16:03

chienchunhung force-pushed the dev/ri-02-transfer-manager branch 2 times, most recently from dbdf18b to 90f5443 Compare October 15, 2026 23:35

chuangz0 mentioned this pull request Oct 21, 2026

[None][feat] Add a Mooncake store driver for the blob backend #20473

Open

1 task done

Shixiaowei02 reviewed Oct 22, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/disaggregation/transfer_manager.py Outdated

Shixiaowei02 reviewed Oct 22, 2026

View reviewed changes

[TRTLLM-16186][feat] lend KV cache pages and recurrent state in place

bc07138

Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

chienchunhung force-pushed the dev/ri-02-transfer-manager branch from ca16f04 to 25860d7 Compare October 23, 2026 23:09

chienchunhung changed the title ~~[None][feat] Add shared KV transfer ownership and progress (RI-02)~~ [None][feat] Add shared KV transfer ownership and progress Oct 23, 2026

chienchunhung force-pushed the dev/ri-02-transfer-manager branch from 25860d7 to 3131de2 Compare October 23, 2026 23:14

[None][feat] Add shared KV runtime assembly

dce241c

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the dev/ri-02-transfer-manager branch from 3131de2 to 1acbe24 Compare October 23, 2026 23:15

[None][feat] Add shared KV transfer ownership and progress

58b8c9b

Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>

chienchunhung force-pushed the dev/ri-02-transfer-manager branch from 1acbe24 to 58b8c9b Compare October 23, 2026 23:31

Conversation

chienchunhung commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

Verification

Notes

Uh oh!

chienchunhung commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 14, 2026

Uh oh!

chienchunhung commented Oct 14, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

chienchunhung commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Oct 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chienchunhung commented Oct 15, 2026

Uh oh!

tensorrt-cicd commented Oct 20, 2026

Uh oh!

tensorrt-cicd commented Oct 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chienchunhung commented Oct 14, 2026 •

edited

Loading