Tabrizian · Oct 29, 2026
@@ -193,13 +193,14 @@ following configuration limits at bring-up.
 
 | Configuration | Mechanism |
 |---|---|
-| Speculative decoding | Rejected draft tokens shrink a request's page list, and the freed slot goes to whichever request allocates next. The connector is never told the tail block moved. |
+| Speculative decoding | Rejected draft tokens shrink a request's page list, and the freed slot goes to whichever request allocates next. The connector is never told the tail block moved. `mooncake-store` is exempt: under a drafter it saves only prompt blocks, which are filled before the first draft. |
 | A capacity scheduler policy other than `GUARANTEED_NO_EVICT` with KV cache manager V1 | V1 does not reset the connector block delta when a request is destroyed and replayed. V2 resets that state and permits replay; an active connector load retains its allocation until completion. |
 | A host or disk cache tier | Tier eviction reassigns the GPU slot. See [KV cache tiers are GPU-only under a connector](#kv-cache-tiers-are-gpu-only-under-a-connector). |
 
 The exact set the runtime refuses depends on your cache configuration; the bring-up error is
 authoritative. Speculative decoding is unsupported with a connector wherever it is not refused —
-the same page-list shrink happens there.
+the same page-list shrink happens there — except with `mooncake-store`, which saves only prompt
+blocks while drafts are on.
 
 ##### `RequestData` fields that may not be populated
 
@@ -487,12 +488,38 @@ settings. A `capacity` server keeps both as configured.
 Checked at startup, before any request is admitted, because each one's failure mode is KV replayed
 without all of the state it was computed with — a wrong answer rather than a slow one.
 
-Beyond beam search, attention data parallelism, speculative decoding and Mamba/hybrid models, which
-are rejected for [every connector](#a-page-slot-must-not-be-reassigned-underneath-the-connector),
-this one also rejects KV cache manager V1, context parallelism, pipeline parallelism,
-sliding-window attention, and `sparse_attention_config` with an index-V cache unless
-`sparse_disable_index_value=True`; it bypasses individual requests that carry a LoRA adapter or
-multimodal content without hashes, leaving the rest of the deployment served.
+Beyond beam search, attention data parallelism and Mamba/hybrid models, which are rejected for
+[every connector](#a-page-slot-must-not-be-reassigned-underneath-the-connector), this one also
+rejects KV cache manager V1, context parallelism, pipeline parallelism, and
+`sparse_attention_config` with an index-V cache unless `sparse_disable_index_value=True`; it
+bypasses individual requests that carry a LoRA adapter or multimodal content without hashes,
+leaving the rest of the deployment served. Speculative decoding is accepted: only prompt blocks
+are saved while drafts are on.
+
+With sliding-window layers it also rejects `kv_cache_config.enable_swa_scratch_reuse`, a nonzero
+`speculative_config.context_recompute_tail`, and `scheduler_config.enable_prefix_aware_scheduling=False`.
+Each would let it save or load a window page it never verified.
+
+### Sliding-window layers
+
+A sliding-window group keeps only the blocks its window still reads, so the connector saves its
+pages once per prompt, in the prompt's last chunk: the window's sinks and tail at the prompt end.
+That is where the next turn of a conversation resumes. A lookup takes the longest full-attention
+hit, then walks back up to `block_reuse_config.swa_endpoint_rewind_tokens` for a prompt end whose
+window is complete in the pool. With none found, it looks further back for the latest earlier end
+that has one. A stored prefix is loaded through a prefix reservation: the request waits outside
+the batch while loader threads move its pages, and the server keeps computing the others.
+
+### Environment variables
+
+| Variable | Default | Effect |
+|---|---|---|
+| `TRTLLM_MOONCAKE_STORE_LOAD_THREADS` | `2` | Loader threads per loading rank. Each pins its own staging pool, which the node memory check counts. |
+| `TRTLLM_MOONCAKE_STORE_SYNC_LOAD` | `0` | `1` loads reserved prefixes synchronously in the executor's step and starts no loader threads. |
+| `TRTLLM_MOONCAKE_STORE_PROBE_CHUNK` | `4096` | Keys per existence-probe call; longer probes go out as concurrent chunks. `0` sends one call. |
+| `TRTLLM_MOONCAKE_STORE_WINDOW_FALLBACK` | `1` | `0` reports a miss when no window is stored within the rewind, instead of looking further back. |
+| `TRTLLM_MOONCAKE_STORE_WINDOW_FIRST` | `1` | `0` queues a prompt's full-attention blocks for saving before its window pages. |
+| `TRTLLM_MOONCAKE_STORE_RESAVE_MIN_TOKENS` | `0` | A prompt that computed at least this many tokens lists its full-attention blocks again at its last chunk. This refreshes the chain's head in the pool and rewrites blocks evicted during a long prefill. `0` turns it off. |
 
 ### How pages reach the pool
 

@@ -165,6 +165,10 @@ class KvCacheLayerGroupLayout:
     #: Attention window for this group, or None for full attention.
     window_size: Optional[int]
     regions: Tuple[KvCacheRegion, ...]
+    #: Leading tokens a sliding window keeps however far it has moved, or None.
+    #: With ``window_size`` it fixes which blocks the group still reads: the
+    #: same ``AttnLifeCycle`` the cache masks stale pages with.
+    num_sink_tokens: Optional[int] = None
 
     @property
     def bytes_per_page(self) -> int:
@@ -254,6 +258,11 @@ def _window_size(init_config, local_layer_id: int) -> Optional[int]:
     return None if window is None else int(window)
 
 
+def _num_sink_tokens(init_config, local_layer_id: int) -> Optional[int]:
+    sinks = getattr(init_config.layers[local_layer_id], "num_sink_tokens", None)
+    return None if sinks is None else int(sinks)
+
+
 def build_kv_cache_layout_v2(manager: "KVCacheManagerV2") -> KvCacheLayout:
     """Describe the cache's page regions for a KV connector.
 
@@ -318,6 +327,7 @@ def build_kv_cache_layout_v2(manager: "KVCacheManagerV2") -> KvCacheLayout:
                 layer_ids=tuple(global_by_local[lid] for lid in local_layer_ids),
                 window_size=_window_size(init_config, local_layer_ids[0]),
                 regions=tuple(regions),
+                num_sink_tokens=_num_sink_tokens(init_config, local_layer_ids[0]),
             )
         )
 

@@ -25,7 +25,9 @@
 bytes a page is. A change to any of those reads as a miss rather than as garbage.
 """
 
+import functools
 import hashlib
+import struct
 from dataclasses import dataclass, field
 from typing import List, Optional, Sequence, Tuple
 
@@ -123,14 +125,21 @@ def extend(self, tokens: Sequence[int]) -> Sequence[bytes]:
                 f"token list shrank from {len(self._hashes)} to {num_full_blocks} "
                 "full blocks; a hash chain belongs to exactly one request"
             )
-        for ordinal in range(len(self._hashes), num_full_blocks):
-            start = ordinal * self._tokens_per_block
-            block = tokens[start : start + self._tokens_per_block]
-            parent = self._hashes[-1] if self._hashes else self._seed
-            # Fixed-width little-endian token ids: a delimiter-free encoding
-            # would let two different token sequences serialize identically.
-            payload = b"".join(int(token).to_bytes(8, "little", signed=True) for token in block)
-            self._hashes.append(_digest(parent, payload))
+        first = len(self._hashes)
+        if first == num_full_blocks:
+            return self._hashes
+        # Fixed-width little-endian token ids: a delimiter-free encoding would
+        # let two different token sequences serialize identically. The new
+        # blocks are packed in one call and sliced per block, which gives the
+        # same payloads as packing each token, at a fraction of the cost on the
+        # scheduler's path.
+        start, end = first * self._tokens_per_block, num_full_blocks * self._tokens_per_block
+        packed = struct.pack(f"<{end - start}q", *tokens[start:end])
+        step = 8 * self._tokens_per_block
+        parent = self._hashes[-1] if self._hashes else self._seed
+        for offset in range(0, len(packed), step):
+            parent = _digest(parent, packed[offset : offset + step])
+            self._hashes.append(parent)
         return self._hashes
 
 
@@ -149,9 +158,9 @@ class KeyNamespace:
     tokens_per_block: int
     bytes_per_page: int
 
-    @property
+    @functools.cached_property
     def prefix(self) -> str:
-        """The literal string every key in this namespace starts with."""
+        """The literal string every key in this namespace starts with (computed once: a lookup builds thousands)."""
         return (
             f"{self.namespace}/{self.model_key}"
             f"/w{self.world_size}r{self.rank}"

@@ -23,7 +23,7 @@
 from dataclasses import dataclass, field
 from typing import List
 
-__all__ = ["MooncakeStoreMetadata", "PageTransfer", "RequestTransfers"]
+__all__ = ["MooncakeStoreMetadata", "PageTransfer", "PrefixLoadTransfers", "RequestTransfers"]
 
 
 @dataclass
@@ -50,13 +50,28 @@ class RequestTransfers:
     pages: List[PageTransfer] = field(default_factory=list)
 
 
+@dataclass
+class PrefixLoadTransfers:
+    """The pages of one confirmed prefix reservation.
+
+    Reported back through `get_finished_prefix_loads` once they are on the
+    device. An asynchronous one is moved off the executor thread while its
+    request waits outside the batch.
+    """
+
+    reservation_id: int
+    is_async: bool
+    transfers: RequestTransfers
+
+
 @dataclass
 class MooncakeStoreMetadata:
     """Loads to perform before the next forward pass, saves to start after it."""
 
     loads: List[RequestTransfers] = field(default_factory=list)
     saves: List[RequestTransfers] = field(default_factory=list)
+    prefix_loads: List[PrefixLoadTransfers] = field(default_factory=list)
 
     def __bool__(self) -> bool:
         """Whether there is any work at all this iteration."""
-        return bool(self.loads or self.saves)
+        return bool(self.loads or self.saves or self.prefix_loads)