Support prefill/decode disaggregation

What problem are you facing?

Prefill is compute-bound, decode is memory-bandwidth-bound. Running them on the same GPU pool means optimizing for neither. Modelplane has no way to express disaggregated serving today.

How could Modelplane help solve your problem?

LLM inference has two phases. Prefill processes all the input tokens in parallel. It's compute-bound and determines time-to-first-token. Decode generates output tokens one at a time. It's memory-bandwidth-bound and determines inter-token latency. When both phases share a GPU pool, a prefill burst interferes with in-flight decodes, causing unpredictable tail latency. You can't independently tune parallelism for each phase.

Disaggregation runs them on separate pools. A prefill instance processes the prompt, then transfers its KV cache (the intermediate state it computed for every input token) to a decode instance over RDMA. This lets you use different parallelism strategies per phase. For example TP=1 prefill replicas for throughput, TP=4 decode replicas for latency. llm-d's benchmarks show 40-50% lower end-to-end latency for workloads with long input sequences (high input:output token ratio). It's not always the right choice. Short prompts and small models don't benefit enough to justify the KV cache transfer overhead. But for large models with long contexts it's a significant win.

KServe (which Modelplane uses as its inference stack) already supports this. An optional prefill section on LLMInferenceService tells the controller to create separate Deployments for prefill and decode. KServe v0.18 ships decode templates that include the llm-d routing sidecar (which coordinates KV cache transfer via NIXL), and supports per-phase autoscaling via WVA.

I think the natural place to express disaggregation in Modelplane is split across two resources: engine configuration on ClusterModel (or Model), and the topology choice plus replica counts on ModelDeployment. The platform team declares that disagg is available for a model on certain hardware. The ML team activates it for their deployment.

The serving profile gains a topology discriminator, a prefill block with its own engine spec and per-pod resources, and optional per-profile resources for the decode side. Each phase's engine spec is complete and independent, with no merging and no shared base args. This follows the existing serving profile philosophy where each profile is a complete, tested configuration.

apiVersion: modelplane.ai/v1alpha1
kind: ClusterModel
metadata:
  name: llama-405b
spec:
  model:
    name: meta-llama/Llama-3.1-405B-Instruct
  source: HuggingFace
  huggingFace:
    repo: meta-llama/Llama-3.1-405B-Instruct
  resources:
    vram: "810Gi"

  serving:
  - name: vllm-disagg
    topology: PrefillDecode
    environmentSelector:
      matchLabels:
        modelplane.ai/rdma: "true"
    engine:
      name: vLLM
      image: vllm/vllm-openai:v0.9.1
      args:
      - "--tensor-parallel-size=4"
      - "--block-size=128"
      - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_both"}'
    resources:
      gpu: 4
      cpu: "16"
      memory: "64Gi"
    prefill:
      engine:
        name: vLLM
        image: vllm/vllm-openai:v0.9.1
        args:
        - "--tensor-parallel-size=1"
        - "--block-size=128"
        - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_both"}'
        - "--gpu-memory-utilization=0.9"
      resources:
        gpu: 1
        cpu: "8"
        memory: "16Gi"

  - name: vllm-unified
    engine:
      name: vLLM
      image: vllm/vllm-openai:v0.9.1
      args:
      - "--max-model-len=32768"
      - "--quantization=fp8"

On the ModelDeployment side, topology is explicit. It appears on both the serving profile and the deployment. The ML team opts in to disagg by setting topology: PrefillDecode and providing prefill.scaling. Both are required. There's no default ratio, because nothing else in the API has cross-resource defaulting and this shouldn't either. An ML team that doesn't set topology: PrefillDecode skips disagg profiles during matching and gets unified serving transparently.

apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: llama-405b-prod
  namespace: ml-team-a
spec:
  modelRef:
    kind: ClusterModel
    name: llama-405b
  environments: 2
  topology: PrefillDecode
  scaling:
    signal: Fixed
    fixed:
      replicas: 2
  prefill:
    scaling:
      signal: Fixed
      fixed:
        replicas: 8

Profile matching gains a topology filter. A PrefillDecode deployment only matches PrefillDecode profiles, and a Unified deployment only matches Unified profiles. If the ML team asks for disagg but no environment has an RDMA-capable disagg profile, the deployment fails visibly rather than silently falling back to unified. I think this is the right behavior. Silent degradation from disagg to unified would be a surprising performance cliff.

This work should include a bump to KServe v0.18. The prefill field exists in v0.16 (what we ship today) but v0.18 is substantially more mature. It ships decode templates with the llm-d routing sidecar built in, so Modelplane doesn't need to inject it. v0.18 also adds WVA-based autoscaling on each WorkloadSpec, which handles the per-phase scaling problem (our current Envoy-metric KEDA approach only sees traffic hitting decode pods and can't scale prefill). I'd make concurrency-based autoscaling mutually exclusive with disagg for now. The current KEDA approach doesn't generalize, and WVA is a different scaling model that deserves its own design work.

ML teams who want different P/D engine configuration (different parallelism, different P:D ratio baked into the profile itself) can create a namespaced Model with their own disagg serving profile. That's the existing break-glass path and it works here without any special handling.

Note the sketched API is stale, but we do still need to support profile/decode per the latest design/design.md.

Routing layer: a swappable endpoint-picker on an InferencePool

Multi-pod serving and prefill/decode disaggregation both need model-aware routing: send each request to the pod with the warmest prefix cache and lowest load, and for disaggregation, sequence prefill then decode. The llm-d backend doesn't do this yet. It emits a plain HTTPRoute to a Service that round-robins across gang leaders. design.md already states the intended mechanism ("llm-d adds model-aware routing and prefill/decode coordination"); this fills it in.

The seam is the Gateway API Inference Extension. An InferencePool selects the deployment's serving pods and references an endpoint-picker (EPP) through extensionRef. The gateway forwards candidate-pod metadata to the EPP over ext-proc, and the EPP returns the pod to use. The EPP is the swappable part: point extensionRef at a different picker to replace it.

The default EPP is the llm-d inference-scheduler. The picker implementation moved out of kubernetes-sigs/gateway-api-inference-extension into llm-d/llm-d-inference-scheduler; GIE now defines only the InferencePool API and the Endpoint Picker Protocol, and llm-d hosts the reference picker, with prefix-cache, KV-cache, load, and prefill/decode scorers. Defaulting to it gives prefix-aware routing without configuration while staying on the neutral protocol, so any conforming picker can replace it. Prefix-cache scoring defaults to approximate, which needs no engine wiring; precise scoring (reading vLLM KV-block state via KV-events) is a later opt-in.

The EPP is expressed the way the engine is: ML-team owned, a curated PodSpec subset, turnkey by default. A top-level optional routing block sits beside workers, since one EPP fronts the whole deployment's pods:

spec:
  workers: { ... }
  # Optional. Omit for the default llm-d EPP (approximate prefix-aware routing).
  routing:
    template:
      spec:
        containers:
        - name: epp        # image/args default to the llm-d EPP; override to swap or tune

This follows workers.template, the same curated PodSpec subset, and the engine's turnkey-default-with-override model, so there's no picker discriminator to carry.

Modelplane composes the InferencePool, the EPP Deployment from routing.template, and an HTTPRoute to the InferencePool, in place of today's plain Service routing. It is on by default only for the multi-pod (llm-d) path; single-pod native serving keeps its current Service routing. ServingStack installs the InferencePool CRDs and an ext-proc-capable Envoy Gateway per workload cluster, matching the control-plane InferenceGateway's Envoy. This pod-picking layer is separate from control-plane routing (InferenceGateway plus ModelService weighted routing across replicas); the two stack.

Prefill/decode builds on this. The prefill block in design.md provides the decode and prefill workloads (kv_producer/kv_consumer), both mount the ModelCache through the existing cache_mounts helper, the EPP's prefill/decode scorer sequences the two phases, and NIXL moves the KV cache between them over RDMA on the same cluster. One EPP serves both unified routing and disaggregation.

Open questions

One EPP per deployment, or shared per cluster (llm-d runs per-pool, which suggests per-deployment).
How this pod-picking layer and the control-plane InferenceGateway (Explore inference-aware routing on the control plane gateway #8) compose into one routing story.
Which llm-d EPP image and version to pin, and the default scorer set (prefix, KV, load; the prefill/decode scorer only when prefill is set).

Following up on per-role placement and safeguards for the prefill block.

The prefill block stays self-contained: its own nodeSelector, workers.count, topology, and template. That already lets an operator target different hardware for prefill and decode through the normal capability-matching nodeSelector, with the in-cluster scheduler doing the pool/node placement. Modelplane doesn't make that hardware choice, it exposes the knob, the same as the unified path. The reason to use it: prefill is compute-bound (wants high FLOPS) and decode is memory-bandwidth-bound (wants high memory bandwidth), so they often suit different GPU classes. The bound is co-location: prefill and decode of a replica stay on one InferenceCluster (KV transfer needs it), so distinct hardware means different pools within that cluster, not different clusters.

Constraints to document now and enforce later:

Co-location: a disaggregated replica's prefill and decode must be schedulable on one cluster. The fleet scheduler rejects the deployment if no matched cluster can host both.
Interconnect: KV transfer needs NVLink (same node) or RDMA/InfiniBand (across nodes); over PCIe or ethernet it bottlenecks. Gate on a cluster/pool capability attribute (e.g. networkInterNode), the same way other hardware requirements are matched.
Connector and model compatibility: both roles run a compatible KV connector (NixlConnector, paired kv_role) on the same model and dtype, with compatible parallelism so the KV layout matches.
Both roles explicit: a disaggregated deployment sets both workers.count (decode) and prefill.workers.count. There's no implicit prefill:decode ratio.
When to use (guidance, not a hard rule): disaggregation pays off for large models under load with strict TTFT/ITL targets, long context, and a fast interconnect. For small models, short context, or low traffic, aggregated serving (optionally with chunked prefill) is simpler and usually faster.

Opened #116 to land the design and docs for this ahead of the implementation.

It specifies disaggregation on the current backend-dispatcher architecture: a self-contained prefill block on the ModelDeployment, KV transfer over NIXL via the engine's --kv-transfer-config, and routing through the same swappable EPP on a GAIE InferencePool that multi-node serving already uses, so no bespoke prefill/decode proxy. It also adds a Disaggregated Serving section to docs/concepts.md and documents the correctness constraints (co-location, interconnect, connector/model compatibility) to enforce as the matching surface matures.

The KServe LLMInferenceService.prefill sketch in the issue body is superseded by design/disaggregation.md. No code yet; this is to make the implementation straightforward to pick up.

Add a prefill/decode disaggregation design doc

Document disaggregated serving in the concepts guide

Match the concepts guide to the ModelCache source discriminator

Add a prefill/decode disaggregation design doc

Document disaggregated serving in the concepts guide

Verify the llm-d EPP mechanism and Envoy AI Gateway for Phase 2 routing

Add disaggregated serving on top of the unopinionated engine shape

Add disaggregated serving on the unopinionated engine shape

Route multi-pod Unified serving through an endpoint picker

Navigation Menu

Repository navigation

Support prefill/decode disaggregation #34

What problem are you facing?

How could Modelplane help solve your problem?

Routing layer: a swappable endpoint-picker on an InferencePool

Open questions

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support prefill/decode disaggregation #34

Description

What problem are you facing?

How could Modelplane help solve your problem?

Activity

negz commented on May 26, 2026

dennis-upbound commented on Jun 9, 2026

Routing layer: a swappable endpoint-picker on an InferencePool

Open questions

dennis-upbound commented on Jun 9, 2026

dennis-upbound commented on Jun 9, 2026

1 remaining item

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions