Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,243 changes: 1,243 additions & 0 deletions docs/content/guides/inference_guide.md

Large diffs are not rendered by default.

Binary file added docs/content/media/inference-stack.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 5 additions & 2 deletions docs/content/models/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,8 +175,11 @@ The sidecar and the picker must match your engines:
| vLLM with `MooncakeConnector` | `mooncake` | Any |
| SGLang, any transfer backend | `sglang` | `always-disagg-pd-decider`. An SGLang decode engine rejects a request that skips prefill. |

The sidecar's `--vllm-port` names the decode engine's port, whatever the engine

<!-- vale ai-tells.BareNames = NO -->
The sidecar's `--vllm-port` specifies the decode engine's port, whatever the engine
is.
<!-- vale ai-tells.BareNames = YES -->

A decode pod without the sidecar sends every request to decode alone. vLLM
serves it, and SGLang rejects it.
Expand Down Expand Up @@ -219,7 +222,7 @@ and no other container declares it. A `Worker` member doesn't need a container
on 8000, because worker pods don't receive requests.

Behind the sidecar, the decode port appears in three places: the `--port` flag
on the engine, its `http` port, and the sidecar flag that names the engine's
on the engine, its `http` port, and the sidecar flag that specifies the engine's
port. Keep the three the same. Modelplane doesn't read engine args to check
them.

Expand Down
2 changes: 1 addition & 1 deletion docs/utils/vale/.vale.ini
Original file line number Diff line number Diff line change
Expand Up @@ -55,5 +55,5 @@ alex.ProfanityMaybe = NO
alex.ProfanityUnlikely = NO
alex.Suicide = NO

TokenIgnores = (\(#.*\)),(\]\(),(http.*),(\!\[),(\d.*px),(\d\.\d\.\d),(\{\{<[^>]+>\}\})
TokenIgnores = (\(#.*\)),(\]\(),(https?://[^\s)]+),(\!\[),(\d.*px),(\d\.\d\.\d),(\{\{<[^>]+>\}\})
BlockIgnores = (\{\{<\s*\/?.*>\}\}),(\{\{\<\/\*.*\*\/\>\}\})
1 change: 1 addition & 0 deletions docs/utils/vale/styles/Modelplane/Spelling.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ filters:
- '#\w+'
- \d+\.\d+[BbMmKkTt]
- \w+'s
- '\w+(-\w+)+'
107 changes: 83 additions & 24 deletions docs/utils/vale/styles/config/vocabularies/Modelplane/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -200,12 +200,12 @@ TTFT
A[0-9]+B
IQ4_XS
llama\.cpp
multi-GPU
[Mm]ulti-GPU
[Vv][Rr][Aa][Mm]
tensor
[Tt]ensor
tokenizer
[Ii]nference
prefill
[Pp]refill
[Cc]loud

# Cloud providers
Expand Down Expand Up @@ -324,8 +324,8 @@ stdin
stdout
subnets?
tolerations
pipeline
parallelism
[Pp]ipeline
[Pp]arallelism
[Aa]utoscaling
config
args
Expand All @@ -352,7 +352,7 @@ base64-encoded
bring-your-own
built-in
call-outs
cache-aware
[Cc]ache-aware
cloud-native
distributed-systems
performance-sensitive
Expand All @@ -372,7 +372,7 @@ double-check
dry-run
dynamic-PVC
e\.g\.
end-to-end
[Ee]nd-to-end
EKS-based
event-driven
first-class
Expand Down Expand Up @@ -402,7 +402,7 @@ multi-pod
multi-region
multi-tenant
multi-tenancy
mixture-of-experts
[Mm]ixture-of-experts
namespace-scoped
namespaced
namespaces?
Expand Down Expand Up @@ -431,7 +431,7 @@ resource-intensive
self-contained
self-service
self-signed
server-side
[Ss]erver-side
serving-stack
[Ss]ingle-node
single-pod
Expand Down Expand Up @@ -462,8 +462,8 @@ workload-cluster
[Rr]e-converge[sd]?

# Style exceptions
several
above
[Ss]everal
[Aa]bove
below
the cloud
disabled?
Expand Down Expand Up @@ -550,7 +550,7 @@ routable
pipeline-parallel
Qwen2.5-14B

agent
[Aa]gent
requirement
PrefillDecode

Expand Down Expand Up @@ -810,12 +810,12 @@ TTFT
A[0-9]+B
IQ4_XS
llama\.cpp
multi-GPU
[Mm]ulti-GPU
[Vv][Rr][Aa][Mm]
tensor
[Tt]ensor
tokenizer
[Ii]nference
prefill
[Pp]refill
[Cc]loud

# Cloud providers
Expand Down Expand Up @@ -934,8 +934,8 @@ stdin
stdout
subnets?
tolerations
pipeline
parallelism
[Pp]ipeline
[Pp]arallelism
[Aa]utoscaling
config
args
Expand All @@ -962,7 +962,7 @@ base64-encoded
bring-your-own
built-in
call-outs
cache-aware
[Cc]ache-aware
cloud-native
distributed-systems
performance-sensitive
Expand All @@ -982,7 +982,7 @@ double-check
dry-run
dynamic-PVC
e\.g\.
end-to-end
[Ee]nd-to-end
EKS-based
event-driven
first-class
Expand Down Expand Up @@ -1012,7 +1012,7 @@ multi-pod
multi-region
multi-tenant
multi-tenancy
mixture-of-experts
[Mm]ixture-of-experts
namespace-scoped
namespaced
namespaces?
Expand Down Expand Up @@ -1041,7 +1041,7 @@ resource-intensive
self-contained
self-service
self-signed
server-side
[Ss]erver-side
serving-stack
[Ss]ingle-node
single-pod
Expand Down Expand Up @@ -1072,8 +1072,8 @@ workload-cluster
[Rr]e-converge[sd]?

# Style exceptions
several
above
[Ss]everal
[Aa]bove
below
the cloud
disabled?
Expand Down Expand Up @@ -1160,7 +1160,7 @@ routable
pipeline-parallel
Qwen2.5-14B

agent
[Aa]gent
requirement
PrefillDecode

Expand Down Expand Up @@ -1212,3 +1212,62 @@ per-deployment
opt-out
Framebuffer
Mooncake

# Inference field guide
DeepSeek
R1
o1
SRE
SLOs?
[Gg]oodput
FinOps
Ollama
benchmarked
dtypes?
CUDA
p50
p90
declaratively
[Ss]harding
agentic
subagents?
GPTQ
[Ee]vals?
[Qq]uantizing
SXM
datacenter
B200
RTX
safetensors
repos?
ServiceMonitor
teardown
torch\.compile
FlashAttention
PagedAttention
DistServe
Pyrra
GuideLLM
SemiAnalysis
AgentX
upskill
FP4
TGI
\d+(\.\d+)?x
vCPUs?
[Cc]onfigs
READMEs
runbooks?
Zeoli
Kiely
ByteByteGo
AIPerf
Viktor
Farcic
Vishakha
Sadhwani
Hamel
Husain
Shreya
Lamhot
Siagian
Loading