[Blog] Pushing the Limits of Serving DeepSeek-V4-Pro

[TianyuZhang1214]

Summary

This PR adds a new blog post, “Pushing the Limits of Serving DeepSeek-V4-Pro,” describing how we systematically optimize DeepSeek-V4-Pro serving on compute-constrained H20 GPUs.

Rather than forcing every workload into one universal configuration, the post develops scenario-specific profiles for different context lengths, latency targets, throughput requirements, and KV-cache capacity constraints.

What This Post Covers

• Hardware and capacity design. We map Prefill to H20-96GB and Decode to H20-141GB based on their different serving-state requirements, then expand capacity with Humming MXFP4AFP8 and Online C128.
• Prefill optimization. PP2 and PP4 cover different context-length ranges, supported by MoE-TP, symmetric-memory collectives, fused communication and computation, and production-shape Humming tuning.
• Low-latency Decode. PP2-TP8 combines DSpark with pipeline-aware speculative execution, while single-node TP8 establishes the latency reference.
• High-throughput Decode. DP/EP scaling, refine-path optimization, expert placement, Humming fusion, and communication–computation overlap improve performance across concurrency levels.
• Profile trade-offs. The evaluation connects latency, throughput, context length, concurrency, and KV-cache capacity to concrete serving-profile choices.

Key Results

• BS1 Decode reaches 271 tokens/s on H20-141GB, substantially narrowing the observed serving-performance gap with the public B300 result of 383.7 tokens/s.
• Optimized Prefill reaches up to 8.45k input tokens/s per node.
• The long-context PP4 profile processes a 1M-token prompt in 43.7 seconds.
• The DP16-EP16 high-throughput efficiency reference reaches 4.67k output tokens/s per node (27.4 ms TPOT).
• Humming MXFP4AFP8 and Online C128 jointly expand full-token capacity to 3.88× the FP8 baseline for DP32-EP32 and 10.14× for PP2-TP8.

Takeaway

The contribution is not a single benchmark or configuration. It is a reusable methodology in which workload characteristics, SLOs, and measured hardware behavior jointly determine the serving topology and execution path, providing a practical reference for serving frontier models under compute, memory, bandwidth, and interconnect constraints.

[ispobock]

could you specify the benchmark settings for the numbers?

[TianyuZhang1214]

Thanks! We kept the introduction concise and added the full benchmark setup and log-based throughput methodology in Appendix B.3.

[ispobock]

how about the accuracy loss?

[TianyuZhang1214]

Thanks! We added our 95.5% GSM8K1000 result in Appendix D.2, together with the public Humming/SGLang accuracy results for reference.

[ispobock]

gsm8k is not very stong to detect long output accuracy issues. Maybe you can try AIME and GPQA with repeat later. It's not a blocker for this blog.

[ispobock]

the font size is too small in this image

[TianyuZhang1214]

Thanks! We increased the text size throughout Figure 2 to improve readability at the rendered width and on smaller screens.