Repository navigation
[Feature]: Option to run the OpenEngine gRPC server on pygrpc-native #21102
Description
🚀 The feature, motivation and pitch
The OpenEngine frontend runs out of CPU before the GPU does. #19870 works around this with more frontend processes (2.31x tokens/s with four, at about 3x the CPU), and #19760 trimmed the per-token work in the adapter. Neither touches the gRPC server itself, and grpc.aio still handles every call's events in Python, on the same event loop as the servicers.
pygrpc-native is a drop-in replacement for grpc.aio.server(). It takes the same generated servicers and the same options, but does HTTP/2 and the per-call gRPC work in Rust, outside the GIL. I maintain it. LMCache already ships an optional gRPC server mode built on its lower-level API (LMCache/LMCache#6031), covered by LMCache's CI.
I'd like to add an opt-in --openengine-grpc-server pygrpc-native flag to trtllm-serve. The default stays grpcio, and the dependency goes in a new tensorrt_llm[openengine-native] extra, so nothing changes for anyone who doesn't ask for it. It works with multiple frontends as well, since pygrpc-native has honored grpc.so_reuseport since 0.5.0.
To see whether it helps, I ran the real trtllm-serve --grpc --grpc-protocol openengine path with only the engine mocked, and changed nothing but the line that creates the server. With one frontend on 4 cores:
| grpc.aio | pygrpc-native | |
|---|---|---|
| Frontend CPU per token, 40 req/s, 32 connections | 222 µs | 157 µs (-29%) |
| Frontend CPU per token, 40 req/s, 1 connection | 190 µs | 144 µs (-25%) |
| Tokens/s at saturation, 32 connections | 7,456 | 11,834 (1.60x) |
| Tokens/s at saturation, 1 connection | 8,320 | 11,885 (1.43x) |
The setup and the rest of the results are in the PR.
Alternatives
More frontend processes (#19870) also raise throughput, but they do it with more cores, and each client connection still lands on a single frontend. The two work together: every frontend can run on pygrpc-native.
Additional context
PR: #21103
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Activity