Splitwise: Efficient Generative LLM Inference Using Phase Splitting

By P. Patel, Esha Choukse, Chaojie Zhang, Í. Goiri, Aashaka Shah, Saeed Maleki, Ricardo Bianchini. International Symposium on Computer Architecture, 2023

759 citations recorded by the source corpus.

Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Unlike prompt computation, token generation does not need the compute capability of the latest GPUs and can be run with lower power and cost. Based on these insights, we propose Splitwise, a model deployment and scheduling technique that splits the two phases of LLM inference requests on to separate machines. Splitwise enables phase-specific resource management using hardware that is well suited for each phase. Request state is transferred efficiently between machines using optimized network libraries on the fast back-plane interconnects available in today’s GPU clusters. Using Splitwise, we design homogeneous and heterogeneous LLM inference clusters optimized for throughput, cost, and power Compared to current designs, Splitwise clusters achieve up to $1.4 \times$ higher throughput at $\mathbf{2 0 \%}$ lower cost. Alternatively, they can deliver $2.35 \times$ more throughput under the same power and cost budgets.

Subfields: Artificial Intelligence, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/ad9146d98ae95bbeeef460abe083ecc2c4798672

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xin Jin
2. Yinmin Zhong
3. Xuanzhe Liu
4. Junda Chen
5. Shengyu Liu
6. Jianbo Hu
7. Yibo Zhu
8. Hao Zhang

Citations in the corpus

• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “From a TBT minimization point of view, a smaller token bud-get is preferable because iterations with fewer prefill tokens have lower latency.”
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, methodology): “In this case, techniques such as chunked-prefill with piggyback [3,9] may be preferred since it can fill each batch to the compute-bound threshold, thereby maintaining higher GPU utilization in every iteration.”
• Atom: Low-bit Quantization for Efficient and Accurate LLM Serving (2023, background): “However, asymmetric weight-activation quantization can lead to additional calculations during matrix multiplication since: where three additional cross-terms will be calculated.”