TurboTransformers: an efficient GPU serving system for transformer models

By Jiarui Fang, Yang Yu, Chen-liang Zhao, Jie Zhou. 2020

180 citations recorded by the source corpus.

The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years. Unlike the Recurrent Neural Network (RNN) models, transformers are able to process on dimensions of sequence lengths in parallel, therefore leads to better accuracy on long sequences. However, efficient deployments of them for online services in data centers equipped with GPUs are not easy. First, more computation introduced by transformer structures makes it more challenging to meet the latency and throughput constraints of serving. Second, NLP tasks take in sentences of variable length. The variability of input dimensions brings a severe problem to efficient memory management and serving optimization. To solve the above challenges, this paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework. Three innovative features make it stand out from other similar works. An efficient parallel algorithm is proposed for GPU-based batch reduction operations, like Softmax and LayerNorm, which are major hot spots besides BLAS routines. A memory allocation algorithm, which better balances the memory footprint and allocation/free efficiency, is designed for variable-length input situations. A serving framework equipped with a new batch scheduler using dynamic programming achieves the optimal throughput on variable-length requests. The system can achieve the state-of-the-art transformer model serving performance on GPU platforms and can be seamlessly integrated into your PyTorch code with a few lines of code.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In ML Systems & Hardware

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “These systems utilize GPU kernel optimizations [1, 29, 31, 56], advanced batching mechanisms [14, 60], model parallelism [1, 41, 60], and parameter sharing [64] for efficient serving.”
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale (2022, methodology): “TurboTransformers [26] similarly fuses elementwise and reduction operators for transformer inference.”
Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management (2021, background): “the theoretical estimations is that activation tensors whose life cycles do not overlap may reuse the same memory space [29], [30].”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “TurboTransformers [285] Fast Inference A library open source by WeChat AI to get fast inference using transformer models.”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “, 2022), TurboTransformers (Fang et al., 2021), DeepSpeed Inference (Aminabadi et al.”
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (2023, unknown): “LLM-Speciﬁc Serving Optimizations: The prominence of Transformers has led to specialized serving systems [9, 36, 53].”
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024, methodology): “They range from general-purpose production-grade systems like TorchServe [7] and NVIDIA Triton [19] to systems optimized specifically for Transformer-based LLMs [9, 18, 21, 33, 50, 53, 54, 60].”
LightSeq: A High Performance Inference Library for Transformers (2021, methodology): “Then, we make a fair comparison with TensorFlow, PyTorch, FasterTransformer, and TurboTransformers on machine translation and text generation to show the efﬁciency of LightSeq.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, unknown): “FasterTransformer [4], LightSeq [61], TurboTransformers [22] and EET [36] are such examples.”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “More recently, systems such as Orca [75], vLLM [53], FlexGen [63], FasterTransformers [7], LightSeq [70], and TurboTransformers [42] propose domain-specific optimizations for transformer inference.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “Given the rapid development of LLM, there is an emergence of systems that are specialized for LLM inference, such as Faster Transformer (NVIDIA), Orca (Yu et al., 2022), LightSeq (Wang et al., 2021), PaLM inference (Pope et al., 2022), TurboTransformers (Fang et al., 2021), and Deepspeed-Inference (”