LightSeq: A High Performance Inference Library for Transformers

By Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, Lei Li. 2021

72 citations recorded by the source corpus.

Transformer and its variants have achieved great success in natural language processing. Since Transformer models are huge in size, serving these models is a challenge for real industrial applications. In this paper, we propose , a highly efficient inference library for models in the Transformer family. includes a series of GPU optimization techniques to both streamline the computation of Transformer layers and reduce memory footprint. supports models trained using PyTorch and Tensorflow. Experimental results on standard machine translation benchmarks show that achieves up to 14x speedup compared with TensorFlow and 1.4x speedup compared with , a concurrent CUDA implementation. The code will be released publicly after the review.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In ML Systems & Hardware

Citations in the corpus

Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, neutral): “These systems utilize GPU kernel optimizations [1, 29, 31, 56], advanced batching mechanisms [14, 60], model parallelism [1, 41, 60], and parameter sharing [64] for efficient serving.”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “…have witnessed the emergence of systems specialized for LLM inference, such as FasterTrans-former (NVIDIA, 2022), Orca (Yu et al., 2022), Light-Seq (Wang et al., 2021), PaLM inference (Pope et al., 2022), TurboTransformers (Fang et al., 2021), DeepSpeed Inference (Aminabadi et al., 2022), and…”
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (2024, methodology): “More recently, systems such as Orca [75], vLLM [53], FlexGen [63], FasterTransformers [7], LightSeq [70], and TurboTransformers [42] propose domain-specific optimizations for transformer inference.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “LightSeq [282] Fast Inference High performance training and inference library for sequence processing and generation.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, unknown): “FasterTransformer [4], LightSeq [61], TurboTransformers [22] and EET [36] are such examples.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Besides, some of libraries like FastSeq [196], DeepSpeed [156], and LightSeq [187] are useful to increase the inference speed of models.”