DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

By Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Duo Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase and 1 more. 2022

640 citations recorded by the source corpus.

The landscape of transformer model inference is increasingly diverse in model size, model characteristics, latency and throughput requirements, hardware requirements, etc. With such diversity, designing a versatile inference system is challenging. DeepSpeed-Inference addresses these challenges by (1) a multi-GPU inference solution to minimize latency while maximizing throughput for both dense and sparse transformers when the model fits in aggregate GPU memory, and (2) a heterogeneous inference solution that leverages CPU/NVMe/GPU memory to enable high-throughput inference for models larger than aggregate GPU memory. DeepSpeed-Inference reduces latency by 6.4× and increases throughput by 1.5 ×over the state-of-the-art. It enables trillion parameter scale inference under real-time latency constraints by leveraging hundreds of GPUs, an unprecedented scale for inference. It can inference 25 ×larger models than with GPU-only solutions, while delivering a high throughput of 84 TFLOPS (over 50% of A6000 peak).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Efficiently Scaling Transformer Inference (2022, positive): “DeepSpeed Inference (Aminabadi et al., 2022) further enables ZeRO ofﬂoad to use CPU and NVMe memory in addition to the GPU memory.”
Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “These systems utilize GPU kernel optimizations [1, 29, 31, 56], advanced batching mechanisms [14, 60], model parallelism [1, 41, 60], and parameter sharing [64] for efficient serving.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “…works particularly well when processing large batches of data, during training Pudipeddi et al. (2020); Ren et al. (2021) or large-batch non-interactive inference Aminabadi et al. (2022); Sheng et al. (2023), where each layer processes a lot of tokens each time the layer is loaded from RAM.”
ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, background): “Furthermore, model size reduction can lead to dramatic increases in inference speed by allowing a model to fit within a single GPU or node where it otherwise could not [36].”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “All existing systems (Aminabadi et al., 2022; Hug-gingFace, 2022) traverse the graph row-by-row, as shown in Fig.”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, background): “DS-MoE is part of a larger DeepSpeed-inference effort presented in [51].”
Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, neutral): “Extensive research efforts are dedicated to optimizing the memory footprint (Sheng et al., 2023) and operation subroutines (Aminabadi et al., 2022) of language models.”
Splitwise: Efficient Generative LLM Inference Using Phase Splitting (2023, methodology): “LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, methodology): “In distributed settings, we evaluate LP against TP (supported by deepspeed (Aminabadi et al., 2022)) and PP (supported by accelerate (Gugger et al., 2022)).”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “Similar observations were made by (Li et al., 2022).”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, methodology): “Specifically, we adapt the kernel from Deepspeed (DS) (Aminabadi et al., 2022) by adding the KV cache sparsity operation.”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “Performance experiments show our framework achieves 29 , 29 , 3 higher throughputs compared to three leading inference systems, DeepSpeed Zero-Inference (Aminabadi et al., 2022), Hugging Face Ac-celerate (HuggingFace), and FlexGen (Sheng et al., 2023) respectively.”
SGLang: Efficient Execution of Structured Language Model Programs (2023, neutral): “Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (2023, neutral): “Model ofﬂoading, which partitions the model between GPU and CPU at the Transformer layer level [3,14,37].”