DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

By Samyam Rajbhandari, Conglong Li, Zi-Mei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, Yuxiong He. 2022

591 citations recorded by the source corpus.

As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model. Its training cost saving is demonstrated from encoder-decoder models (prior works) to a 5x saving for auto-aggressive language models (this work along with parallel explorations). However, due to the much larger model size and unique architecture, how to provide fast MoE model inference remains challenging and unsolved, limiting its practical usage. To tackle this, we present DeepSpeed-MoE, an end-to-end MoE training and inference solution as part of the DeepSpeed library, including novel MoE architecture designs and model compression techniques that reduce MoE model size by up to 3.7x, and a highly optimized inference system that provides 7.3x better latency and cost compared to existing MoE inference solutions. DeepSpeed-MoE offers an unprecedented scale and efficiency to serve massive MoE models with up to 4.5x faster and 9x cheaper inference compared to quality-equivalent dense models. We hope our innovations and systems help open a promising path to new directions in the large model landscape, a shift from dense to sparse MoE models, where training and deploying higher-quality models with fewer resources becomes more widely possible.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Unified Scaling Laws for Routed Language Models (2022, methodology): “We note there are several libraries available for supporting large-scale Routing, including DeepSpeed [Kim et al., 2021, Rajbhandari et al., 2022] and GSPMD [Xu et al., 2021].”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “Rajbhandari et al. (2022) designs PR-8 MoE which uses top-2 routing, but each token is sent to a shared dense layer and a single expert of its choosing (instead of two experts).”
DeepSpeed- Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale (2022, methodology): “The classic all-to-all communication primitive has been used to implement expert parallelism [23], [15], [22].”
Tutel: Adaptive Mixture-of-Experts at Scale (2022, background): “While GShard (Lepikhin et al., 2021) provides a computation logic that ensures algorithmic correctness of MoE, several popular MoE frameworks (Ott et al., 2019; Rajbhandari et al., 2022) follow the same logic but perform poorly on a large scale.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, background): “AI21 Labs [37], DeepMind [26,48], Google [15,21,63], Meta Platforms [10,67], Microsoft [50], Microsoft & NVIDIA [59], and OpenAI [12] have reported that every language processing task, including translation [11, 17], classification [20, 53], question-answering [32, 33, 40] and more, can be cast as a”
Qwen2 Technical Report (2024, background): “Recently, there has been a notable trend towards integrating both shared and routing-specific experts within MoE layers (Rajbhandari et al., 2022; Dai et al., 2024).”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “In practice, most MoE models features alternate FFN replacement (1/2) with MoE layers [25], [33], [64], [90].”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “At the systems level, DeepSpeed-MoE [118] provides an end-to-end framework for training and serving very large MoE models efficiently, and recent surveys on MoE in large language models synthesize developments in routing, optimization and deployment [5].”