Efficient Large Scale Language Modeling with Mixtures of Experts

By Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srini Iyer, Ramakanth Pasunuru and 14 more. 2021

248 citations recorded by the source corpus.

Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using $\sim$4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “2 Trillion parameter GLaM model [11], and others [1, 60] are able to provide a large effective model size despite using relatively fewer training and inference FLOPs.”
Unified Scaling Laws for Routed Language Models (2022, background): “The underlying theory is less understood, but there is some exploration of this space including Hutter [2021] and Bahri et al. [2021]. These studies, and ours, are mutually reliant on a large corpus of work improving the scalability of Transformers. This includes models like GPT-2 [Radford et al., 2”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Following OPT, we use Fully Sharded Data Parallel (Artetxe et al., 2021) and the Megatron-LM Tensor Parallelism (Shoeybi et al.”
Emergent Abilities of Large Language Models (2022, positive): “One direction is using sparse mixture-of-experts architectures (Lepikhin et al., 2021; Fedus et al., 2021; Artetxe et al., 2021), which scale up the number of parameters in a model while maintaining constant computational costs for an input.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “We trained OPT-175B on 992 80GB A100 GPUs, by utilizing Fully Sharded Data Parallel (Artetxe et al., 2021) with Megatron-LM Tensor Parallelism (Shoeybi et al., 2019).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “Whilst the performance of the released 20B parameter model is impressive in many respects, outperforming our previous best performing model, GPT-J-6B, on most benchmarked datasets, it is clear from comparing to Fairseq’s 13B parameter model [Artetxe et al., 2021], and extrapolating based on OpenAI’s”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, background): “7B models described in [8]; the Fairseq architecture description can be found in Table 8 in the appendix.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We compare with the following models: the GPT-3 series (Brown et al., 2020), the FairSeq series (Artetxe et al., 2021), the GPT-Neo(X)/J models (Black et al., 2021; Wang & Komatsuzaki, 2021; Black et al., 2022), the OPT series (Zhang et al., 2022), the BigScience Architecture and Scaling Pile model…”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “The release of spare expert model checkpoints by Artetxe et al. (2021) and Fedus et al. (2021) allows broader groups to analyze and explain these dynamics.”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…a surge of research and development in autoregressive language models (Radford et al., 2019; Lewis et al., 2019; Raffel et al., 2019; Brown et al., 2020; Rae et al., 2021; Artetxe et al., 2021; Smith et al., 2022), with ever increasing size and natural language understanding (NLU) capabilities.”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “A.1 MODEL ARCHITECTURE For model architecture we use the same exact architecture for CM3-Medium and CM3-Large as the dense 2.7B and 13B models described in Artetxe et al. (2021).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “To the best of our knowledge, this paper is the ﬁrst that provides an empirical analysis that investigates why in-context learning achieves performance gains over zero-shot inference.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Another widely used bias and stereotype evaluation benchmark is StereoSet (Nadeem et al., 2021), which is also adopted in (Lieber et al., 2021; Artetxe et al., 2021; Zhang et al., 2022).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “For more information on how these pretrained models are trained, see Artetxe et al. (2021).”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, background): “AI21 Labs [37], DeepMind [26,48], Google [15,21,63], Meta Platforms [10,67], Microsoft [50], Microsoft & NVIDIA [59], and OpenAI [12] have reported that every language processing task, including translation [11, 17], classification [20, 53], question-answering [32, 33, 40] and more, can be cast as a”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, background): “Recently, there has been a growing number of MoE works achieving state-of-the-art quality and remarkable efficiency gains on both language and vision tasks (Lepikhin et al., 2021; Fedus et al., 2022; Riquelme et al., 2021; Artetxe et al., 2021; Du et al., 2021; Zoph et al., 2022; Mustafa et al., 202”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “Model-parallel, fully sharded data-parallel, and other distributed training strategies (Artetxe et al., 2021) confer different scaling patterns that may change the conclusions that we report in this work.”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, methodology): “While recent studies like [29] and [30] discuss reduction of FLOPs, it is pertinent to mention that unlike training, inference latency and cost do not depend on computation alone.”