Designing Effective Sparse Expert Models

By Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Joseph Dean, Noam Shazeer, William Fedus. 2022

118 citations recorded by the source corpus.

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3).

Subfields: Artificial Intelligence, General Health Professions, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Health Professions

In Model Architectures

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “These include both dense transformer models [6, 30, 48, 38, 52] and mixture-of-expert (MoE) models [11, 12, 60].”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “, 2020) and ST-MoE-32B (Zoph et al., 2022) and show that PaLM obtains competitive close-to-SOTA performance.”
Emergent Abilities of Large Language Models (2022, positive): “One direction is using sparse mixture-of-experts architectures (Lepikhin et al., 2021; Fedus et al., 2021; Artetxe et al., 2021; Zoph et al., 2022), which scale up the number of parameters in a model while maintaining constant computational costs for an input.”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “Further, state-of-the-art results on many benchmarks are currently held by sparse expert models such as ST-MoE (Zoph et al., 2022).”
Efficient Transformers: A Survey (2020, methodology): “We lightly revised it again in Mar by adding newer SOTA sparse models such as ST-MoE-32B (Zoph et al., 2022).”
Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints (2022, background): “Recently, there has been a growing number of MoE works achieving state-of-the-art quality and remarkable efficiency gains on both language and vision tasks (Lepikhin et al., 2021; Fedus et al., 2022; Riquelme et al., 2021; Artetxe et al., 2021; Du et al., 2021; Zoph et al., 2022; Mustafa et al., 202”
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024, unknown): “An MoE layer is composed of multiple experts, where each expert is structurally identical to a standard FFN.”
Unifying Language Learning Paradigms (2022, methodology): “Firstly, ST-MoE-32B has 200B+ parameters and is costs equivalent to a 32B dense model.”
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022, methodology): “Recent work, such as (Zoph et al., 2022), has played out the promise of MoE models by achieving SOTA results on a number of NLP benchmarks.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, background): “We believe it’s due to a phenomena mentioned in (Zoph et al., 2022) that MoE specializes in multilingual settings but not by language.”
On the Representation Collapse of Sparse Mixture of Experts (2022, unknown): “In addition to the supervised learning scenario, there has been work on exploring SMoE under the pre-training-ﬁne-tuning paradigm, and observing discrepancies between strong pre-training quality and poor ﬁne-tuning performance (Fedus et al., 2021; Artetxe et al., 2021; Zoph et al., 2022).”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “Experiments find that 100× smaller LMs augmented with Mind’s Eye can achieve similar reasoning capabilities as vanilla large models, and its promptingbased nature avoids the instability issues of training mixture-of-expert models (Zoph et al., 2022).”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, unknown): “: U NIFIED -IO achieves respectable results on three NLP tasks but lags behind SOTA models (Smith et al., 2022; Zoph et al., 2022; He et al., 2021).”