Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
By Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, J. Dean, Joseph Dean. 2017
5,113 citations recorded by the source corpus.
The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Graham Neubig
- Kyunghyun Cho
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Jianfeng Gao
- Ruslan Salakhutdinov
- Mohit Bansal
- Kyunghyun Cho
- Richard Socher
- Tie-Yan Liu
In Efficiency & Compression
- Luke S. Zettlemoyer
- Quoc V. Le
- Alexander M. Rush
- Dan Alistarh
- Barret Zoph
- Yi Tay
- Joseph Dean
- Tim Dettmers
Citations in the corpus
- Attention is All you Need (2017, background): “Recent work has achieved significant improvements in computational efficiency through factorization tricks [18] and conditional computation [26], while also improving model performance in case of the latter.”
- Factorization tricks for LSTM networks (2017, methodology): “LSTM networks (Hochreiter & Schmidhuber, 1997) have been successfully used in language modeling (Jozefowicz et al., 2016; Shazeer et al., 2017), speech recognition (Xiong et al., 2016), machine translation (Wu et al., 2016), and many other tasks.”
- Character-Level Language Modeling with Deeper Self-Attention (2018, background): “A Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al. 2017) allowed language modeling experiments with a greatly increased number of parameters by only accessing a small portion of parameters every time step, showing a reduction in bits per word.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “The “bitter lesson” of machine learning research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise [Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et ”
- PaLM: Scaling Language Modeling with Pathways (2022, background): “Model sparsity like Mixture-of-Experts allows for scaling model sizes by allowing different examples to use different subsets of parameters (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2021; Du et al., 2021; Zoph et al., 2022).”
- Mixtral of Experts (2024, methodology): “Moreover, the MoE layer can be distributed to multiple GPUs through standard Model Parallelism techniques, and through a particular kind of partitioning strategy called Expert Parallelism (EP) [28].”
- Generating Wikipedia by Summarizing Long Sequences (2018, background): “We also added in some experiments one mixture of experts (MoE) layer (Shazeer et al., 2017) to increase the network’s capacity.”
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Finally, while scaling up existing approaches is one way to improve model quality, some approaches or architectures are more efficient to train (Shazeer et al., 2017; Vaswani et al., 2017; Wu et al., 2019), more sample efficient, or faster at inference (Gu et al.”
- Mesh-TensorFlow: Deep Learning for Supercomputers (2018, methodology): “MPI-alltoall is used in the case where different dimensions in the input and the output are split across the same mesh dimension, as might be the case when switching between data-parallelism and model-parallelism for different layers of the same model, as in [15].”
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost (2018, methodology): “Recent work has shown that for problems where vast quantities of data are available, e.g. language modeling and machine translation, task performance improves consistently as model size increases, even in the regime of models with several billions of parameters (Shazeer et al., 2017).”
- Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “There has been previous work (Eigen et al., 2013; Shazeer et al., 2017) proposing architectures that can be categorized as instantiations of MoC, since the mixture structure is employed in the feature space.”
- One Epoch Is All You Need (2019, background): “Some promising ideas that tend to overfit include mixture-of-experts (Shazeer et al., 2017) and optimization methods exploiting the second order information such as K-FAC (Osawa et al., 2018).”
- Beyond human-level accuracy: computational challenges in deep learning (2019, background): “Google has also stated they want to train language models on a trillion word dataset [32].”
- Unified Scaling Laws for Routed Language Models (2022, methodology): “Throughout this work we have focused on a narrow subset of possible Routing Net architectures, which we believe are representative of recent work on large scale Routing Nets (Roller et al., 2021; Fedus et al., 2021; Lewis et al., 2021; Shazeer et al., 2017; Artetxe et al., 2021; Lepikhin et al., 202”
- Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Some prior works have chosen to use the mixture-of-experts (MoE) [32, 35, 61] technique to scale to larger model sizes more economically, producing large-scale models that selectively use a subset of its parameters in each forward pass.”