Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

By Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, J. Dean, Joseph Dean. 2017

5,113 citations recorded by the source corpus.

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Yoshua Bengio
  5. Quoc V. Le
  6. Percy Liang
  7. Graham Neubig
  8. Kyunghyun Cho

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Yi Yang

In Model Architectures

  1. Quoc V. Le
  2. Yoshua Bengio
  3. Jianfeng Gao
  4. Ruslan Salakhutdinov
  5. Mohit Bansal
  6. Kyunghyun Cho
  7. Richard Socher
  8. Tie-Yan Liu

In Efficiency & Compression

  1. Luke S. Zettlemoyer
  2. Quoc V. Le
  3. Alexander M. Rush
  4. Dan Alistarh
  5. Barret Zoph
  6. Yi Tay
  7. Joseph Dean
  8. Tim Dettmers

Citations in the corpus