Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT
By James Lee-Thorp, J. Ainslie. 2022
15 citations recorded by the source corpus.
We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model. Sparse Mixer slightly outperforms (<1%) BERT on GLUE and SuperGLUE, but more importantly trains 65% faster and runs inference 61% faster. We also present a faster variant, prosaically named Fast Sparse Mixer, that marginally underperforms BERT on SuperGLUE, but trains and runs nearly twice as fast. We justify the design of these two models by carefully ablating through various mixing mechanisms, MoE configurations and hyperparameters. Sparse Mixer overcomes many of the latency and stability concerns of MoE models and offers the prospect of serving sparse student models, without resorting to distilling them to dense variants.
Subfields: Large Language Models, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Large Language Models
- Joseph Dean
- William Fedus
- Barret Zoph
- Yi Tay
- Mostafa Dehghani
- Donald Metzler
- Luke S. Zettlemoyer
- Neil Houlsby
In Efficiency & Compression
- Joseph Dean
- Barret Zoph
- William Fedus
- Hinrich Schütze
- Timo Schick
- Mostafa Dehghani
- Yi Tay
- Luke S. Zettlemoyer
Citations in the corpus
- A Review of Sparse Expert Models in Deep Learning (2022, methodology): “Lee-Thorp and Ainslie (2022) introduces the Fast Sparse Mixer, which is an encoder only model that achieves 89% training and 98% inference speedups over BERT (Devlin et al., 2018).”