Dense-to-Sparse Gate for Mixture-of-Experts

By Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma, Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang, Bin Cui. 2021

36 citations recorded by the source corpus.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Joseph Dean
2. William Fedus
3. Barret Zoph
4. Julian McAuley
5. Canwen Xu
6. Quoc V. Le
7. Frank Hutter
8. Song Han

Citations in the corpus

• A Review of Sparse Expert Models in Deep Learning (2022, background): “Nie et al. (2021) begins routing as a soft gating function where all experts are trained (e.g. a dense model) and anneals down to the standard top-1 routing algorithm.”
• A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “, 2021) Transformer (M6) k top-1 expert capacity DTS (Nie et al., 2021) Transformer (GPT) dynamic sparsity scheduler”