Deep Mixture of Experts via Shallow Embedding

By Xin Wang, Fisher Yu, Lisa Dunlap, Yian Ma, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, Joseph E. Gonzalez. 2018

154 citations recorded by the source corpus.

Subfields: Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Lu Yuan
2. Xiyang Dai
3. Dongdong Chen
4. Shiji Song
5. Yinpeng Chen
6. Mengchen Liu
7. Gao Huang
8. Jianfeng Gao

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Le Yang
5. Yizeng Han
6. Honghui Wang
7. Lihi Zelnik-Manor
8. Dong Yu

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “Notably, it has been proven that a dynamic network with an adaptive width can preserve the representation power of an unsparsifiedmodel [79].”
• XNAS: Neural Architecture Search with Expert Advice (2019, methodology): “Following works build upon this idea and include a gating mechanism per mixture [51, 41], and some further suggest sparsity regularization over experts via the gating mechanism [37, 44].”
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts (2021, methodology): “Recently, deep mixture of experts (MoE) based approaches [14, 15] have been intensively investigated and applied in different tasks such as language modeling [16, 17] and image classification [18, 19, 20, 21].”
• Residual Mixture of Experts (2022, methodology): “MoE has been widely used in Natural Language Processing [18, 22, 35, 44] and computer vision [1, 16, 19, 41, 53, 58].”
• A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Snowflake Arctic [29] employs a similar design, utilizing a Dense-MoE hybrid transformer architecture to effectively overlap communication with computation.”
• Scaling Vision with Sparse Mixture of Experts (2021, methodology): “For computer vision, previous work on MoEs [21, 2, 25, 1, 63, 47, 64] focused on architectures whose scale is considerably smaller than that of both language models and our model.”
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, methodology): “As a result, it has attracted increasing attention in recent years in both NLP (Venugopalan et al. 2016; Wang et al. 2018a) and computer vision communities (Krishna et al. 2017).”
• Multi-site fMRI Analysis Using Privacy-preserving Federated Learning and Domain Adaptation: ABIDE Results (2020, background): “1) Mixture of Experts (MoE) domain adaptation: Mixture of Experts (MoE) [37], [38], [39] is an approach to conditionally combine experts to process each input.”
• Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “Switch Transformers [29] and GLaM [28] further refined gating and load-balancing schemes to train language models with hundreds of billions to over a trillion parameters at practical cost, while MoE via shallow embedding explored channel-wise MoE routing within CNNs [142].”