A Review of Sparse Expert Models in Deep Learning

By William Fedus, Joseph Dean, Barret Zoph. 2022

209 citations recorded by the source corpus.

Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in deep learning. This class of architecture encompasses Mixture-of-Experts, Switch Transformers, Routing Networks, BASE layers, and others, all with the unifying idea that each example is acted on by a subset of the parameters. By doing so, the degree of sparsity decouples the parameter count from the compute per example allowing for extremely large, but efficient models. The resulting models have demonstrated significant improvements across diverse domains such as natural language processing, computer vision, and speech recognition. We review the concept of sparse expert models, provide a basic description of the common algorithms, contextualize the advances in the deep learning era, and conclude by highlighting areas for future work.

Subfields: Artificial Intelligence, Information Systems, Radiology, Nuclear Medicine and Imaging, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Richard Socher
2. Mohit Bansal
3. Luke S. Zettlemoyer
4. Caiming Xiong
5. Jianfeng Gao
6. Graham Neubig
7. Kyunghyun Cho
8. Percy Liang

In Information Systems

1. Jacob Devlin
2. Joseph Dean
3. Amir Salman Avestimehr
4. Suhas N. Diggavi
5. Ronald C. Taylor

In Model Architectures

1. Jonathan Heek
2. James Bradbury
3. Aakanksha Chowdhery
4. Shivani Agrawal
5. Jacob Devlin
6. Joseph Dean
7. Anselm Levskaya
8. Richard Socher

Citations in the corpus

• Mixtral of Experts (2024, neutral): “For a more in-depth overview, see [12].”
• Efficiently Scaling Transformer Inference (2022, positive): “Sparsity techniques, such as task-based mixture of expert architectures (Fedus et al., 2022; Kudugunta et al., 2021; Lepikhin et al., 2020; Shazeer et al., 2017), and adaptive computation techniques that allocate different amounts of compute per input and generation timestep (Jaszczur et al.”
• A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Theincorporation ofMoEarchitecturesintoLLMsisnowaprevalentpractice, allowingthesemodelstoachievesignificantparameterscale-ups andconsequentenhancementsincapabilities[25],[26],[32], [34],[65].”
• ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth (2023, unknown): “[10] for a review.”