ST-MoE: Designing Stable and Transferable Sparse Expert Models

By Barret Zoph, Irwan Bello, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, J. Dean, Joseph Dean, Noam Shazeer, William Fedus. 2022

423 citations recorded by the source corpus.

Scale has opened new frontiers in natural language processing -- but at a high cost. In response, Mixture-of-Experts (MoE) and Switch Transformers have been proposed as an energy efficient path to even larger and more capable language models. But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning. Our work focuses on these issues and acts as a design guide. We conclude by scaling a sparse model to 269B parameters, with a computational cost comparable to a 32B dense encoder-decoder Transformer (Stable and Transferable Mixture-of-Experts or ST-MoE-32B). For the first time, a sparse model achieves state-of-the-art performance in transfer learning, across a diverse set of tasks including reasoning (SuperGLUE, ARC Easy, ARC Challenge), summarization (XSum, CNN-DM), closed book question answering (WebQA, Natural Questions), and adversarially constructed tasks (Winogrande, ANLI R3).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1bc9865ebf52b59abac7f5ee4456ff2ac37fcff3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Jamba: A Hybrid Transformer-Mamba Language Model (2024, methodology): “These choices were inspired by prior work on MoE [7, 49] and verified in preliminary experiments.”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “, 2019), ST-MoE (Zoph et al., 2022)) have also been very popular as the goto model of choice, especially in smaller computational regimes (e.”
Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (2022, methodology): “Sparsely activated mixture of expert (MoE) models have recently been used with great effect to scale up both vision [1, 2] and text models [3, 4].”
A Survey on Mixture of Experts in Large Language Models (2024, positive): “…[26], [33], [34], [65], [66], [67] have attempted to address this challenge by incorporating auxiliary loss functions to encourage even token distribution across experts, these solutions can still lead to training instability [35] and often neglect the relative importance of different tokens [85].”
Optimizing Mixture of Experts using Dynamic Recompilations (2022, background): “ST-MoE is currently leading most NLP benchmarks (Zoph et al., 2022) and V-MoE is matching the state-of-the-art predictive performance on image tasks with as little as half of the compute at inference time (Riquelme et al., 2021).”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Encoder-decoder LLMs (e.g., T0 [55], ST-MoE [56], and GLM-130B [57]) are able to directly resolve tasks that generate sentences based on some context, such as summariaztion, translation, and question answering [58].”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “Later work such as ST-MoE studies the stability of such routers, proposes auxiliary objectives such as the z -loss, and analyzes how routing schemes and noise influence expert utilization and transfer performance [13, 160].”
Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025, unknown): “Training MoE models effectively is known to be challenging due to stability issues and difficulties in enforcing balanced expert exploration (Zoph et al., 2022; Cai et al., 2025; Lepikhin et al., 2020).”