Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints

By Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, J. Ainslie, Yi Tay, Yi Tay, Mostafa Dehghani, Neil Houlsby. 2022

215 citations recorded by the source corpus.

Training large, deep neural networks to convergence can be prohibitively expensive. As a result, often only a small selection of popular, dense models are reused across different contexts and tasks. Increasingly, sparsely activated models, which seek to decouple model size from computation costs, are becoming an attractive alternative to dense models. Although more efficient in terms of quality and computation cost, sparse models remain data-hungry and costly to train from scratch in the large scale regime. In this work, we propose sparse upcycling -- a simple way to reuse sunk training costs by initializing a sparsely activated Mixture-of-Experts model from a dense checkpoint. We show that sparsely upcycled T5 Base, Large, and XL language models and Vision Transformer Base and Large models, respectively, significantly outperform their dense counterparts on SuperGLUE and ImageNet, using only ~50% of the initial dense pretraining sunk cost. The upcycled models also outperform sparse models trained from scratch on 100% of the initial dense pretraining computation budget.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023, methodology): “show that language model checkpoints with multihead attention (MHA) can be uptrained (Komatsuzaki et al., 2022) to use MQA with a small fraction of original training compute.”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Buildinguponthesparseupcycling technique[47],theSkywork-MoEmodel[65]leveragesthe foundationalarchitectureofitspre-developedSkywork-13B model[158],adoptingitsdensecheckpointsasafoundation forinitialstates.”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, unknown): “MiniCPM-MoE is initialized utilizing Sparse Upcycling (Komatsuzaki et al., 2022).”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, background): “In contrast, sparse Mixtures of Experts (MoE) (Jacobs et al., 1991; Eigen et al., 2013) effectively scale model capacity by using fixed activated parameters to process data, which has thrived in the field of NLP (Fedus et al., 2022; Zoph et al., 2022; Komatsuzaki et al., 2022).”
Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “Following Komatsuzaki et al. (2023), we transform every alternate MLP layer into an MoE layer with 8 experts and top-2 routing, starting from the second layer.”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “Other works such as BASE layers [73], hash layers [123] and sparse-upcycling [65] show that MoE-like conditional computation and expert modularity can be realized either by routing dense blocks or by converting pre-trained dense models into sparse expert collections.”