Taming Sparsely Activated Transformer with Stochastic Experts

By Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, Jianfeng Gao. 2021

152 citations recorded by the source corpus.

Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. However, SAMs are reported to be parameter inefficient such that larger models do not always lead to better performance. While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts. In this paper, we propose a new expert-based model, THOR (Transformer witH StOchastic ExpeRts). Unlike classic expert-based models, such as the Switch Transformer, experts in THOR are randomly activated for each input during training and inference. THOR models are trained using a consistency regularized loss, where experts learn not only from training data but also from other experts as teachers, such that all the experts make consistent predictions. We validate the effectiveness of THOR on machine translation tasks. Results show that THOR models are more parameter efficient in that they significantly outperform the Transformer and MoE models across various settings. For example, in multilingual translation, THOR outperforms the Switch Transformer by 2 BLEU scores, and obtains the same BLEU score as that of a state-of-the-art MoE model that is 18 times larger. Our code is publicly available at: https://github.com/microsoft/Stochastic-Mixture-of-Experts.

Subfields: Artificial Intelligence, Computer Science Applications, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

A Review of Sparse Expert Models in Deep Learning (2022, methodology): “Zuo et al. (2021) introduced THOR, an algorithm which randomly selects two experts for each input during training and inference and found improvements of 2 BLEU points over standard MoE models.”
Mixture-of-Experts with Expert Choice Routing (2022, background): “THOR [37] randomly activates experts during training and inference and is trained with a consistency regularization loss.”
A Survey on Dynamic Neural Networks for Natural Language Processing (2022, background): “THOR (Zuo et al., 2022) is a special form of MoE layer, which completely discards the conditional routing mechanism and instead optimizes the consistency between a randomly selected pair of experts.”
A Survey on Mixture of Experts in Large Language Models (2024, unknown): “Zuo et al. [81] introduces THOR, an algorithm that randomly allocates two experts to each input during training and inference with a consistency regularized loss promoting consistent predictions.”
Designing Effective Sparse Expert Models (2022, methodology): “However, this leads to an interesting question of how architectures that eliminate learned routing Roller et al. (2021); Zuo et al. (2021) appear to perform well.”
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale (2022, methodology): “Following by this, more recent work [26, 27, 28] exhibit the sparsity advantage of MoE.”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “However, this leads to an interesting question of how architectures that eliminate learned routing Roller et al. (2021); Zuo et al. (2021) appear to perform well.”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “Other approaches, such as stochastic experts and locality-sensitive hashing-based routing, treat routing as a stochastic process, using randomness to regularize the model and to reduce communication overhead in large distributed systems [108, 161].”