SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts

By Zhao You, Shulin Feng, Dan Su, Dong Yu. 2021

81 citations recorded by the source corpus.

Recently, Mixture of Experts (MoE) based Transformer has shown promising results in many domains. This is largely due to the following advantages of this architecture: firstly, MoE based Transformer can increase model capacity without computational cost increasing both at training and inference time. Besides, MoE based Transformer is a dynamic network which can adapt to the varying complexity of input instances in realworld applications. In this work, we explore the MoE based model for speech recognition, named SpeechMoE. To further control the sparsity of router activation and improve the diversity of gate values, we propose a sparsity L1 loss and a mean importance loss respectively. In addition, a new router architecture is used in SpeechMoE which can simultaneously utilize the information from a shared embedding network and the hierarchical representation of different MoE layers. Experimental results show that SpeechMoE can achieve lower character error rate (CER) with comparable computation cost than traditional static networks, providing 7.0%-23.0% relative CER improvements on four evaluation datasets.

Subfields: Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

1. Joseph Dean
2. Zhao You
3. Dong Yu
4. Dan Su
5. Noam Shazeer
6. Yanping Huang
7. Barret Zoph
8. Jinyu Li

Citations in the corpus

• A Review of Sparse Expert Models in Deep Learning (2022, background): “And while most work has been in natural language processing, they have also been successfully used in a variety of domains including computer vision (Puigcerver et al., 2020), speech recognition (You et al., 2021) and multi-modal learning (Mustafa et al., 2022).”
• Speechmoe2: Mixture-of-Experts Model with Improved Routing (2021, methodology): “With this property, they are more adaptable to various inputs, and can also scale up to very large models to better perform while keeping computation cost constant. our previous work investigated one type of dynamic model in speech recognition, named SpeechMoE [10].”
• ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “Additionally, Automatic Speech Recognition has been improved by the SpeechMoE variants (You et al., 2021a;b).”
• Designing Effective Sparse Expert Models (2022, methodology): “Additionally, Automatic Speech Recognition has been improved by the SpeechMoE variants (You et al., 2021a;b).”