Speechmoe2: Mixture-of-Experts Model with Improved Routing

By Zhao You, Shulin Feng, Dan Su, Dong Yu. 2021

41 citations recorded by the source corpus.

Mixture-of-experts based acoustic models with dynamic routing mechanisms have proved promising results for speech recognition. The design principle of router architecture is important for the large model capacity and high computational efficiency. Our previous work SpeechMoE only uses local grapheme embedding to help routers to make route decisions. To further improve speech recognition performance against varying domains and accents, we propose a new router architecture which integrates additional global domain and accent embedding into router input to promote adaptability. Experimental results show that the proposed SpeechMoE2 can achieve lower character error rate (CER) with comparable parameters than SpeechMoE on both multi-domain and multi-accent task. Primarily, the proposed method provides up to 1.6% ∼ 4.8% relative CER improvement for the multi-domain task and 1.9% ∼ 17.7% relative CER improvement for the multi-accent task respectively. Besides, increasing the number of experts also achieves consistent performance improvement and keeps the computational cost constant.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Irwan Bello
2. Quoc V. Le
3. Barret Zoph
4. Joseph Dean
5. William Fedus
6. Noam Shazeer
7. Yanping Huang
8. Nan Du

Citations in the corpus

• A Review of Sparse Expert Models in Deep Learning (2022, background): “SpeechMoE2 (You et al., 2022) further improves on the SpeechMoE’s results by making a new routing algorithm that adds in new auxiliary information for making routing decisions.”
• Designing Effective Sparse Expert Models (2022, methodology): “Additionally, Automatic Speech Recognition has been improved by the SpeechMoE variants (You et al., 2021a;b).”
• ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “Additionally, Automatic Speech Recognition has been improved by the SpeechMoE variants (You et al., 2021a;b).”