Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition

By Ken'ichi Kumatani, R. Gmyr, Andres Felipe Cruz Salinas, Linquan Liu, Wei Zuo, Dev Patel, Eric Sun, Yu Shi. 2021

23 citations recorded by the source corpus.

The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up with a simple routing algorithm in order to achieve better accuracy. More specifically, we apply the sparsely-gated MoE technique to two types of networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer (T-T). We demonstrate through a set of ASR experiments on multiple language data that the MoE networks can reduce the relative word error rates by 16.3% and 4.6% with the S2S-T and T-T, respectively. Moreover, we thoroughly investigate the effect of the MoE on the T-T architecture in various conditions: streaming mode, non-streaming mode, the use of language ID and the label decoder with the MoE.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Quoc V. Le
3. Li Dong
4. Mohit Bansal
5. Jianfeng Gao
6. Luke S. Zettlemoyer
7. Barret Zoph
8. Graham Neubig

In Electrical and Electronic Engineering

1. Furu Wei
2. Li Dong
3. Noam Shazeer
4. Barret Zoph
5. William Fedus
6. Irwan Bello

In Signal Processing

1. Furu Wei
2. Noam Shazeer
3. Shaohan Huang
4. Li Dong
5. Shuming Ma
6. Barret Zoph
7. William Fedus
8. Irwan Bello

Citations in the corpus

• Designing Effective Sparse Expert Models (2022, methodology): “Kumatani et al. (2021) reduced word error rates using MoE models in Sequence-to-Sequence Transformer and Transformer Transducer.”
• On the Representation Collapse of Sparse Mixture of Experts (2022, methodology): “To explore the potential of SMoE, recent studies apply SMoE in a wide range of machine learning problems such as machine translation (Lepikhin et al., 2021), image classification (Riquelme et al., 2021), speech recognition (Kumatani et al., 2021).”
• ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, methodology): “Kumatani et al. (2021) reduced word error rates using MoE models in Sequence-to-Sequence Transformer and Transformer Transducer.”