Joint Unsupervised and Supervised Training for Multilingual ASR

By Junwen Bai, Bo Li, Yu Zhang, Ankur Bapna, Nikhil Siddhartha, Khe Chai Sim, Tara N. Sainath. 2021

66 citations recorded by the source corpus.

Self-supervised training has shown promising gains in pretraining models and facilitating the downstream finetuning for speech recognition, like multilingual ASR. Most existing methods adopt a 2-stage scheme where the self-supervised loss is optimized in the first pretraining stage, and the standard supervised finetuning resumes in the second stage. In this paper, we propose an end-to-end (E2E) Joint Unsupervised and Supervised Training (JUST) method to combine the supervised RNN-T loss and the self-supervised contrastive and masked language modeling (MLM) losses. We validate its performance on the public dataset Multilingual LibriSpeech (MLS), which includes 8 languages and is extremely imbalanced. On MLS, we explore (1) JUST trained from scratch, and (2) JUST finetuned from a pretrained checkpoint. Experiments show that JUST can consistently outperform other existing state-of-the-art methods, and beat the monolingual baseline by a significant margin, demonstrating JUST’s capability of handling low-resource languages in multilingual ASR. Our average WER of all languages outperforms average monolingual baseline by 33.3%, and the state-of-the-art 2-stage XLSR by 32%. On low-resource languages like Polish, our WER is less than half of the monolingual baseline and even beats the supervised transfer learning method which uses external supervision.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Dong Yu
4. Shujie Liu
5. Furu Wei
6. Ye Jia
7. Zhuo Chen
8. Ankur Bapna

In Experimental and Cognitive Psychology

1. Colin Cherry
2. Heiga Zen
3. Frank K. Soong

In Speech & Audio

1. Jinyu Li
2. Yu Zhang
3. Ye Jia
4. Heiga Zen
5. Bhuvana Ramabhadran
6. Andrew Rosenberg
7. Shujie Liu
8. Pedro J. Moreno Mengibar

In Signal Processing

1. Jinyu Li
2. Dong Yu
3. Zhuo Chen
4. Shujie Liu
5. Yu Zhang
6. Yu Wu
7. Shinji Watanabe
8. Takuya Yoshioka

Citations in the corpus

• MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “by utilizing additional supervised data, joint unsupervised and supervised training on transcribed speech [25] or paired Masked Language Modeling (MLM) objectives on Machine Translation (MT) [26] or Speech Translation (ST) data [27].”
• mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Past work also leverages supervised ASR data to build multilingual representations of speech (Kannan et al., 2019; Bai et al., 2021), similar to how multilingual machine translation in NLP is used to build multilingual representations (Eriguchi et al., 2018; Siddhant et al., 2020).”
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “UniSpeech [7] and JUST [45] combine SSL and supervised learning for ASR, and show impressive results on multi-lingual test sets.”
• Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, background): “There has been a great deal of attention to multi-lingual automatic speech recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”