Sequence-Based Multi-Lingual Low Resource Speech Recognition

By Siddharth Dalmia, Ramon Sanabria, Florian Metze, Alan W. Black. 2018

101 citations recorded by the source corpus.

Techniques for multi-lingual and cross-lingual speech recognition can help in low resource scenarios, to bootstrap systems and enable analysis of new languages and domains. End-to-end approaches, in particular sequence-based techniques, are attractive because of their simplicity and elegance. While it is possible to integrate traditional multi-lingual bottleneck feature extractors as front-ends, we show that end-to-end multi-lingual training of sequence models is effective on context independent models trained using Connectionist Temporal Classification (CTC) loss. We show that our model improves performance on Babel languages by over 6% absolute in terms of word/phoneme error rate when compared to mono-lingual systems built in the same setting for these languages. We also show that the trained model can be adapted cross-lingually to an unseen language using just 25% of the target data. We show that training on multiple languages is important for very low resource cross-lingual target scenarios, but not for multi-lingual testing scenarios. Here, it appears beneficial to include large well prepared datasets.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Xu Tan
3. Tie-Yan Liu
4. Jinyu Li
5. Shinji Watanabe
6. Graham Neubig
7. Yi Ren
8. Zhou Zhao

In Speech & Audio

1. Jinyu Li
2. Xu Tan
3. Tao Qin
4. Tie-Yan Liu
5. Sheng Zhao
6. Yi Ren
7. Shinji Watanabe
8. Zhou Zhao

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Tao Qin
4. Tie-Yan Liu
5. Shinji Watanabe
6. Yi Ren
7. Zhou Zhao
8. Sheng Zhao

Citations in the corpus

• Unsupervised Pretraining Transfers Well Across Languages (2020, neutral): “The model is then finetuned on the target low-resource language to fit its specificities [12].”
• Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, background): “, 2018b), low-resource ASR (Chuangsuwanich, 2016; Dalmia et al., 2018; Zhou et al., 2018), TTS with minimal speaker data (Chen et al.”
• Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, unknown): “In [14, 15, 16, 17], similar empirical results also indicated that [ja]設置しました”
• Universal Phone Recognition with a Multilingual Allophone System (2020, methodology): “The second group, private phoneme models, treats phonemes from each language as completely different classes performs phoneme classification separately for each language [11, 2, 12].”
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, background): “Transfer learning from highresource languages [16, 17, 18, 19] is known to be an efficient way to improve end-to-end ASR for low-resource languages.”
• Symbolic Inductive Bias for Visually Grounded Learning of Spoken Language (2018, background): “More recently, Dalmia et al. (2018) show that exploiting this effect is especially useful for low-resource languages.”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “A multilingual E2E model already captures lots of information across languages, which makes the transfer learning using the target language data very effective [38, 49, 108, 114, 229].”
• Massively Multilingual Adversarial Speech Recognition (2019, background): “These approaches have also been deployed in multilingual settings (Toshni-wal et al., 2017; Chiu et al., 2018; M¨uller et al., 2017; Dalmia et al., 2018; Watanabe et al., 2017a).”
• Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling (2018, methodology): “An extensive analysis on multilingual CTC mainly focusing on improving under limited data condition is performed in [15].”
• An open-source end-to-end ASR system for Brazilian Portuguese using DNNs built from newly assembled corpora (2020, background): “[13] shared the same weights from the recurrent layers over different languages and trained additional layers to develop a multilingual ASR system, which improved the final word error rate by over 6% when compared to monolingual systems.”
• Equity Beyond Bias in Language Technologies for Education (2019, background): “The relative sparsity of data for these variations may have resulted in unacceptable modeling accuracy in the past (Blodgett et al., 2016), but we are now closing that gap (Dalmia et al., 2018; Sitaram et al., 2019).”