Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion

By Hirofumi Inaguma, Jaejin Cho, Murali Karthick Baskar, Tatsuya Kawahara, Shinji Watanabe. 2018

46 citations recorded by the source corpus.

This work explores better adaptation methods to low-resource languages using an external language model (LM) under the framework of transfer learning. We first build a language-independent ASR system in a unified sequence-to-sequence (S2S) architecture with a shared vocabulary among all languages. During adaptation, we perform LM fusion transfer, where an external LM is integrated into the decoder network of the attention-based S2S model in the whole adaptation stage, to effectively incorporate linguistic context of the target language. We also investigate various seed models for transfer learning. Experimental evaluations using the IARPA BABEL data set show that LM fusion transfer improves performances on all target five languages compared with simple transfer learning when the external text data is available. Our final system drastically reduces the performance gap from the hybrid systems.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Graham Neubig
2. Jinyu Li
3. Yu Zhang
4. Shinji Watanabe
5. Kyunghyun Cho
6. Yoshua Bengio
7. Luke S. Zettlemoyer
8. Dong Yu

In Representation & Self-Supervised Learning

1. Michael Auli
2. Alexei Baevski
3. Alexis Conneau
4. Yu Zhang
5. Abdel-rahman Mohamed
6. Sebastian Ruder
7. Jinyu Li
8. Ye Jia

In Speech & Audio

1. Jinyu Li
2. Shinji Watanabe
3. Yu Zhang
4. Wei-Ning Hsu
5. Alexei Baevski
6. Michael Auli
7. Abdel-rahman Mohamed
8. Alexis Conneau

Citations in the corpus

• Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, methodology): “For BABEL, the most comparable work is [33] and in Table 3, XLSR10 shows a relative word error reduction of 16% compared to their BLSTM-HMM baseline.”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “A multilingual E2E model already captures lots of information across languages, which makes the transfer learning using the target language data very effective [38, 49, 108, 114, 229].”
• Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages (2020, methodology): “Some DNN-based systems have successfully applied deep transfer learning, including TTS [31], image classification [34], [35], machine translation [36], [37], automatic speech recognition [38]–[40], language identification [41], and sentiment classification [42].”
• Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, background): “Related work on how to best integrate language models into end-to-end ASR includes deep and shallow fusion [21], cold fusion [22], or transfer fusion [23].”
• Massively Multilingual Adversarial Speech Recognition (2019, methodology): “Our baseline approach to multilingual knowledge transfer is most similar to Inaguma et al. (2018), and involves training a hybrid CTC-attention seed model.”
• XTREME-S: Evaluating Cross-lingual Speech Representations (2022, background): “Recent work revived this dataset with different preprocessing [16, 17, 18, 4].”