Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling

By Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Reddy Mallidi, Nelson Yalta, Martin Karafiát, Shinji Watanabe, Takaaki Hori. 2018

130 citations recorded by the source corpus.

Sequence-to-sequence (seq2seq) approach for low-resource ASR is a relatively new direction in speech research. The approach benefits by performing model training without using lexicon and alignments. However, this poses a new problem of requiring more data compared to conventional DNN-HMM systems. In this work, we attempt to use data from 10 BABEL languages to build a multilingual seq2seq model as a prior model, and then port them towards 4 other BABEL languages using transfer learning approach. We also explore different architectures for improving the prior multilingual seq2seq model. The paper also discusses the effect of integrating a recurrent neural network language model (RNNLM) with a seq2seq model during decoding. Experimental results show that the transfer learning approach from the multilingual model shows substantial gains over monolingual models across all 4 BABEL languages. Incorporating an RNNLM also brings significant improvements in terms of %WER, and achieves recognition performance comparable to the models trained with twice more training data.

Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Machine Translation

In Speech & Audio

In Signal Processing

Citations in the corpus

Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “Multilingual sequence-to-sequence (seq2seq) models have been shown to improve performance in the cross-lingual setting [6], where the model is first pre-trained on a language group and then fine-tuned to a specific target language.”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, methodology): “We adopt the same setup as [6] and pretrain on ten languages: Bengali (bn), Cantonese (zh), Georgian (ka), Haitian (ht), Kurmanji (ku), Pashto (ps), Tamil (ta), Turkish (tr), Tokpisin (tp), Vietnamese (vi).”
Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “The pre-trained model is trained using 10 BABEL languages as described in [30].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “set and then training an E2E model with all the data [38, 131, 149, 276, 298].”
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion (2018, background): “When compared to the previous work using the same data [7], CF-transfer yielded 21.”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “…despite much work on multilingual speech recognition [Burget et al., 2010, Lin et al., 2009, Heigold et al., 2013, Bourlard et al., 2011, Cho et al., 2018, Toshniwal et al., 2018, Kannan et al., 2019, Li et al., 2019, Pratap et al., 2020b], many deployed systems are still trained on…”
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, background): “Transfer learning from highresource languages [16, 17, 18, 19] is known to be an efficient way to improve end-to-end ASR for low-resource languages.”
Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings (2018, background): “Attention-based encoder-decoder networks have achieved stateof-the art performance in ASR when trained on over 12k hours of transcribed speech [1], but their performance lags behind conventional systems in more moderate resource conditions and has only just begun to be studied in low-resource condit”
Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, background): “[13] showed that transfer learning could be adopted for end-to-end multilingual models.”