Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model

By Anjuli Kannan, A. Datta, Tara N. Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Z. Chen, Seungjin Lee. 2019

194 citations recorded by the source corpus.

Multilingual end-to-end (E2E) models have shown great promise in expansion of automatic speech recognition (ASR) coverage of the world's languages. They have shown improvement over monolingual systems, and have simplified training and serving by eliminating language-specific acoustic, pronunciation, and language models. This work presents an E2E multilingual system which is equipped to operate in low-latency interactive applications, as well as handle a key challenge of real world data: the imbalance in training data across languages. Using nine Indic languages, we compare a variety of techniques, and find that a combination of conditioning on a language vector and training language-specific adapter layers produces the best model. The resulting E2E multilingual model achieves a lower word error rate (WER) than both monolingual E2E models (eight of nine languages) and monolingual conventional systems (all nine languages).

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Past work also leverages supervised ASR data to build multilingual representations of speech (Kannan et al., 2019; Bai et al., 2021), similar to how multilingual machine translation in NLP is used to build multilingual representations (Eriguchi et al., 2018; Siddhant et al., 2020).”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, methodology): “Introduction The use of multilingual ASR systems [1, 2, 3, 4, 5, 6, 7, 8, 9] that simultaneously transcribe multiple languages has recently become popular to increase language coverage, but covering all of the world’s ∼7000 languages is still far ahead.”
Learning Robust and Multilingual Speech Representations (2020, neutral): “These results suggests, in practice, that one could use a universal speech feature extractor for many languages instead of training one for each language individually (Kannan et al., 2019).”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, methodology): “promising capabilities of such E2E models in modeling dialects of a particular language [16], languages from the same family [17] and languages from different families [18–21].”
Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, background): “There has been a great deal of attention to multi-lingual automatic speech recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
End-to-End Speech Recognition: A Survey (2023, background): “also help improve ASR development for low resource tasks, again both for classical [298], as well as for E2E systems [299], [300].”
Self-Supervised Representation Learning: Introduction, advances, and challenges (2021, unknown): “However, very rapid progress is currently being made in multilingual [97] speech models and cross-lingual transfer [98], so prospects for transferability seem promising.”
Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, background): “[12] presented a large-scale streaming end-to-end model trained on nine Indian languages using CTC.”
Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “In addition, adapter modules can also be used to connect available pre-trained models such as an ASR model and a multilingual denoising auto-encoder to perform the multilingual speech-to-text translation task.”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “…speech recognition [Burget et al., 2010, Lin et al., 2009, Heigold et al., 2013, Bourlard et al., 2011, Cho et al., 2018, Toshniwal et al., 2018, Kannan et al., 2019, Li et al., 2019, Pratap et al., 2020b], many deployed systems are still trained on data in a single language despite the need to…”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “In [138], language-specific adapt layers are used to further boost the multilingual model performance.”