Multilingual Speech Recognition with a Single End-to-End Model

By Shubham Toshniwal, Tara N. Sainath, Ron J. Weiss, Bo Li, Pedro J. Moreno Mengibar, Eugene Weinstein, Kanishka Rao. 2017

292 citations recorded by the source corpus.

Training a conventional automatic speech recognition (ASR) system to support multiple languages is challenging because the sub-word unit, lexicon and word inventories are typically language specific. In contrast, sequence-to-sequence models are well suited for multilingual ASR because they encapsulate an acoustic, pronunciation and language model jointly in a single network. In this work we present a single sequence-to-sequence ASR model trained on 9 different Indian languages, which have very little overlap in their scripts. Specifically, we take a union of language-specific grapheme sets and train a grapheme-based sequence-to-sequence model jointly on data from all languages. We find that this model, which is not explicitly given any information about language identity, improves recognition performance by 21% relative compared to analogous sequence-to-sequence models trained on each language individually. By modifying the model to accept a language identifier as an additional input feature, we further improve performance by an additional 7% relative and eliminate confusion between different languages.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “Toshniwal et al. (2018) demonstrated jointly training a modern deep learning speech recognition system on several languages with a single model, and Pratap et al. (2020a) scaled this line of work significantly to 50 languages with a billion-parameter model.”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “Introduction The use of multilingual ASR systems [1, 2, 3, 4, 5, 6, 7, 8, 9] that simultaneously transcribe multiple languages has recently become popular to increase language coverage, but covering all of the world’s ∼7000 languages is still far ahead.”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, background): “This has been a long standing interest in the speech community [1, 2, 3, 4, 5, 6, 7] which includes systems able to transcribe multiple languages [8, 9, 10, 11, 12].”
Bytes Are All You Need: End-to-end Multilingual Speech Recognition and Synthesis with Bytes (2018, background): “In addition, a simple 1-hot language ID vector has been found to be effective improving multilingual systems [24, 25].”
An Attention-Based Joint Acoustic and Text on-Device End-To-End Model (2020, methodology): “ASRU, 2017, pp. 193–199.”
SpeakerStew: Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Verification System (2021, background): “Recently, [21] and [22] built a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages.”
Analyzing ASR Pretraining for Low-Resource Speech-to-Text Translation (2019, background): “Although our main focus is monolingual pretraining, we also looked briefly at multilingual pretraining, inspired by recent work on multilingual ASR [29, 30] and evidence that multilingual pretraining followed by fine-tuning on a distinct target language can improve ASR on the target language [11, 31”
Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, background): “There has been a great deal of attention to multi-lingual automatic speech recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model (2019, methodology): “While previous works have investigated various aspects of data sampling [16, 17], as well as architectures that include a language vector [11,18,19], this is the ﬁrst study to apply adapter modules [20] to speech recognition.”
One-to-Many Multilingual End-to-End Speech Translation (2019, methodology): “Target forcing has also been applied to multilingual speech recognition [31, 32] showing to improve the transcription quality, although multilingual ASR shows to be better than its monolingual counterparts even when the language token is not provided.”
Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Veriﬁcation System (2021, background): “Recently, [19] and [20] built a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages.”
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion (2018, methodology): “To obtain a multilingual S2S model, a part of parameters can be shared while preparing the output layers per language [6], and we can further use a unified architecture with a shared vocabulary among multiple languages [15–17].”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “set and then training an E2E model with all the data [38, 131, 149, 276, 298].”
Universal Phone Recognition with a Multilingual Allophone System (2020, background): “While some recent work in multilingual ASR focuses on end-to-end models to directly predict graphemes [8, 9], most systems still depend on phonetically inspired acoustic models.”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “…much work on multilingual speech recognition [Burget et al., 2010, Lin et al., 2009, Heigold et al., 2013, Bourlard et al., 2011, Cho et al., 2018, Toshniwal et al., 2018, Kannan et al., 2019, Li et al., 2019, Pratap et al., 2020b], many deployed systems are still trained on data in a single…”
Multi-Modal Data Augmentation for End-to-end ASR (2018, methodology): “In [22], a multi-task framework is used to jointly perform language-id and speechto-text in a multilingual ASR setting.”
Pre-training on high-resource speech recognition improves low-resource speech-to-text translation (2018, background): “For ASR, training a single model on multiple languages can be effective for all of them [11,12].”
Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling (2018, methodology): “On a similar front, attention models are explored within a multi-lingual setup in [16, 17] based on attention-based seq2seq to build a model from multiple languages.”