Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters

By Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Y. Hannun, Vitaliy Liptchinsky, Gabriel Synnaeve, Ronan Collobert. 2020

172 citations recorded by the source corpus.

We study training a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages, and over-all simplifying deployment of ASR systems that support diverse languages. We perform an extensive benchmark on 51 languages, with varying amount of training data by language(from 100 hours to 1100 hours). We compare three variants of multilingual training from a single joint model without knowing the input language, to using this information, to multiple heads (one per language cluster). We show that multilingual training of ASR models on several languages can improve recognition performance, in particular, on low resource languages. We see 20.9%, 23% and 28.8% average WER relative reduction compared to monolingual baselines on joint model, joint model with language input and multi head model respectively. To our knowledge, this is the first work studying multilingual ASR at massive scale, with more than 50 languages and more than 16,000 hours of audio across them.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Finetuned Language Models Are Zero-Shot Learners (2021, background): “The diverse capabilities of language models at scale have drawn attention to the tradeoffs between specialist models (one model per task) and generalist models (one model for many tasks; Arivazhagan et al., 2019; Pratap et al., 2020), for which our study has potential implications.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “Toshniwal et al. (2018) demonstrated jointly training a modern deep learning speech recognition system on several languages with a single model, and Pratap et al. (2020a) scaled this line of work significantly to 50 languages with a billion-parameter model.”
A Generalist Agent (2022, background): “…Alayrac et al., 2022), code generation (Chen et al., 2021c; Li et al., 2022b), dialogue systems with retrieval capabilities (Nakano et al., 2021; Thoppilan et al., 2022), speech recognition (Pratap et al., 2020), neural machine translation (Johnson et al., 2019) and more (Bommasani et al., 2021).”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, neutral): “This is consistent with results in multilingual machine translation (MT) and automatic speech recognition (ASR) where adding more languages can also lead to degradation for highresource languages (Aharoni et al., 2019; Pratap et al., 2020).”
SpeakerStew: Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Verification System (2021, background): “Recently, [21] and [22] built a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, methodology): “, 50 languages in [140]), the data from those languages is heavily imbalanced, which usually leads to a model performing well on resource-rich languages while failing on low-resource languages.”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, methodology): “Unlike [19], no details were provided on how their best 1B model was developed.”
Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021, background): “sonably performing ASR and MT models cannot be guaranteed (Conneau et al., 2020; Pratap et al., 2020), as well as to finer-grained intent classes and other domains.”
End-to-End Speech Recognition: A Survey (2023, unknown): “Pretraining approaches utilizing untranscribed audio, large-scale semi-supervised data and/or multilingual data [120], [121], [122], [123], [124], [125], [126], [127], [128], [129], [130], [131] would deserve”
Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Veriﬁcation System (2021, background): “Recently, [19] and [20] built a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages.”