Scaling End-to-End Models for Large-Scale Multilingual ASR

By Bo Li, Ruoming Pang, Tara N. Sainath, Anmol Gulati, Yu Zhang, James Qin, Parisa Haghani, W. Ronny Huang, Min Ma, Junwen Bai. 2021

86 citations recorded by the source corpus.

Building ASR models across many languages is a challenging multi-task learning problem due to large variations and heavily unbalanced data. Existing work has shown positive transfer from high resource to low resource languages. However, degradations on high resource languages are commonly observed due to interference from the heterogeneous multilingual data and reduction in per-language capacity. We conduct a capacity study on a 15-language task, with the amount of data per language varying from 7.6K to 53.5K hours. We adopt GShard [1] to efficiently scale up to 10B parameters. Empirically, we find that (1) scaling the number of model parameters is an effective way to solve the capacity bottleneck - our 500M-param model already outperforms monolingual baselines and scaling it to 1B and 10B brought further quality gains; (2) larger models are not only more data efficient, but also more efficient in terms of training cost as measured in TPU days - the 1B-param model reaches the same accuracy at 34% of training time as the 500M-param model; (3) given a fixed capacity budget, adding depth works better than width and large encoders do better than large decoders; (4) with continuous training, they can be adapted to new languages and domains.

Subfields: Artificial Intelligence, Scaling & Learning Theory, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Scaling & Learning Theory

In Speech & Audio

Citations in the corpus

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “In contrast, scaling efforts in speech have focused either on supervised multilingual models [17] or monolingual self-supervised models, counting a billion or more parameters [18], while cross-lingually pretrained speech models are much smaller in scale.”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “Our train and test sets are derived from [42].”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, background): “Moreover, recent studies have shown that a single large model can utilize large data sets more effectively than smaller models [1,4].”
Scaling Speech Technology to 1, 000+ Languages (2023, unknown): “VoxPopuli [Wang et al., 2021] 1 15.3 Maestro [Chen et al., 2022] 14 8.1 RNN-T 1B [Li et al., 2021] 15 7.9 Whisper [Radford et al., 2022] 99 13.6 ∗ 7.3* ML-IO [Tjandra et al., et al., 2022].”
End-to-End Speech Recognition: A Survey (2023, unknown): “…large-scalesemi-superviseddataand/ormultilingualdata[149], [150],[151],[152],[153],[154],[155],[156],[157],[158], [159],[160]woulddeserveaself-containedsurveyandtheyare applicableforhybridDNN/HMMandE2Eapproacheslikewise…”
Multi-modal Deep Learning (2024, methodology): “…risk assessment method for quantifying transformer and secondary conductor overload conditions due to high levels of EV charging demand and rooftop solar PV. Li et al. [10] present an empirical comparison of RNN-T, RNN-AED, and Transformer-AED models in both non-streaming and streaming modes.”
Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition (2021, background): “There has been a great deal of attention to multi-lingual automatic speech recognition (ASR) that can recognize multiple languages [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
Joint Unsupervised and Supervised Training for Multilingual ASR (2021, methodology): “We also include transfer learning models, B0 and E3, from [13], which used heavy supervision from external VoiceSearch (VS) dataset containing 15 languages.”