XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

By Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski and 2 more. 2021

1,102 citations recorded by the source corpus.

This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “On VoxPopuli, however, our system significantly underperforms both XLS-R (Babu et al., 2021) and mSLAM (Bapna et al.”
FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “More recently, mSLAM [2], a joint speech and text multilingual pretrained model, outperformed XLS-R on speech translation and ASR and improved over speech-only baselines on Speech-LangID.”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, background): “Representation learning approaches can be extended to multilingual settings for text [2, 4], speech [24] and joint models [11].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “6B), which is already very competitive with the state-of-the-art, outperforming XLS-R (1B) (Babu et al., 2021), despite having fewer parameters and not using a pre-trained decoder.”
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, neutral): “3a shows the Voicebox inference time to generate an audio sample of 10 seconds (including vocoding and predicting duration) as NFE varies and compares that to VALL-E.”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, neutral): “Similarly to observed in ST tasks (Weiss et al., 2017; Bansal et al., 2019; Jia et al., Table 4: Multilingual X → En S2ST performance.”
HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, neutral): “Unlike HierSpeech [2] which used XLS-R [67], we use a massively multilingual speech (MMS) [68] which is a pretrained Wav2Vec 2.0 with a massive scale.”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “To better understand how the MMS models compare to XLS-R we fine-tuned both for automatic speech recognition on the 61 languages of the FLEURS benchmark for which MMS-lab provides training data.”
Writing System and Speaker Metadata for 2,800+ Language Varieties (2022, background): “, 2020) have led to the development of pretrained multilingual models (Conneau et al., 2020; Chung et al., 2021; Babu et al., 2021), significantly reducing the barriers to extending language technologies to new languages (Pfeiffer et al.”
SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations (2022, methodology): “R model with 1B parameters (Babu et al., 2021) and decoder is initialized from a unit decoder pretrained in an mBART style (Popuri et al.”
HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis (2022, positive): “3B) (Babu et al., 2021) which is pre-trained wav2vec 2.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “and conventionally used for evaluation, while their audio streams are also aggregated to build diversified and large-scale corpora for unsupervised pre-training [109], [123], [218].”
Textless Direct Speech-to-Speech Translation with Discrete Speech Representation (2022, methodology): “The results on CVSS-C are grouped into high/mid/low-resource language pair groups based on the amount of data available in the CVSS-C corpus, following [26, 31].”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “XLS-R (Babu et al., 2022) Speech Classification, Speech Translation, Speech Recognition 2021 Encoder & Decoder Yes VoxPopuli (VP400K), Multilingual Librispeech (MLS), CommonVoice, VoxLingua107, BABEL VoxPopuli, Multilingual Librispeech (MLS), CommonVoice, BABEL”
NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, methodology): “We followed the official train/test split of Wang et al. (2022), i.e. 95 songs and 5 songs for training and evaluation, respectively.”
DUB: Discrete Unit Back-translation for Speech Translation (2023, methodology): “∗ : results from (Wu et al., 2022) ♢ : results from (Babu et al., 2021).”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “The results are grouped into high/mid/low-resource language groups based on the amount of data available in the CVSS-C corpus, following [18, 21].”
Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap (2022, neutral): “w2v2-L-vox [39] VoxPopuli 100k 23 P w2v2-L-xls-r [40] VoxPopuli (372k) ML LibriSpeech (50k) CommonVoice (7k) VoxLingua107 (6.”