Unsupervised Cross-lingual Representation Learning for Speech Recognition

By Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdel-rahman Mohamed, Michael Auli. 2020

1,027 citations recorded by the source corpus.

This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages. We build on a concurrently introduced self-supervised model which is trained by solving a contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The resulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction of 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to the strongest comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong individual models. Analysis shows that the latent discrete speech representations are shared across languages with increased sharing for related languages.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “Speech technology has been rapidly evolved in the past few years, with striking achievements nourished from selfattention models [3, 4], pre-training approaches [5, 6, 7, 2], and massively multilingual speech models [8, 9].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Similarly, in speech understanding, multilingual pre-trained models (Kawakami et al., 2020; Conneau et al., 2020; Babu et al., 2021) based on self-supervised losses (Oord et al.”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “For speech processing, self-supervised approaches such as wav2vec 2.0 [7, 12] have also been extended to the multilingual setting [13, 14].”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Other related work includes pre-training on multiple languages and investigating how representations transfer between languages [18, 19].”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “Furthermore, multilingual pre-training (Conneau et al., 2020) allows cross-lingual transfer, which brings extra gains especially to low-resource languages.”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “To retain the performance of previous tasks during fine-tuning, following Conneau et al. (2020), the training data are re-sampled with respect to tasks with α = 0 .”
Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, positive): “To this end, we resort to XLSR-53: a wav2vec 2.0 model pre-trained on 56k hours of speech in 53 languages (Conneau et al., 2020).”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “In addition, current open source Mandarin corpora are also unable to train a well generalized pre-trained model, since both the Wav2vec 2.0 large model [18] and the XLSR-53 [24] use more than 50000 hours English speech data.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “For many-to-one experiments, we also experiment with a multi-lingual wav2vec 2.0 (XMEF-X), which was pre-trained on raw audio from 53 languages (Conneau et al., 2020).”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, neutral): “(Sohn, 2016; Oord et al., 2018; Wu et al., 2018; Bachman et al., 2019; H´enaff et al., 2019; Baevski et al., 2020; Conneau et al., 2020; Tian et al., 2020; Hjelm et al., 2019; Han et al., 2019; He et al., 2020; Misra & Maaten, 2020; Chen et al., 2020a;b).”
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, methodology): “0 model pre-trained on 56k hours of speech in 53 languages [11].”
UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022 (2022, positive): “This model is xlsr53 [21] fine-tuned on phonetic annotations from word transcriptions obtained using ESpeak(3) and speech samples of CommonVoice [22].”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, result): “To our knowledge, our results are the best reported numbers so far on this multilingual Librispeech dataset in the literature [41, 42].”