Unsupervised Pretraining Transfers Well Across Languages

By Morgane Rivière, Armand Joulin, Pierre-Emmanuel Mazar'e, Pierre-Emmanuel Mazaré, Emmanuel Dupoux. 2020

225 citations recorded by the source corpus.

Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting. This assumes the existence of a parallel corpus of speech and orthographic transcriptions. Recently, contrastive predictive coding (CPC) algorithms have been proposed to pretrain ASR systems with unlabelled data. In this work, we investigate whether unsupervised pretraining transfers well across languages. We show that a slight modification of the CPC pretraining extracts features that transfer well to other languages, being on par or even outperforming supervised pretraining. This shows the potential of unsupervised methods for languages with few linguistic resources.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “It has been observed for Natural Language Processing [12, 39], Speech Recognition [43, 45] and Computer Vision [35].”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “The latent representations are fed to a Transformer network to build contextualized representations and the model is trained via a contrastive task where the true latent is to be distinguished from distractors [54, 49, 48, 28] (§ 2).”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “We compare to m-CPC [40], [41], XLSR-10 [14] and XLSR-53 [14]. respect to the read audiobook domain of CommonVoice.”
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
Unsupervised Speech Recognition (2021, methodology): “As unlabeled text data for Kyrgyz we use the Kyrgyz community corpus 2017,5 and newscrawl 2008-2014 and 2018-2020.6 For Tatar we use the Tatar community corpus 2017,7 and newscrawl 2005-2011.8 We evaluate on the dev and test split of Rivière et al. (2020).”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Other related work includes pre-training on multiple languages and investigating how representations transfer between languages [18, 19].”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “We present Mono-5K and Multi50Ks’ performance with the m-CPC (Rivière et al., 2020) and XLSR (Conneau et al.”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, neutral): “Even though recent work show that unsupervised representations derived from English transfer across languages [28], more work is required to extend this benchmark to more languages, and ultimately, to underresourced languages.”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, neutral): “This experiment demonstrates that the variety of voices synthesized by SPEAR-TTS is independent from the number of distinct speaker voices contained in the parallel data used for training S 1 .”
SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “Discriminative modeling for SSL studied here include CPC [11, 34], wav2vec [12], vq-wav2vec [13], wav2vec 2.0 [14] and HuBERT [35].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “Besides, speech representations also help transfer learning and adaptation across different data distributions [6], [7], [9], [20], [21].”
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain (2020, methodology): “Recent works have demonstrated an interest in unsupervised representation learning as a pretraining method to obtain good speech features for downstream tasks with little labelled data [1, 2, 3, 4, 5].”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Previous SOTAs: We compare to the best end-to-end (E2E) model from previous literature (Wang et al., 2020b; Iranzo-S´anchez et al., 2020) on each translation direction, which is usually the best-performing multilingual model trained with parallel data from all directions (both X-En and En-X) and als”
Are Discrete Units Necessary for Spoken Language Modeling? (2022, methodology): “a) ZeroSpeech 2021 Baseline: The ZeroSpeech 2021 Baseline System [9] is a composite of three components: an acoustic model (CPC, [18], [19]), a clustering module (k-means) and”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, background): “While speech representations can be defined by different transformations on the surface feature, recent researches [1, 2, 3, 4, 5, 6, 7, 8, 9] have shown great success by combining neural networks and self-supervised learning (where learning targets can be derived from the input itself).”