Self-Training and Pre-Training are Complementary for Speech Recognition

By Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, Michael Auli. 2020

180 citations recorded by the source corpus.

Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data. However, it is not clear whether they learn similar patterns or if they can be effectively combined. In this paper, we show that pseudo-labeling and pre-training with wav2vec 2.0 are complementary in a variety of labeled data setups. Using just 10 minutes of labeled data from Libri-light as well as 53k hours of unlabeled data from LibriVox achieves word error rates (WER) of 2.8%/4.8% on the clean and other test sets of Librispeech – rivaling the best published systems trained on 960 hours of labeled data only a year ago. Training on all labeled data of Librispeech achieves WERs of 1.5%/3.1%.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Scaling Laws for Acoustic Models (2021, methodology): “Successful techniques use either autoregressive predictive coding (APC) [7, 8, 9] or contrastive predictive coding (CPC) [10, 11, 12] to pre-train an acoustic model from a large corpus of unlabeled speech.”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “…models from HuggingFace, downloaded as of September 2022 using version 4.21.0 of the transformers library: • facebook/wav2vec2-large-960h-lv60-self (Xu et al., 2021) • facebook/wav2vec2-large-robust-ft-libri-960h (Hsu et al., 2021b) • facebook/wav2vec2-base-100h (Baevski et al., 2020) •…”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “In particular, multilingual pre-trained models like XLS-R expanded the few-shot learning capability of wav2vec 2.0 (Xu et al., 2021) to many other languages, both for speech recognition and speech translation (Wang et al., 2021b).”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “The largest model, XLSR-53, was trained on about 50K hours of public training xls_r data in 53 languages and comprises about 300M parameters [14].”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, background): “Our work is an extension of a host of recent research efforts [12], [23], [26], [27], [29] that have studied semi-supervised”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, positive): “However, as observed in [63] and [40], we expect that HuBERT can achieve comparable or better performance after combining with self-training, since the pre-trained HuBERT model is on par or better than the”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “…et al., 2018; Schneider et al., 2019; Baevski et al., 2020; Conneau et al., 2020; Wu et al., 2020; Nguyen et al., 2020), with semi-supervised learning (self-training) (Kahn et al., 2020a; Pino et al., 2020; Zhang et al., 2020b; Xu et al., 2020) or a combination of both methods (Xu et al., 2020).”
Direct Speech-to-Speech Translation With Discrete Units (2021, background): “0 pre-training and self-training [34] for evaluating the speech output.”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “Section 4 describes the experimental setup, followed by our results and analysis in Section 5 where we apply pre-trained w2v-BERT models to LibriSpeech and voice search ASR.”
Self-Training: A Survey (2022, unknown): “Building upon this idea, [85] showcased that the synergy between self-training and unsupervised pre-training using wav2vec 2.”
XTREME-S: Evaluating Cross-lingual Speech Representations (2022, background): “0 [1] have emerged as the predominant approach for automatic speech recognition (ASR) and direct speech-to-text translation (ST), and have made speech models much more data efficient: ASR models can be learnt with as little as a few hours of labeled data [2, 3].”
TS2Vec: Towards Universal Representation of Time Series (2021, background): “…representation learning has achieved good performances in computer vision (Chen et al. 2020; Wang et al. 2021; Xu et al. 2020; O. Pinheiro et al. 2020), natural language processing (Gao, Yao, and Chen 2021; Logeswaran and Lee 2018) and speech recognition (Baevski et al. 2020; Xu et al. 2021).”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “Indeed, W2V2 has already been studied with focus on the impact of pre- training data (Conneau et al., 2020; Hsu et al., 2021), pre-training task (Hsu et al., 2020), or combination with pseudo labelling (Xu et al., 2020a; Zhang et al., 2020b).”
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis (2023, methodology): “As in [11], content preservation is evaluated by running a publicly available Automatic Speech Recognition (ASR) model [19] on the resynthesized sentence and computing the”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “In addition, it was recently shown that self-training and self-supervised pretraining can be effectively combined [9, 10] for speech recognition to achieve 4.”
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “Here, we analyze and verify our method on using the ASR data compared with the other two methods, including 1) self-training (Xu et al., 2021b) and 2) semi-supervised pre-training which combines supervised and unsupervised learning in a multi-task pre-training process like mSLAM (Bapna et al.,…”