Self-Training for End-to-End Speech Recognition

By Jacob Kahn, Ann Lee, Awni Y. Hannun. 2019

264 citations recorded by the source corpus.

We revisit self-training in the context of end-to-end speech recognition. We demonstrate that training with pseudo-labels can substantially improve the accuracy of a baseline model. Key to our approach are a strong baseline acoustic and language model used to generate the pseudo-labels, filtering mechanisms tailored to common errors from sequence-to-sequence models, and a novel ensemble approach to increase pseudo-label diversity. Experiments on the LibriSpeech corpus show that with an ensemble of four models and label filtering, self-training yields a 33.9% relative improvement in WER compared with a baseline trained on 100 hours of labelled data in the noisy speech setting. In the clean speech setting, self-training recovers 59.3% of the gap between the baseline and an oracle model, which is at least 93.8% relatively higher than what previous approaches can achieve.

Subfields: Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

Citations in the corpus

Self-training Improves Pre-training for Natural Language Understanding (2020, background): “…applied to a variety of tasks, including image recognition (Yalniz et al., 2019; Xie et al., 2020; Zoph et al., 2020), automatic speech recognition (Syn-naeve et al., 2019; Kahn et al., 2020; Park et al., 2020), sequence generation (He et al., 2019), and parsing (McClosky et al., 2006).”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “Next, a student model is trained using the combined supervised and teacher-labeled data either using the standard cross-entropy [9] loss or using a contrastive loss [11] to account for noise in teacher-generated labels.”
Unsupervised Speech Recognition (2021, methodology): “Self-training is one of the simplest but most effective approaches to leveraging unpaired speech (Kahn et al., 2020a).”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “Multilingual speech corpora Librilight (Kahn et al., 2020b) currently represents the largest scale unlabelled speech corpus but it is limited to English.”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “This includes classical self-training [9, 10, 11] which demonstrated strong results [12, 13, 2, 14, 15] by pseudolabeling unannotated audio data and then retraining the final system with the additional labeled data.”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “Self-training has been a popular technique in ASR studied extensively in the literature [25–33].”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “When supervised data is not abundant, self-supervised pre-training (Schneider et al., 2019; Wu et al., 2020) and semi-supervised training (Kahn et al., 2020; Pino et al., 2020) lowers the requirements on super-vision and improves model performance.”
Improved Noisy Student Training for Automatic Speech Recognition (2020, background): “Self-training has a long history of being applied to ASR through numerous studies, e.g., [12, 13, 14, 15, 16, 17, 18, 19].”
Iterative Pseudo-Labeling for Speech Recognition (2020, positive): “IPL is motivated by the simplicity and effectiveness of PL [9, 1].”
Learning Robust and Multilingual Speech Representations (2020, neutral): “2011). Recently, unsupervised representation learning algorithms have shown signiﬁcant improvements at learning representations that correlate well with phonetic structure (van den Oord et al., 2018; Kahn et al., 2019b) and improving downstream speech recognition performance (Schneider et al., 2019;”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, neutral): “In the semi-supervised setting, only a fraction of the dataset is labelled and the rest is unlabelled [1, 2], while in a distant supervision setting, the dataset is mostly or entirely unlabelled, but large quantities of unaligned text provide a language model corpus [3, 4].”
Effectiveness of self-supervised pre-training for speech recognition (2019, methodology): “(Irie et al., 2019; Kahn et al., 2019; Lscher et al., 2019) evaluated semi-supervised self-labeling WER performance on the standard test “clean” and test “other” benchmarks of the Librispeech dataset (Panayotov et al., 2015) when using only 100 hour subset as labeled data.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “self-training [18, 29] or back-translation [30] from unlabeled data have been shown to be useful to ASR or S2T.”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “Section 4 describes the experimental setup, followed by our results and analysis in Section 5 where we apply pre-trained w2v-BERT models to LibriSpeech and voice search ASR.”
Rethinking Pre-training and Self-training (2020, background): “The recent successes of self-training [9–12] raise the question to what degree does self-training work better than pre-training.”
Meta Pseudo Labels (2020, methodology): “The method of Pseudo Labels, also known as self-training, is a simple Semi-Supervised Learning (SSL) approach that has been successfully applied to improve the state-of-the-art of many tasks, such as: image classification [79, 77], object detection, semantic segmentation [89], machine translation [2”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, methodology): “We show that LPM achieves better WER compared to a strong pseudo-label baseline (Kahn et al., 2019a) as well as other forms of knowledge distillation. ar X iv :2 00 2.”
Self-Training for End-to-End Speech Translation (2020, neutral): “[12] and [13] also leverage pseudo-labeling on the LIBRILIGHT dataset [14] to improve the performance of an end-to-end ASR system.”