Iterative Pseudo-Labeling for Speech Recognition

By Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Y. Hannun, Gabriel Synnaeve, Ronan Collobert. 2020

156 citations recorded by the source corpus.

Pseudo-labeling has recently shown promise in end-to-end automatic speech recognition (ASR). We study Iterative Pseudo-Labeling (IPL), a semi-supervised algorithm which efficiently performs multiple iterations of pseudo-labeling on unlabeled data as the acoustic model evolves. In particular, IPL fine-tunes an existing model at each iteration using both labeled data and a subset of unlabeled data. We study the main components of IPL: decoding with a language model and data augmentation. We then demonstrate the effectiveness of IPL by achieving state-of-the-art word-error rate on the Librispeech test sets in both standard and low-resource setting. We also study the effect of language models trained on different corpora to show IPL can effectively utilize additional text. Finally, we release a new large in-domain text corpus which does not overlap with the Librispeech training transcriptions to foster research in low-resource, semi-supervised ASR

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, background): “This propagation can either be done with hard assignments of labels [41, 78, 79] or with a soft assignment [76].”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “Most recent work [51, 58, 17, 42] uses the better performing word pieces [50] for both models.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “For analysis, we derive frame-level forced-aligned phonetic transcripts using a hybrid ASR system to measure the correlation between the k-means cluster assignments and the actual phonetic units.”
Unsupervised Speech Recognition (2021, background): “This is a long-standing research area in speech recognition with the most recent work focusing on self-training (Kahn et al., 2020a) and iterative self-training (Xu et al., 2020b) by carefully filtering the unlabeled data to match the target domain (Park et al., 2020).”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Recently, substantial performance improvements have been made by leveraging audio-only data for speech recognition [14, 15, 16].”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, positive): “This includes classical self-training [9, 10, 11] which demonstrated strong results [12, 13, 2, 14, 15] by pseudolabeling unannotated audio data and then retraining the final system with the additional labeled data.”
Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training (2020, methodology): “ST has recently shown excellent performance for neural sequence generation tasks such as machine translation [15] and ASR [16–18], achieving stateof-the-art performance for semi-supervised ASR when applied in an iterative manner [19].”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “Semi-supervised learning techniques for ASR such as unsupervised speech pretraining [1, 2, 3, 4] and self-training or (iterative) pseudo-labeling [5, 6, 7, 8] have substantially improved performance on the traditional LibriSpeech benchmark, and led to systems that can learn with little supervision.”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “Indeed, W2V2 has already been studied with focus on the impact of pre- training data (Conneau et al., 2020; Hsu et al., 2021), pre-training task (Hsu et al., 2020), or combination with pseudo labelling (Xu et al., 2020a; Zhang et al., 2020b).”
Semi-supervised ASR by End-to-end Self-training (2020, background): “More recent studies [26, 27] have similarly shown the effectiveness of data augmentation for unsupervised data in self-training.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “For comparison, we also show performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student [62]), as well as the current state of the art—conformer XXL + noisy student [255]—which augments SSL with various advanced techniques including self-training.”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, methodology): “In the original IPL algorithm [5], PLs are generated with a beamsearch decoder leveraging an LM and approximating the solution of Eq.”
Contrastive Semi-Supervised Learning for ASR (2021, methodology): “Using multiple rounds of pseudo-labeling for model pre-training was found very effective for the classical GMM/HMM systems in the mid 2000s [5] and more recently for neural systems [11, 32, 8].”