slimIPL: Language-Model-Free Iterative Pseudo-Labeling
By Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, Ronan Collobert. 2020
73 citations recorded by the source corpus.
Recent results in end-to-end ASR have demonstrated the efficacy of simple pseudo-labeling for semi-supervised models trained both with Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq) losses. Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further increase performance in ASR. We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens) assignments, that is without a language model. We call this approach Language-Model-Free IPL (slimIPL) and we give a resultant training setup for CTC and seq2seq models. At inference, our experiments show that decoding with a strong language model is more beneficial with slimIPL than IPL, asIPL exhibits some language model over-fitting issues. Compared to prior work on semi-supervised and unsupervised approaches, slimIPL not only simplifies the training process, but also achieves competitive and state-of-the-art results on LibriSpeech test sets in both standard and low-resource settings.
Subfields: Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Representation & Self-Supervised Learning
- Wei-Ning Hsu
- Abdel-rahman Mohamed
- Alexei Baevski
- Hung-yi Lee
- Michael Auli
- Alexis Conneau
- Karen Livescu
- Kushal Lakhotia
Citations in the corpus
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “11 SlimIPL [54] LS-860 4-gram + Transformer 2.”
- Unsupervised Speech Recognition (2021, neutral): “This is in line with recent observations about overfitting in self-training for speech recognition (Likhomanenko et al., 2021).”
- Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, background): “Semi-supervised learning techniques for ASR such as unsupervised speech pretraining [1, 2, 3, 4] and self-training or (iterative) pseudo-labeling [5, 6, 7, 8] have substantially improved performance on the traditional LibriSpeech benchmark, and led to systems that can learn with little supervision.”
- Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “To speed up the model training (2-3x) and decrease memory usage we are using CAPE positional embedding (Likhomanenko et al., 2021c) instead of relative one (Shaw et al., 2018): both models perform in the same ballpark.”
- Self-Supervised Speech Representation Learning: A Review (2022, methodology): “For comparison, we also show performance from such methods (iterative pseudo labeling (IPL) [63], slimIPL [254], noisy student [62]), as well as the current state of the art—conformer XXL + noisy student [255]—which augments SSL with various advanced techniques including self-training.”