Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

By Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, Yonghui Wu. 2020

334 citations recorded by the source corpus.

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “More recent work has explored billion-parameter models (Zhang et al., 2020) and using up to 1,000,000 hours of training data (Zhang et al., 2021).”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, background): “All fine-tuning data and parameters follow [7].”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “Our finetuning parameters follow (Zhang et al., 2020); for the pre-trained encoder, we use a peak learning rate of 3e− 4 with 5k warm-up steps, while for the decoder, a peak learning rate of 1e − 3 and 1.5k warm-up steps.”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “To our knowledge, this is the largest effort to date, in making speech technology accessible for many more languages using publicly available data.”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “By comparing the second and third rows, we see that pre-training on YT-U is able to benefit a wider variety of domains compared to LibriSpeech.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “1 pre-trained Conformer XXL + Noisy Student [40] LL-60k LSTM 1.”
Unsupervised Speech Recognition (2021, methodology): “We add self-loops that mimic CTC behavior (Zhang et al., 2020a) where blank symbols (silence for the GAN or actual blank symbol for letter models) are mapped to epsilons and consecutive predictions of the same symbol are collapsed.”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “End-to-end speech recognition models have also been benefited from the increases in model capacity [28, 29, 5].”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, background): “Finally, there has been little research (Jia et al., 2019; Tjandra et al., 2019; Zhang et al., 2020a) into speech output due to the lack of open data.”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “The self-supervised pretraining paradigm has significantly advanced the state of the art across various domains, from natural language processing [6, 17–19, 23, 24, 28, 30, 61, 68, 73, 83–85], to computer vision [2, 5, 8, 12, 31, 33, 37, 59, 75, 104, 117], to speech recognition [4, 22, 42, 67, 119] ”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “There have also been studies that show that the gains brought by self-training and unsupervised pre-training are additive in downstream ASR [21, 22].”
Going deeper with Image Transformers (2021, background): “They became a staple in speech recognition too [45, 36], being it in encoder and sequence criterion or encoder-decoder seq2seq [61] conformations, and hold the state of the art to this day [74, 78] with models 36 blocks deep.”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “Recently, and following success on NLP tasks, there is a transition in speech processing towards the Transformer architecture (Vaswani et al., 2017; Dong et al., 2018) and its variants (Zhang et al., 2020a; Baevski et al., 2020b; Gulati et al., 2020; Zhang et al., 2020b; Yeh et al., 2019).”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, methodology): “For competitive results in ASR, a language model (LM) was a key component of pseudo-labeling: it is usually combined with the acoustic model via beam-search decoding [4, 5, 6] or through shallow fusion [7, 8, 9, 10] to generate PLs.”
Prompt Consistency for Zero-Shot Task Generalization (2022, background): “We then vary the number of available prompts by randomly sampling a subset of prompts before training.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “self-supervised learning technologies [10, 42, 97, 338] which can learn a very good representation that carries semantic information.”
Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Such behavior has already been observed in the experiments of [26], where the authors compare two different student models for the same ContextNet teacher [27] in a noisy student setting [28].”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, background): “Large data sets and giant models, together with advances in deep learning algorithms and hardware, enable researchers to push the limit of many machine learning tasks [1–3] including speech [4].”