Improved Noisy Student Training for Automatic Speech Recognition

By Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, Quoc V. Le. 2020

267 citations recorded by the source corpus.

Recently, a semi-supervised learning method known as "noisy student training" has been shown to improve image classification performance of deep networks significantly. Noisy student training is an iterative self-training method that leverages augmentation to improve network performance. In this work, we adapt and improve noisy student training for automatic speech recognition, employing (adaptive) SpecAugment as the augmentation method. We find effective methods to filter, balance and augment the data generated in between self-training iterations. By doing so, we are able to obtain word error rates (WERs) 4.2%/8.6% on the clean/noisy LibriSpeech test sets by only using the clean 100h subset of LibriSpeech as the supervised set and the rest (860h) as the unlabeled set. Furthermore, we are able to achieve WERs 1.7%/3.4% on the clean/noisy LibriSpeech test sets by using the unlab-60k subset of LibriLight as the unlabeled set for LibriSpeech 960h. We are thus able to improve upon the previous state-of-the-art clean/noisy test WERs achieved on LibriSpeech 100h (4.74%/12.20%) and LibriSpeech (1.9%/4.1%).

Subfields: Artificial Intelligence, Ocean Engineering, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Ocean Engineering

In Natural Language Processing

In Signal Processing

Citations in the corpus

Self-training Improves Pre-training for Natural Language Understanding (2020, background): “, 2020), automatic speech recognition (Synnaeve et al., 2019; Kahn et al., 2020; Park et al., 2020), sequence generation (He et al.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “Moreover, when we lower the amount of labeled data to just one hour, we still outperform the previous state of the art self-training method of [42] while using 100 times less labeled data and the same amount of unlabeled data.”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “Downstream NST training for full Voice Search sets: Application of the standard downstream NST recipe used in this paper of pseudo-labeling either YT-U or YT-T, filtering 50% using confidence-per-word and mixing the pseudo-labeled data with labeled data, has not been able to improve full Voice Searc”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “We include comparison with semi-supervised (iterative pseudo labeling (IPL) [12], slimIPL [54], noisy student [61]) and self-supervised approaches (DeCoAR 2.”
Unsupervised Speech Recognition (2021, background): “, 2020b) by carefully filtering the unlabeled data to match the target domain (Park et al., 2020).”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “Multi-lingual training [10, 11], multi-domain training [12, 13], unsupervised pre-training [14, 15], semi-supervised learning [16, 17] and transfer learning [18, 19] are some techniques proposed in the literature to enhance generalization.”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Another line of work investigated filtering the resulting pseudo-labeled data to match the distribution of the original labeled data [15].”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “In [8], filtering teacher-generated transcripts using the confidence derived from the fused ASR model was shown not to be beneficial for the NST pipeline for the LibriSpeech + Libri-Light task.”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, methodology): “These techniques include SpecAugment [39, 40] for data augmentation, Noisy Student Training [41] for self-training, and language model fusion for decoding.”
Meta Pseudo Labels (2020, methodology): “The method of Pseudo Labels, also known as self-training, is a simple Semi-Supervised Learning (SSL) approach that has been successfully applied to improve the state-of-the-art of many tasks, such as: image classification [79, 77], object detection, semantic segmentation [89], machine translation [2”
CVSS Corpus and Massively Multilingual Speech-to-Speech Translation (2022, positive): “Following Jia et al. (2019b), we evaluated the translation quality and speech generation quality of S2ST 2020) trained on LibriSpeech and Libri-Light (Kahn et al., 2020), and computed BLEU using SacreBLEU (Post, 2018) with its default conﬁguration.”
Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation (2022, methodology): “We used the same ASR model from [49] for evaluation, following [9], therefore the results are comparable to [9].”
Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Such behavior has already been observed in the experiments of [26], where the authors compare two different student models for the same ContextNet teacher [27] in a noisy student setting [28].”
Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting (2021, background): “Deep learning has enabled the application of automatic speech recognition (ASR) to commercial services [1]–[7].”
Large-Scale Self- and Semi-Supervised Learning for Speech Translation (2021, methodology): “then used to synthetically annotate unlabeled data in order to train a new student model on the combination of labeled and pseudo-labeled data [35, 23, 36].”
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (2023, methodology): “We utilize noisy student training (NST) [7, 8] to generate pseudo-labeled data to augment supervised training.”
Semi-supervised ASR by End-to-end Self-training (2020, background): “More recent studies [26, 27] have similarly shown the effectiveness of data augmentation for unsupervised data in self-training.”
Joint Masked CPC And CTC Training For ASR (2020, methodology): “More broadly, our single-stage training method can be linked to semi-supervised learning or self-training methods [12, 2, 13, 3, 7] for speech recognition.”