Semi-Supervised Speech Recognition via Local Prior Matching

By Wei-Ning Hsu, Ann Lee, Gabriel Synnaeve, Awni Y. Hannun. 2020

32 citations recorded by the source corpus.

For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability. In this work, we propose local prior matching (LPM), a semi-supervised objective that distills knowledge from a strong prior (e.g. a language model) to provide learning signal to a discriminative model trained on unlabeled speech. We demonstrate that LPM is theoretically well-motivated, simple to implement, and superior to existing knowledge distillation techniques under comparable settings. Starting from a baseline trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data, LPM recovers 54% and 73% of the word error rate on clean and noisy test sets relative to a fully supervised model on the same data.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “Pseudo-labeling (PL), also known as self-training and belongs to the family of semi-supervised learning techniques, has been the dominant approach for utilizing unlabeled speech and audio with successful applications dating back to the mid1990s [7]–[10].”
Unsupervised Speech Recognition (2021, methodology): “There has also been recent work on distilling knowledge from a strong prior such as a language model to provide a learning signal to a discriminative model trained on unlabeled speech (Hsu et al., 2020).”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Therefore, purely supervised training is impractical for the vast majority of the 7,000 languages spoken around the world [5] which is why there has been a lot of interest in how to better use unlabeled speech data [6, 7, 8].”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “Many algorithms exist which incorporate unlabelled data to improve ASR in the low-resource setting, including CPC [7, 8], pseudo-labeling [9], local prior matching [10], pseudo-label augmentation [11], adversarial training [12] and back translation [13].”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, background): “On the other hand, seq2seq models showed greater grace working with noisy labels both for personal assistant domains [16] and audio books [49, 50].”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, methodology): “For competitive results in ASR, a language model (LM) was a key component of pseudo-labeling: it is usually combined with the acoustic model via beam-search decoding [4, 5, 6] or through shallow fusion [7, 8, 9, 10] to generate PLs.”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, methodology): “Once this encoder is trained and ﬁxed, a BERT transformer model and pre-training loss is applied over latent discrete sequences as a feature extractor for an ASR model [9] or to directly ﬁne-tune the BERT transformer model using a CTC loss for low-resource scenarios [10].”
Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training (2020, unknown): “ST has recently shown excellent performance for neural sequence generation tasks such as machine translation [15] and ASR [16–18], achieving state-of-the-art performance for semi-supervised ASR when applied in an iterative manner [19].”