Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition

By Shaoshi Ling, Yuzong Liu, Julian Salazar, Katrin Kirchhoff. 2019

150 citations recorded by the source corpus.

We propose a novel approach to semi-supervised automatic speech recognition (ASR). We first exploit a large amount of unlabeled audio data via representation learning, where we reconstruct a temporal slice of filterbank features from past and future context frames. The resulting deep contextualized acoustic representations (DeCoAR) are then used to train a CTC-based end-to-end ASR system using a smaller amount of labeled audio data. In our experiments, we show that systems trained on DeCoAR consistently outperform ones trained on conventional filterbank features, giving 42% and 19% relative improvement over the baseline on WSJ eval92 and LibriSpeech test-clean, respectively. Our approach can drastically reduce the amount of labeled data required; unsupervised training on LibriSpeech then supervision with 100 hours of labeled data achieves performance on par with training on all 960 hours directly.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Scaling Laws for Acoustic Models (2021, methodology): “Following the design in [13], the first of”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “These networks are in turn fine-tuned on supervised data [37–43].”
Improved Noisy Student Training for Automatic Speech Recognition (2020, background): “In this approach, a separate task/loss to (pre-)train the acoustic model (or a part thereof) is introduced to help the model learn a good representation of the input, e.g., [24, 25, 26, 27, 28, 29, 30].”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “WER Test WER Labeled Unlabeled clean other clean other Semi-supervision with PL [10] LS-100 LS-360 GCNN 5.37 22.13 5.93 24.07 Local Prior Matching [11] LS-100 LS-860 GCNN 4.87 13.84 4.88 15.28 DeCoAR [9] LS-100 LS-860 4-gram - - 4.74 12.20 vq-wav2vec + BERT [8] LS-100 LS-860 4-gram 4.0 10.9 4.5 12.1”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “, 2020; 2019; Chung & Glass, 2020; Chung & Glass, 2020), or recover the masked frames from the corrupted speech with a non-autoregressive model (Liu et al., 2020a;b;c; Ling et al., 2020; Ling & Liu, 2020).”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “…include spectrograms (Chung et al., 2019; Wang et al., 2020; Chi et al., 2021; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski et al., 2019; Hsu et al., 2021), derived signal processing features (Pascual et al.,…”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Signiﬁcant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are ﬁne-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “2 wav2vec-large [12] 960 hr 100 hr - 6.”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “DeCoAR stands for deep contextualized acoustic representations , and was proposed in our previous work [1].”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “Recently, self-supervised learning—a paradigm that treats the input itself or modiﬁcations of the input as learning targets—has obtained promising results for learning such transformations [1, 2, 4, 5, 6, 7, 8, 9, 3, 10].”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “One of the main differences of data2vec to most other masked prediction work (Devlin et al., 2019; Baevski et al., 2020b; Ling et al., 2020; Bao et al., 2021; He et al., 2021; Wei et al., 2021) is that the features of the training targets are contextualized since the features are built with self-att”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, positive): “This idea of learning contextual embedding based on the local neighbors in the sequence have been found useful in the field of learning word embedding [20, 12] and have been extended to speech representation learning before [21, 22, 23, 24, 25].”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “Some work has extended APC and CPC inspired models with separate encoders operating in opposite directions (Ling et al. 2020; Kawakami et al. 2020; Borgholt et al. 2021b), but these models are still restricted to process left and right context separately.”
Scaling Effect of Self-Supervised Speech Models (2021, background): “Among recent self-supervised speech studies, Mockingjay [10], TERA [11], APC [12] and DeCoAR [13] learn speaker embedding representations based on self-supervised spectrogram reconstructions or predictions.”
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data (2021, background): “Masked reconstruction (Liu et al., 2020; Ling et al., 2020a;b) has also been widely investigated which masks part of the input and learn to reconstruct it.”
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, methodology): “In language, BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), UniLM (Dong et al., 2019), and BART (Lewis et al., 2020) have been successfully applied to natural language inference (Zhang et al., 2020b), question answering (Zhu et al., 2018), and summarization (Zhu et al., 2019).”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “Prediction-based self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions [29, 30, 31, 32, 33, 34, 35] or to contrast the target unseen frame with randomly sampled ones [5, 36, 6, 11].”