Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction

By Weiran Wang, Qingming Tang, Karen Livescu. 2020

99 citations recorded by the source corpus.

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “Our approach encodes speech audio via a multi-layer convolutional neural network and then masks spans of the resulting latent speech representations [26, 56], similar to masked language modeling [9].”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Another line of work is pre-training representations on unlabeled speech followed by fine-tuning on labeled data [16, 17, 18, 19, 20, 21, 22, 23, 24].”
Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022, methodology): “Besides, several works (Wang et al., 2020; Xiang et al., 2020) utilize both domain-related corpus and new pre-training objectives for continual pre-training, or revise the Transformer structure of PLMs for better adaption (Ghosal et al., 2020).”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Examples of information include spectrograms (Chung et al., 2019; Wang et al., 2020; Chi et al., 2021; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski et al., 2019; Hsu et al., 2021), derived signal processing features…”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Signiﬁcant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are ﬁne-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “slices [12]–[14], [20]–[22], [24]–[26].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “In speech, instead of prediction, we randomly mask temporal slices of acoustic features and attempt to reconstruct them [21, 22, 23, 24, 25, 16].”
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain (2020, neutral): “Interestingly, Contrastive Predictive Coding (CPC) [6], an emerging representation learning algorithm, turns out to be also useful as a pretraining method in the low resource setting: it yields good speech features that can be fine-tuned for downstream supervised tasks with little labelled data [7, ”
Vector-Quantized Autoregressive Predictive Coding (2020, methodology): “For experiments, we mainly follow the implementations in the original papers [9, 3], except that the number of layers are reduced to match ours to minimize the architectural differences.”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, methodology): “RNN-MLM [8] 3-layer Bi-GRU Global O(T · d) 72x 32.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “While temporal masking is most common, frequency masking has also been adopted for representation learning (Wang, Tang, and Livescu 2020).”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Wang et al. (2020b) is a concurrent work that proposes a text-free end-to-end image-to-speech model, which simpliﬁes the task by using pairs of image and synthesized speech generated from a single-speaker TTS model to reduce the acoustic variation.”
Towards Unsupervised Learning of Speech Features in the Wild (2021, methodology): “The encoder architecture is the same (5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256), for the context network, we used 2-layer LSTM, and for the prediction network, we used a multi-head transformer [35], each of the 12 heads predicting one…”
Joint Unsupervised and Supervised Training for Multilingual ASR (2021, background): “In speech recognition, recent works have shown successes of the 2-stage pretrain-finetune schemes [4, 5, 6, 7, 8].”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “Other studies [12, 13, 14, 15, 16, 17] were motivated by NLP and applied similar methods on speech tasks.”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “Predictionbased self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions [29, 30, 31, 32, 33, 34, 35] or to contrast the target unseen frame with randomly sampled ones [5, 36, 6, 11].”
Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing (2021, methodology): “While a range of self-supervised methods have been developed for speech [6–11], in this paper we focus on contrastive predictive coding (CPC) [12].”