Generative Pre-Training for Speech with Autoregressive Predictive Coding

By Yu-An Chung, James R. Glass. 2019

187 citations recorded by the source corpus.

Learning meaningful and general representations from unannotated speech that are applicable to a wide range of tasks remains challenging. In this paper we propose to use autoregressive predictive coding (APC), a recently proposed self-supervised objective, as a generative pre-training approach for learning meaningful, non-specific, and transferable speech representations. We pre-train APC on large-scale unlabeled data and conduct transfer learning experiments on three speech applications that require different information about speech characteristics to perform well: speech recognition, speech translation, and speaker identification. Extensive experiments show that APC not only outperforms surface features (e.g., log Mel spectrograms) and other popular representation learning methods on all three tasks, but is also effective at reducing downstream labeled data size and model parameters. We also investigate the use of Transformers for modeling APC and find it superior to RNNs.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Scaling Laws for Acoustic Models (2021, methodology): “The two models evaluated in this work share the same basic auto-predictive coding (APC) design, shown in Figure 1.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “creasing interests recently, where a model is tasked to predict the content of the unseen regions [4], [44]–[50] or to contrast the target unseen frame with randomly sampled ones [1]– [3], [6].”
RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “Preva-lent methods usually requires the model to predict the content of unseen regions (Hsu et al., 2021; Baevski et al., 2022; Chung et al., 2021; Chung and Glass, 2020) or to contrast the target unseen frame with randomly sampled ones (Baevski et al., 2020; Conneau et al., 2021).”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Signiﬁcant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are ﬁne-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
Vector-Quantized Autoregressive Predictive Coding (2020, methodology): “We follow [14] and consider a 259-class classi-ﬁcation task where each class corresponds to an unique speaker, using 80% of si284 for training, the other 10% for validation, and reporting classiﬁcation error rate on the rest 10%.”
Contrastive Semi-Supervised Learning for ASR (2021, background): “Recently, self-supervised representation learning approaches have been proposed for ASR pre-training with competitive performance especially with a very small amount of labeled data [16, 17, 19, 20, 21, 22, 18, 23].”
Injecting Text in Self-Supervised Speech Pretraining (2021, methodology): “Pretraining techniques such as MoCo [4], Contrastive Predicting Coding (CPC) [5, 6], Autoregressive Per-dictive Coding (APC) [7], SimCLR [8], etc., generalize using un-transcribed speech, however, they cannot leverage unspoken text, thereby limiting the power of the learned representations.”
CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learning (2022, background): “Along this idea, a number of training approaches have been studied [2, 3, 4, 5, 6, 7], that include wav2vec [8, 9, 10], HuBERT [5], and the recent data2vec [7].”
Generative Pre-training for Speech with Flow Matching (2023, methodology): “One branch of self-supervised learning takes the autoregressive approach to learn from predicting the future, such as contrastive predictive coding (Oord et al., 2018) and autoregre-sive predictive coding (Chung & Glass, 2020).”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “self-supervised learning technologies [10, 42, 97, 338] which can learn a very good representation that carries semantic information.”
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020, methodology): “After training, the RNN hidden states are taken as the learned representations, and are shown to contain speech information such as phonetic and speaker content that are useful in a variety of speech tasks (Chung and Glass, 2020).”