Improved Speech Representations with Multi-Target Autoregressive Predictive Coding

By Yu-An Chung, James R. Glass. 2020

59 citations recorded by the source corpus.

Training objectives based on predictive coding have recently been shown to be very effective at learning meaningful representations from unlabeled speech. One example is Autoregressive Predictive Coding (Chung et al., 2019), which trains an autoregressive RNN to generate an unseen future frame given a context such as recent past frames. The basic hypothesis of these approaches is that hidden states that can accurately predict future frames are a useful representation for many downstream tasks. In this paper we extend this hypothesis and aim to enrich the information encoded in the hidden states by training the model to make more accurate future predictions. We propose an auxiliary objective that serves as a regularization to improve generalization of the future frame prediction task. Experimental results on phonetic classification, speech recognition, and speech translation not only support the hypothesis, but also demonstrate the effectiveness of our approach in learning representations that contain richer phonetic content.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “Recent works propose to predict future frames from the history with an autoregressive model (Chung et al., 2020; 2019; Chung & Glass, 2020; Chung & Glass, 2020), or recover the masked frames from the corrupted speech with a non-autoregressive model (Liu et al.”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Examples of information include spectrograms (Chung et al., 2019; Wang et al., 2020; Chi et al., 2021; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski et al., 2019; Hsu et al., 2021), derived signal processing features…”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Significant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are fine-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, positive): “In [10], the APC objective is extended to multi-target training.”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “APC [19] and its follow-up [20] are examples to use autoregressive reconstruction loss.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, background): “…Lewis et al., 2019; Lample and Con-neau, 2019; Dong et al., 2019; Liu et al., 2020; Tang et al., 2020; Rivi`ere et al., 2020; Kawakami et al., 2020; Chung and Glass, 2020; Baevski et al., 2020), which has achieved state-of-the-art results when ﬁnetuning on downstream tasks in NLP (Liu et al.,…”
Vector-Quantized Autoregressive Predictive Coding (2020, methodology): “The exact same setup described in [24] is used in our experiments.”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, methodology): “APC can be improved by enforcing constraints that information from past sequences be stored in the representation [10] or by imposing an information bottleneck via vector quantization [11].”
Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, background): “Various objectives have been proposed, such as autoencoding [41]–[43] sometimes with an autoregressive model [44], [45] or contrastive learning [24], [25], [46], [47].”
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, methodology): “Pre-trained speech models have been applied to ASR (Ling et al., 2020; Chung and Glass, 2020a; Baevski et al., 2020b), phoneme recognition (Song et al., 2020; Liu et al., 2020a), speech translation (Nguyen et al., 2020; Chung et al., 2019c), and speech synthesis (Chung et al., 2019b), to name a few.”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “Examples of information include spectrograms (Wang et al., 2020; Chi et al., 2021; Chung and Glass, 2020), cluster indices (Hsu et al.”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “Prediction-based self-supervised learning has gathered increasing interests recently, where a model is tasked to predict the content of the unseen regions [29, 30, 31, 32, 33, 34, 35] or to contrast the target unseen frame with randomly sampled ones [5, 36, 6, 11].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “In [97], the APC objective is extended to multi-target training.”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, background): “[12, 13, 14] proposed Autoregressive Predictive Coding (APC) objective that predicts unseen future frame given past frames and also achieved good results on phonetic classification, speech recognition, and speech translation.”