Vector-Quantized Autoregressive Predictive Coding

By Yu-An Chung, Hao Tang, James R. Glass. 2020

129 citations recorded by the source corpus.

Autoregressive Predictive Coding (APC), as a self-supervised objective, has enjoyed success in learning representations from large amounts of unlabeled data, and the learned representations are rich for many downstream tasks. However, the connection between low self-supervised loss and strong performance in downstream tasks remains unclear. In this work, we propose Vector-Quantized Autoregressive Predictive Coding (VQ-APC), a novel model that produces quantized representations, allowing us to explicitly control the amount of information encoded in the representations. By studying a sequence of increasingly limited models, we reveal the constituents of the learned representations. In particular, we confirm the presence of information with probing tasks, while showing the absence of information with mutual information, uncovering the model's preference in preserving speech information as its capacity becomes constrained. We find that there exists a point where phonetic and speaker information are amplified to maximize a self-supervised objective. As a byproduct, the learned codes for a particular model capacity correspond well to English phones.

Subfields: Artificial Intelligence, Cancer Research, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cancer Research

In Generative Models

In Signal Processing

Citations in the corpus

Scaling Laws for Acoustic Models (2021, methodology): “The two models evaluated in this work share the same basic auto-predictive coding (APC) design, shown in Figure 1.”
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “Recent works propose to predict future frames from the history with an autoregressive model [28]– [31], or recover the masked frames from the corrupted speech”
SUPERB: Speech processing Universal PERformance Benchmark (2021, neutral): “…- 0 10ms waveform - - - PASE+ [16] SincNet, 7-Conv, 1-QRNN 7.83M 10ms waveform LS 50 hr multi-task santi-pdp / pase APC [7] 3-GRU 4.11M 10ms FBANK LS 360 hr F-G iamyuanchung / APC VQ-APC [32] 3-GRU 4.63M 10ms FBANK LS 360 hr F-G + VQ iamyuanchung / VQ-APC NPC [33] 4-Conv mentioning speciﬁcally.”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “classification on LibriSpeech only serves as a sanity check for the presence of speaker identity [1], [8], [11], [20], [21].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “In VQ-APC [27], the authors use VQ as a way to limit model capacity and control information needed in encoding representation.”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, methodology): “The model pre-trained with NPC on LibriSpeech is fixed and used to extract representations from the Wall Street Journal corpus (WSJ) [18] for the following tasks where the setting aligns with the previous work [11].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “In VQ-APC [98], quantization is used with the APC”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “Finally, Chung, Tang, and Glass (2020) explore quantization between different layers in the APC model, but find that continuous representations consistently perform better than their quantized counterparts on a downstream phoneme classification task Given our previous discussion of how it might not…”
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022, unknown): “We evaluate the tasks on 15 upstream models, which are PASE+ (Ravanelli et al., 2020), APC (Chung et al., 2019), VQ-APC (Chung et al., 2020), NPC (Liu et al., 2020a), Mockingjay (Liu et al., 2020c), TERA (Liu et al., 2020b), DeCoAR 2.0 (Ling and Liu, 2020), Modiﬁle CPC (Rivière et al., 2020), wav2ve”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “…signal conditioned on some other part of the signal, using standard prediction losses such as L2 distance or cross-entropy (Chung et al. 2019; Chung, Tang, and Glass 2020; Liu et al. 2020; Liu, Li, and Lee 2021; Ling and Liu 2020; Hsu et al. 2021a; Chen et al. 2021), using a contrastive…”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, methodology): “Prior work in self-supervised speech recognition has begun to synthesize parts of CPC and MLM to produce methodologies for self-learning with raw waveforms [13, 44, 45, 41, 31].”