Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies

By Alexander H. Liu, Yu-An Chung, James R. Glass. 2020

96 citations recorded by the source corpus.

Self-supervised speech representations have been shown to be effective in a variety of speech applications. However, existing representation learning methods generally rely on the autoregressive model and/or observed global dependencies while generating the representation. In this work, we propose Non-Autoregressive Predictive Coding (NPC), a self-supervised method, to learn a speech representation in a non-autoregressive manner by relying only on local dependencies of speech. NPC has a conceptually simple objective and can be implemented easily with the introduced Masked Convolution Blocks. NPC offers a significant speedup for inference since it is parallelizable in time and has a fixed inference time for each time step regardless of the input sequence length. We discuss and verify the effectiveness of NPC by theoretically and empirically comparing it with other methods. We show that the NPC representation is comparable to other methods in speech experiments on phonetic and speaker classification while being more efficient.

Subfields: Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
SUPERB: Speech processing Universal PERformance Benchmark (2021, neutral): “…- 0 10ms waveform - - - PASE+ [16] SincNet, 7-Conv, 1-QRNN 7.83M 10ms waveform LS 50 hr multi-task santi-pdp / pase APC [7] 3-GRU 4.11M 10ms FBANK LS 360 hr F-G iamyuanchung / APC VQ-APC [32] 3-GRU 4.63M 10ms FBANK LS 360 hr F-G + VQ iamyuanchung / VQ-APC NPC [33] 4-Conv mentioning speciﬁcally.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “In non-autoregressive predictive coding (NPC) [107], time masking is introduced through masked convolution blocks.”
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, methodology): “Pre-trained speech models have been applied to ASR (Ling et al., 2020; Chung and Glass, 2020a; Baevski et al., 2020b), phoneme recognition (Song et al., 2020; Liu et al., 2020a), speech translation (Nguyen et al., 2020; Chung et al., 2019c), and speech synthesis (Chung et al., 2019b), to name a few.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “(16) Non-autoregressive predictive coding (NPC, Liu, Chung, and Glass 2021) tries to resolve this by using a convolutional neural network where the kernel is masked instead of the input.”
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022, unknown): “, 2020), NPC (Liu et al., 2020a), Mockingjay (Liu et al.”