Unsupervised Speech Representation Learning Using WaveNet Autoencoders

By Jan Chorowski, Ron J. Weiss, Samy Bengio, Aäron van den Oord. 2019

340 citations recorded by the source corpus.

We consider the task of unsupervised extraction of meaningful latent representations of speech by applying autoencoding neural networks to speech waveforms. The goal is to learn a representation able to capture high level semantic content from the signal, e.g. phoneme identities, while being invariant to confounding low level details in the signal such as the underlying pitch contour or background noise. Since the learned representation is tuned to contain only phonetic content, we resort to using a high capacity WaveNet decoder to infer information discarded by the encoder from previous samples. Moreover, the behavior of autoencoder models depends on the kind of constraint that is applied to the latent representation. We compare three variants: a simple dimensionality reduction bottleneck, a Gaussian Variational Autoencoder (VAE), and a discrete Vector Quantized VAE (VQ-VAE). We analyze the quality of learned representations in terms of speaker independence, the ability to predict phonetic content, and the ability to accurately reconstruct individual spectrogram frames. Moreover, for discrete encodings extracted using the VQ-VAE, we measure the ease of mapping them to phonemes. We introduce a regularization scheme that forces the representations to focus on the phonetic content of the utterance and report performance comparable with the top entries in the ZeroSpeech 2017 unsupervised acoustic unit discovery task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “prior assumption, such as continuous [29], discrete [31], [42],”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, background): “These networks are in turn fine-tuned on supervised data [37–43].”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “, [14, 15, 16, 17]), where the focus is to learn phonetic or word-like units, or representations that can distinguish among such units.”
Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, neutral): “Recently, a similar paradigm has been used to infer audio representations [11, 12], including learning speech representations with mutual information [13, 14].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “Baseline models In Table 1, we compare our results to those derived from two of the topperforming submissions to the ZeroSpeech 2019 challenge: a re-implementation of WaveNetVQ (Chorowski et al., 2019) provided by Cho et al. (2019) and FHVAE-DPGMM (Feng et al., 2019).”
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, background): “Some recent works [8, 9, 10] successfully performed phonetic clustering to a good degree with proper quantization during learning the representations.”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, background): “A popular approach to discover discrete units is via autoencoding (Tjandra et al., 2019; Eloff et al., 2019; Chorowski et al., 2019) sometimes coupled with an autoregressive model (Chung et al., 2019).”
VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, methodology): “We also attempted further enhancement of the synthesized voice using several techniques, such as WaveNet [26, 27] and GAN-based voice conversion [28].”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “In contrast to the end-to-end methodology of [33], we train the compression model separately from the vocoder.”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “There has been work on unsupervised learning for speech but the resulting representations have not been applied to improve supervised speech recognition (Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019).”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, methodology): “Predictive coding on wave samples [21] has a long and influential history in speech processing, and its recent neural version [22] and variants, such as Contrastive Predictive Coding (CPC) [23], have also been used to learn speech representation [11].”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “The research line of generative learning can be traced back to the auto-encoding model, which reconstructs the whole speech from latent variables, either continuous [25]– [27] or discrete [28].”
Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders (2021, background): “good inductive bias for speech and perform well on unsupervised acoustic unit discovery tasks [10, 11].”
The Zero Resource Speech Challenge 2019: TTS without T (2019, background): “Recent work shows that such autoencoders can be trained with a discrete intermediate representation [20, 21].”
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition (2019, methodology): “WaveNet auto-encoders [22] proposed contrastive predictive coding (CPC) to learn speech representations and was applied on unsupervised acoustic unit discovery task.”