Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge

By Benjamin van Niekerk, Leanne Nortje, Herman Kamper. 2020

127 citations recorded by the source corpus.

In this paper, we explore vector quantization for acoustic unit discovery. Leveraging unlabelled data, we aim to learn discrete representations of speech that separate phonetic content from speaker-specific details. We propose two neural models to tackle this challenge - both use vector quantization to map continuous features to a finite set of codes. The first model is a type of vector-quantized variational autoencoder (VQ-VAE). The VQ-VAE encodes speech into a sequence of discrete units before reconstructing the audio waveform. Our second model combines vector quantization with contrastive predictive coding (VQ-CPC). The idea is to learn a representation of speech by predicting future acoustic units. We evaluate the models on English and Indonesian data for the ZeroSpeech 2020 challenge. In ABX phone discrimination tests, both models outperform all submissions to the 2019 and 2020 challenges, with a relative improvement of more than 30%. The models also perform competitively on a downstream voice conversion task. Of the two, VQ-CPC performs slightly better in general and is simpler and faster to train. Finally, probing experiments show that vector quantization is an effective bottleneck, forcing the models to discard speaker information.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, background): “Another line of work explored quantization of the continuous speech data to identify speech units in order to learn representations (Baevski et al., 2020b,a; Liu et al., 2019; van Niekerk et al., 2020; Baevski et al., 2020c).”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “The authors in [20] suggested using transformer [21] together with a VQVAE model for unsupervised unit discovery, and in [22] they combine vector quantization with contrastive predictive coding for acoustic unit discovery.”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Tjandra et al. (2020) suggested to use transformer (Vaswani et al., 2017) together with a VQ-VAE model for unsupervised unit discovery, and van Niekerk et al. (2020) combines vector quantization together with contrastive predictive coding for acoustic unit discovery.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “SSL techniques are also benchmarked with the ZeroSpeech challenge [265], [266].”
NoiseVC: Towards High Quality Zero-Shot Voice Conversion (2021, methodology): “Such an idea has been implemented in VC model [26], but they can only do many-to-many conversions.”
The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units (2020, background): “BN [12] (SU) VQVAE (1), VQCPC (2) WaveRNN”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “Although vector quantization was introduced by the VQ-VAE which is, in some ways, a latent variable model, it has been applied to self-supervised methods (van Niekerk, Nortje, and Kamper 2020; Baevski, Schneider, and Auli 2019).”
Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders (2020, unknown): “As shown in Figure 2, the IN-WAE obtains a better MOS score while a worse ABX score than [39].”