VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019

By Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Watiasri Sakti, Haizhou Li, Satoshi Nakamura. 2019

76 citations recorded by the source corpus.

We describe our submitted system for the ZeroSpeech Challenge 2019. The current challenge theme addresses the difficulty of constructing a speech synthesizer without any text or phonetic labels and requires a system that can (1) discover subword units in an unsupervised way, and (2) synthesize the speech with a target speaker's voice. Moreover, the system should also balance the discrimination score ABX, the bit-rate compression rate, and the naturalness and the intelligibility of the constructed voice. To tackle these problems and achieve the best trade-off, we utilize a vector quantized variational autoencoder (VQ-VAE) and a multi-scale codebook-to-spectrogram (Code2Spec) inverter trained by mean square error and adversarial loss. The VQ-VAE extracts the speech to a latent space, forces itself to map it into the nearest codebook and produces compressed representation. Next, the inverter generates a magnitude spectrogram to the target voice, given the codebook vectors from VQ-VAE. In our experiments, we also investigated several other clustering algorithms, including K-Means and GMM, and compared them with the VQ-VAE result on ABX scores and bit rates. Our proposed approach significantly improved the intelligibility (in CER), the MOS, and discrimination ABX scores compared to the official ZeroSpeech 2019 baseline or even the topline.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “Other related work includes learning representations from auto-encoding the input data [52, 11] or directly predicting future timesteps [8].”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, background): “A popular approach to discover discrete units is via autoencoding (Tjandra et al., 2019; Eloff et al., 2019; Chorowski et al., 2019) sometimes coupled with an autoregressive model (Chung et al., 2019).”
A Survey on Neural Speech Synthesis (2021, methodology): “Techniques Data Work Self-supervised Training Unpaired text or speech [51, 368, 433, 78, 140, 346, 197, 352, 71] Cross-lingual Transfer Paired text and speech [42, 390, 12, 60, 271, 105] Cross-speaker Transfer Paired text and speech [216, 125, 59, 39] Speech chain/Back transformation Unpaired text o”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Learning low bitrate speech representations in an unsupervised manner, attracts attention from both the machine learning and the speech communities (Liu et al., 2019a; Feng et al., 2019; Nayak et al., 2019; Tjandra et al., 2019; Schneider et al., 2019; Baevski et al., 2020a; Chen and Hain, 2020; Mor”
Speech-to-Speech Translation Between Untranscribed Unknown Languages (2019, background): “Even though the lack of natural speech dataset in this paper, VQ-VAE and codebook inverter can be applied and has shown a great performance on multispeaker natural speech [14, 13].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “Following the success of representation learning, baseline and challenge techniques have shifted from purely generative models [327], [328], deep autoencoders [83], [329], and incorporation of neural-network-”
Exploring TTS without T Using Biologically/Psychologically Motivated Neural Network Modules (ZeroSpeech 2020) (2020, background): “Even though the participants in the previous challenge mainly investigated mechanically sophisticated systems [1, 2], we addressed the challenge using biologically/psychologically motivated ANN modules, by considering the unsupervised learning of human language as a biological/psychological problem.”
Cyclic Spectral Modeling for Unsupervised Unit Discovery into Voice Conversion with Excitation and Waveform Modeling (2020, methodology): “A lot of work have been proposed for the unsupervised discovery of subword units [3, 6, 8, 11, 12, 13].”
Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders (2020, background): “However, it is notable that VQVAE based systems [24, 25] suffered from poor quality of the reconstructed speech.”
Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge (2020, methodology): “In Table 2, we compared our proposed model with our submission from previous challenge [2] and we also explored different codebook sizes to determine the trade-off between the ABX error rate and bitrate.”
Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, background): “Various objectives have been proposed, such as autoencoding [41]–[43] sometimes with an autoregressive model [44], [45] or contrastive learning [24], [25], [46], [47].”
Textually Pretrained Speech Language Models (2023, background): “The common approach is to first encode the speech into a continuous representation and then quantize the representation to achieve a sequence of discrete tokens [Tjandra et al., 2019, 2020, Lakhotia et al., 2021, Borsos et al., 2022].”
Unsupervised Learning of Disentangled Speech Content and Style Representation (2020, background): “[17, 18, 19] leverages vector-quantized VAE [20] (VQVAE) to learn a discrete sequence representation of speech.”
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge (2020, background): “More recent studies have explored neural networks with intermediate discretization [20–23].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “Most work applied pre-trained speech models to only inference problems such as ASR (Baevski et al., 2019) or phone discrimination (Kharitonov et al., 2020), with a notable exception being Tjandra et al. (2019a), which focuses on text-free machine translation.”
Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction (2020, methodology): “In this paper, we focus on Vector Quantized Variational AutoEncoder (VQ-VAE) [4] since it is a promising selfsupervised representation learning approach suitable for tasks such as voice conversion (VC) [4, 5], text-to-speech (TTS) synthesis [6, 7, 8], speech coding [9], and even music generation [10”