Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge

By Andros Tjandra, Sakriani Watiasri Sakti, Satoshi Nakamura. 2020

42 citations recorded by the source corpus.

In this paper, we report our submitted system for the ZeroSpeech 2020 challenge on Track 2019. The main theme in this challenge is to build a speech synthesizer without any textual information or phonetic labels. In order to tackle those challenges, we build a system that must address two major components such as 1) given speech audio, extract subword units in an unsupervised way and 2) re-synthesize the audio from novel speakers. The system also needs to balance the codebook performance between the ABX error rate and the bitrate compression rate. Our main contribution here is we proposed Transformer-based VQ-VAE for unsupervised unit discovery and Transformer-based inverter for the speech synthesis given the extracted codebook. Additionally, we also explored several regularization methods to improve performance even further.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “VQ-VAE has been a successful paradigm in speech and image representation, and especially as a unit of modeling for TTS [21, 40].”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “The authors in [20] suggested using transformer [21] together with a VQVAE model for unsupervised unit discovery, and in [22] they combine vector quantization with contrastive predictive coding for acoustic unit discovery.”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Tjandra et al. (2020) suggested to use transformer (Vaswani et al., 2017) together with a VQ-VAE model for unsupervised unit discovery, and van Niekerk et al. (2020) combines vector quantization together with contrastive predictive coding for acoustic unit discovery.”
Textually Pretrained Speech Language Models (2023, background): “The common approach is to first encode the speech into a continuous representation and then quantize the representation to achieve a sequence of discrete tokens [Tjandra et al., 2019, 2020, Lakhotia et al., 2021, Borsos et al., 2022].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “SSL techniques are also benchmarked with the ZeroSpeech challenge [265], [266].”
Unsupervised Learning of Disentangled Speech Content and Style Representation (2020, background): “[10, 11, 12] leverages vector-quantized VAE [13] (VQVAE) to learn a discrete sequence representation of speech.”