A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural $F_0$ Model for Statistical Parametric Speech Synthesis

By Xin Eric Wang, Shinji Takaki, Junichi Yamagishi, Simon King, Keiichi Tokuda. 2020

44 citations recorded by the source corpus.

Recurrent neural networks (RNNs) can predict fundamental frequency (F0) for statistical parametric speech synthesis systems, given linguistic features as input. However, these models assume conditional independence between consecutive <inline-formula><tex-math notation="LaTeX">$F_0$</tex-math></inline-formula> values, given the RNN state. In a previous study, we proposed autoregressive (AR) neural <inline-formula><tex-math notation="LaTeX">$F_0$</tex-math></inline-formula> models to capture the causal dependency of successive <inline-formula><tex-math notation="LaTeX">$F_0$</tex-math></inline-formula> values. In subjective evaluations, a deep AR model (DAR) outperformed an RNN. Here, we propose a Vector Quantized Variational Autoencoder (VQ-VAE) neural <inline-formula><tex-math notation="LaTeX">$F_0$</tex-math></inline-formula> model that is both more efficient and more interpretable than the DAR. This model has two stages: one uses the VQ-VAE framework to learn a latent code for the <inline-formula><tex-math notation="LaTeX">$F_0$</tex-math></inline-formula> contour of each linguistic unit, and other learns to map from linguistic features to latent codes. In contrast to the DAR and RNN, which process the input linguistic features frame-by-frame, the new model converts one linguistic feature vector into one latent code for each linguistic unit. The new model achieves better objective scores than the DAR, has a smaller memory footprint and is computationally faster. Visualization of the latent codes for phones and moras reveals that each latent code represents an <inline-formula><tex-math notation="LaTeX">$F_0$</tex-math></inline-formula> shape for a linguistic unit.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lei He
2. Junichi Yamagishi
3. Sheng Zhao
4. Xu Tan
5. Chenpeng Du
6. Yanqing Liu
7. Yiwei Guo
8. Frank K. Soong

In Signal Processing

1. Junichi Yamagishi
2. Lei He
3. Sheng Zhao
4. Yiwei Guo
5. Chenpeng Du
6. Yanqing Liu
7. Xie Chen
8. K. Yu

Citations in the corpus

• VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature (2022, background): “The effectiveness of VQVAE is also explored in the related literature [17, 18].”
• Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction (2020, methodology): “In this paper, we focus on Vector Quantized Variational AutoEncoder (VQ-VAE) [4] since it is a promising selfsupervised representation learning approach suitable for tasks such as voice conversion (VC) [4, 5], text-to-speech (TTS) synthesis [6, 7, 8], speech coding [9], and even music generation [10”
• FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, background): “The effectiveness of VQVAE is also explored in [65] .”
• Controllable Neural Prosody Synthesis (2020, background): “More recently, deep learning methods such as variational autoencoders (VAEs) [9], deep autoregressive (DAR) neural networks [10], and vector-quantized VAEs (VQVAEs) [11] were shown to be effective at generating F0 contours of speech from text features.”
• Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters (2026, unknown): “…reference audio encodings [1]–[3], explicit pitch and duration targets [28], [29], adversarial style discriminators built on SSL representations [4], large-scale speech-token language modeling [5], prosody-focused self-supervised objectives [30], [31], or discrete latent style spaces [32], [33].”