Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction

By Yi Zhao, Haoyu Li, Cheng-I Lai, Jennifer Williams, Erica Cooper, Junichi Yamagishi. 2020

21 citations recorded by the source corpus.

Vector Quantized Variational AutoEncoders (VQ-VAE) are a powerful representation learning framework that can discover discrete groups of features from a speech signal without supervision. Until now, the VQ-VAE architecture has previously modeled individual types of speech features, such as only phones or only F0. This paper introduces an important extension to VQ-VAE for learning F0-related suprasegmental information simultaneously along with traditional phone features.The proposed framework uses two encoders such that the F0 trajectory and speech waveform are both input to the system, therefore two separate codebooks are learned. We used a WaveRNN vocoder as the decoder component of VQ-VAE. Our speaker-independent VQ-VAE was trained with raw speech waveforms from multi-speaker Japanese speech databases. Experimental results show that the proposed extension reduces F0 distortion of reconstructed speech for all unseen test speakers, and results in significantly higher preference scores from a listening test. We additionally conducted experiments using single-speaker Mandarin speech to demonstrate advantages of our architecture in another language which relies heavily on F0.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Erica Cooper
2. Junichi Yamagishi
3. Yi Zhao
4. Jennifer Williams
5. Esa Rahtu
6. Vladimir E. Iashin
7. Xu Tan
8. Tao Qin

In Computer Vision and Pattern Recognition

1. Erica Cooper
2. Junichi Yamagishi
3. Yi Zhao
4. Esa Rahtu
5. Vladimir E. Iashin
6. Luisa Verdoliva

In Signal Processing

1. Junichi Yamagishi
2. Erica Cooper
3. Yi Zhao
4. Jennifer Williams
5. Esa Rahtu
6. Vladimir E. Iashin
7. Xin Eric Wang
8. Xu Tan

Citations in the corpus

• Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, neutral): “The study by [23] is the closest to our work.”
• Taming Visually Guided Sound Generation (2021, background): “Applications of VQVAE for content generation include images [51, 61], audio [15, 37, 61, 71], and videos [49, 65].”
• Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm (2020, methodology): “Building upon the open-source implementation provided by [11], we likewise added an encoder and VQ codebook component to the original VQ-VAE architecture.”