Voice conversion from non-parallel corpora using variational auto-encoder

By Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, Hsin-Min Wang. Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 2016

319 citations recorded by the source corpus.

We propose a flexible framework for spectral conversion (SC) that facilitates training with unaligned corpora. Many SC frameworks require parallel corpora, phonetic alignments, or explicit frame-wise correspondence for learning conversion functions or for synthesizing a target spectrum with the aid of alignments. However, these requirements gravely limit the scope of practical applications of SC due to scarcity or even unavailability of parallel corpora. We propose an SC framework based on variational auto-encoder which enables us to exploit non-parallel corpora. The framework comprises an encoder that learns speaker-independent phonetic representations and a decoder that learns to reconstruct the designated speaker. It removes the requirement of parallel corpora or phonetic alignments to train a spectral conversion system. We report objective and subjective evaluations to validate our proposed method and compare it to SC methods that have access to aligned corpora.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/5ced178c1065c26a83afc2734ec8dba0f58f8cec

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Junichi Yamagishi
2. Hung-yi Lee
3. James R. Glass
4. Wei-Ning Hsu
5. Haizhou Li
6. Xin Eric Wang
7. Xu Tan
8. Tao Qin

Citations in the corpus

• Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations (2018, methodology): “With VAEs, the encoder learns the speaker-independent linguistic content, which can then be used by the de-coder to generate the voice with a specified speaker id [22, 23, 24].”
• An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, methodology): “The decoder reconstructs the utterance by conditioning on the latent code extracted by the encoder, and separately on a speaker code, which could be an one-hot vector [43], [237] for a close set of speakers, or an i-vector [159], bottleneck speaker representation [238], or d-vector [239] for an open”
• Emotional Voice Conversion: Theory, Databases and ESD (2021, background): “Studies have shown the effectiveness of auto-encoder (Hsu et al., 2016) and its variants (Hsu et al.”
• Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder (2018, background): “Several recent studies have proposed using VAEs for modeling speech [20, 21, 14, 22, 23].”
• Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion (2019, background): “…also attempted to encode speech content into various representations for voice conversion, including the use of disentangled autoencoders [11], VAEs [12] or GANs [13]. continuous vectors are the most common approach, however when it comes to encoding speech content, they may not be the best choice.”
• An Introduction to Variational Autoencoders (2019, background): “conversion from non-parallel corpora (Hsu et al., 2016), discriminationaware (fair) representations (Louizos et al.”
• How Far Are We from Robust Voice Conversion: A Survey (2020, background): “Recently, several models were proposed to utilize non-parallel data [1, 2, 3, 4, 5].”
• Unsupervised Singing Voice Conversion (2019, background): “In [10], the notion of a single encoder and a parameterized decoder, where the parameters represent the identity, was introduced.”
• Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion (2018, methodology): “There are 30 hidden layers in our WaveNet Vocoder, and dilations are [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]× 3 respectively.”
• Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet (2019, methodology): “To make use of non-parallel source-target speech corpora, voice conversion based on the variational autoencoder (VAE) [13] and cycleconsistent generative adversarial network [14, 15, 16, 17] were proposed.”
• Learning Latent Representations for Speech Generation and Transformation (2017, background): “While there are many works investigating generative models for natural images [12, 13], little work has been done on learning speech generation with deep probabilistic generative models [14, 15].”
• NAUTILUS: A Versatile Voice Cloning System (2020, methodology): “[35] used variational auto-encoder (VAE), while Kameoka et al.”
• StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks (2018, methodology): “To overcome the shortcomings and limitations of CVAEVC [22] and CycleGAN-VC [26], this paper proposes a nonparallel many-to-many VC method using a recently proposed novel GAN variant called StarGAN [42], which offers the advantages of CVAE-VC and CycleGAN-VC concurrently.”
• Voice Conversion from Unaligned Corpora Using Variational Autoencoding Wasserstein Generative Adversarial Networks (2017, methodology): “Recent works have proven the viability of speech modeling with VAEs [3, 4].”
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis (2018, methodology): “In [49], [50], this was used to identify matching data frames for non-parallel voice conversion.”
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, methodology): “However, despite several attempts, no notable progress was made until recent years, which saw the advent of DNNs. Recently proposed methods exploit the representation power of DNNs by means of VAEs [92], [93], GANs [94], [95], and seq2seq models [96], [97].”
• TTS Skins: Speaker Conversion via ASR (2019, background): “[24], employs, similarly to us, one encoder and a parameterized decoder, where the parameters represent the identity.”
• Virtual Phone Discovery for Speech Synthesis Without Text (2019, methodology): “Recently, techniques have been developed without the use of parallel data for voice conversion [16, 17, 18, 19].”
• ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech (2019, methodology): “The VAE makes it possible to achieve non-parallel VC based on an unsupervised factorization of speech spectral features via autoencoding.”