Voice Conversion from Unaligned Corpora Using Variational Autoencoding Wasserstein Generative Adversarial Networks

By Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, Hsin-Min Wang. 2017

326 citations recorded by the source corpus.

Building a voice conversion (VC) system from non-parallel speech corpora is challenging but highly valuable in real application scenarios. In most situations, the source and the target speakers do not repeat the same texts or they may even speak different languages. In this case, one possible, although indirect, solution is to build a generative model for speech. Generative models focus on explaining the observations with latent variables instead of learning a pairwise transformation function, thereby bypassing the requirement of speech frame alignment. In this paper, we propose a non-parallel VC framework with a variational autoencoding Wasserstein generative adversarial network (VAW-GAN) that explicitly considers a VC objective when building the speech model. Experimental results corroborate the capability of our framework for building a VC system from unaligned data, and demonstrate improved conversion quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, background): “GANs are known to be effective for achieving high-quality voice conversion with clean input data [29, 30].”
Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, background): “More recently, speaker adaptation has been tackled with generative adversarial networks (GANs) (Hsu et al., 2017).”
Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, background): “generative adversarial networks [17]–[20] and their combinations [21]–[23].”
How Generative Adversarial Networks and Its Variants Work : An Overview of GAN (2017, methodology): “VAW-GAN [47] is a voice conversion system combining GAN and VAE frameworks.”
An Introduction to Variational Autoencoders (2019, background): “Other hybrids have been proposed as well (Larsen et al., 2016; Brock et al., 2017; Hsu et al., 2017).”
Cycle-Consistent Speech Enhancement (2018, methodology): “In speech area, it has been applied to speech enhancement [24, 25, 26, 27], voice conversion [28, 29], acoustic model adaptation [30, 31, 32], noise-robust [33, 34] and speaker-invariant [35, 36] ASR using gradient reversal layer (GRL) [37].”
TTS Skins: Speaker Conversion via ASR (2019, unknown): “In a follow-up work [25], a WGAN [26] loss term was added, in order to improve the naturalness of the resulting audio.”
CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks (2018, methodology): “A few attempts [40]–[43] have recently been made to develop methods that are completely free from parallel data and extra modules, similarly to ours.”
Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion (2022, methodology): “Due to the ability of SRD to disentangle latent space, there are many other methods [5, 7, 8, 12, 13, 14] based on SRD for VC.”
Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis (2018, methodology): “In [49], [50], this was used to identify matching data frames for non-parallel voice conversion.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “…2020a), while others have implemented autoen-coder models with various constraints applied to the latent representations, such as discrete-ness (van den Oord et al., 2017; Eloff et al., 2019; Chorowski et al., 2019) or disentanglement (Hsu et al., 2017c; Hsu and Glass, 2018b; Khurana et al., 2019).”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “reconstruction of speech in voice conversion studies [8], [9], [11], [12], [47], [48], [85], [86], and continue to play a major role today [17], [21], [22].”
StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks (2018, methodology): “We chose the VAEGAN-based approach [23] as a comparison for our experiments.”