High-Quality Nonparallel Voice Conversion Based on Cycle-Consistent Adversarial Network

By Fuming Fang, Junichi Yamagishi, Isao Echizen, Jaime Lorenzo-Trueba. 2018

144 citations recorded by the source corpus.

Although voice conversion (VC) algorithms have achieved remarkable success along with the development of machine learning, superior performance is still difficult to achieve when using nonparallel data. In this paper, we propose using a cycle-consistent adversarial network (CycleGAN) for nonparallel data-based VC training. A CycleGAN is a generative adversarial network (GAN) originally developed for unpaired image-to-image translation. A subjective evaluation of inter-gender conversion demonstrated that the proposed method significantly outperformed a method based on the Merlin open source neural network speech synthesis system (a parallel VC system adapted for our setup) and a GAN-based parallel VC system. This is the first research to show that the performance of a nonparallel VC method can exceed that of state-of-the-art parallel VC methods.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Speech & Audio

In Signal Processing

Citations in the corpus

VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, background): “GANs are known to be effective for achieving high-quality voice conversion with clean input data [29, 30].”
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications (2020, background): “GANs also have achievements in sequential data such as natural language, music, speech, voice [376], [377], and time series [378]–[381].”
Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNet (2019, methodology): “To make use of non-parallel source-target speech corpora, voice conversion based on the variational autoencoder (VAE) [13] and cycleconsistent generative adversarial network [14, 15, 16, 17] were proposed.”
Zero-Shot Voice Style Transfer with Only Autoencoder Loss (2019, methodology): “Similarly, StarGAN (Kaneko & Kameoka, 2017) and CycleGAN (Zhu et al., 2017), which consist of encoder-decoder architectures with GAN, are applied to voice conversion (Kameoka et al., 2018b; Fang et al., 2018).”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, methodology): “CycleGAN has been successfully applied in mono-lingual [48], [203], cross-lingual voice conversion [204], emotional voice con-”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Voice conversion has recently seen progress using neural approaches [45, 49, 50, 46, 27, 12, 14, 69, 56, 87], but the most relevant work to our own is the ZeroSpeech 2019 challenge [24, 94, 10], which explicitly addresses the unsupervised learning of discrete speech units which can be used as the ba”
Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, methodology): “This includes the model using auto-encoders [15], variational auto-encoders [16], generative adversarial networks [17–20] and their combinations [21–23].”
Emotional Voice Conversion: Theory, Databases and ESD (2021, background): “he cycle-consistency loss eliminates the need for parallel training ata through circular conversion, and the identity-mapping loss helps reserve the linguistic content between the input and output (Kaneko nd Kameoka, 2017; Fang et al., 2018; Kaneko and Kameoka, 2018).”
Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion (2022, methodology): “[13] T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo, “CycleGANVC3: Examining and Improving CycleGAN-VCs for MelSpectrogram Conversion,” in Proc.”