StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks

By Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, Nobukatsu Hojo. 2018

419 citations recorded by the source corpus.

This paper proposes a method that allows non-parallel many-to-many voice conversion (VC) by using a variant of a generative adversarial network (GAN) called StarGAN. Our method, which we call StarGAN-VC, is noteworthy in that it (1) requires no parallel utterances, transcriptions, or time alignment procedures for speech generator training, (2) simultaneously learns many-to-many mappings across different attribute domains using a single generator network, (3) is able to generate converted speech signals quickly enough to allow real-time implementations and (4) requires only several minutes of training examples to generate reasonably realistic sounding speech. Subjective evaluation experiments on a non-parallel many-to-many speaker identity conversion task revealed that the proposed method obtained higher sound quality and speaker similarity than a state-of-the-art method based on variational autoencoding GANs.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, background): “Voice conversion [27, 39], emotion conversion [53, 34], speech enhancement [63, 11, 55] belong to this category.”
Emotional Voice Conversion: Theory, Databases and ESD (2021, background): “Recently, voice conversion techniques with non-parallel training data have been studied, for example, domain translation (Kaneko and Kameoka, 2017; Kameoka et al., 2018), multitask learning (Zhang et al., 2019a) and speaker disentanglement (Hsu et al., 2016, 2017; Tanaka et al., 2019) are among the…”
Unsupervised Speech Decomposition via Triple Information Bottleneck (2020, methodology): “ACVAE-VC (Kameoka et al., 2018a) introduces an auxiliary classifer for the conversion outputs, and encourages the converted speech to be correctly classifed as the source speaker.”
Didispeech: A Large Scale Mandarin Speech Corpus (2020, background): “The focus of speech synthesis is changing from synthesizing human-like speech of a given target speaker to more complex tasks, such as voice conversion (VC) [14, 15] and multi-speaker TTS [16].”
Towards Robust Neural Vocoding for Speech Generation: A Survey (2019, background): “Most speech generation models, such as text-to-speech [1, 2, 3, 4] and voice conversion [5, 6, 7, 8], do not output waveform directly.”
Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, background): “…theory by (Qian et al., 2019), a variational autoencoder (Saito et al., 2018), generative adversarial networks (GAN) by (Kaneko et al., 2019; Kameoka et al., 2018), normalizing flows (Merritt et al., 2022) and self-supervised speech representations (S3Rs) (Li et al., 2023; Lin et al., 2021;…”
An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, methodology): “However, despite several attempts, no notable progress was made until recent years, which saw the advent of DNNs. Recently proposed methods exploit the representation power of DNNs by means of VAEs [92], [93], GANs [94], [95], and seq2seq models [96], [97].”
Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, methodology): “This includes the model using auto-encoders [15], variational auto-encoders [16], generative adversarial networks [17–20] and their combinations [21–23].”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, methodology): “Accuracy=93.42% ▪ StarGAN Yu et al. [163] CNN Deep features Accuracy=99.43% CelebA ▪ Poor performance on post-processing operations.”
How Far Are We from Robust Voice Conversion: A Survey (2020, background): “StarGANVC [7] adopts conditional input to achieve many-to-many speaker voice conversion.”
NoiseVC: Towards High Quality Zero-Shot Voice Conversion (2021, background): “Generative models like variational autoencoder (VAE) [8, 9], generative adversarial networks (GAN) [10, 11] or Flow [12] are actively explored for the VC task as well.”
Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach (2018, background): “Recent studies have shown remarkable success in multiple domain transfer (MDT) [ 6 , 13 , 20 , 21 ] though not in the context of the image classiﬁcation, rather in the context of image generation. choi et al. [ 13 ] proposed StarGAN , a generative adversarial network capable of learning mappings…”
Review of end-to-end speech synthesis technology based on deep learning (2021, background): “Voice conversion (VC) model can disentangle the speaker-dependent timbre feature from speech [33, 34, 91, 165, 183], but cannot extract other style features such as the content, pitch and rhythm of speech.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Voice conversion has recently seen progress using neural approaches [45, 49, 50, 46, 27, 12, 14, 69, 56, 87], but the most relevant work to our own is the ZeroSpeech 2019 challenge [24, 94, 10], which explicitly addresses the unsupervised learning of discrete speech units which can be used as the ba”
An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning (2020, background): “[47], CycleGAN [48]–[50], and many-to-many mapping with StarGAN [51] further advance the state-of-the-art.”
Zero-Shot Voice Style Transfer with Only Autoencoder Loss (2019, methodology): “However, CVAE alone does not guarantee distribution matching, and often suffers from over-smoothing of the conversion output (Kameoka et al., 2018b).”
NAUTILUS: A Versatile Voice Cloning System (2020, positive): “[32] used generative adversarial network (GAN) to train”