StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion

By Ali Asghar Zare, Nima Mesgarani. 2021

139 citations recorded by the source corpus.

We present an unsupervised non-parallel many-to-many voice conversion (VC) method using a generative adversarial network (GAN) called StarGAN v2. Using a combination of adversarial source classifier loss and perceptual loss, our model significantly outperforms previous VC models. Although our model is trained only with 20 English speakers, it generalizes to a variety of voice conversion tasks, such as any-to-many, cross-lingual, and singing conversion. Using a style encoder, our framework can also convert plain reading speech into stylistic speech, such as emotional and falsetto speech. Subjective and objective evaluation experiments on a non-parallel many-to-many voice conversion task revealed that our model produces natural sounding voices, close to the sound quality of state-of-the-art text-to-speech (TTS) based voice conversion methods without the need for text labels. Moreover, our model is completely convolutional and with a faster-than-real-time vocoder such as Parallel WaveGAN can perform real-time voice conversion.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “We randomized the model order and kept their labels hidden, similar to the MUSHRA approach [61, 62].”
StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis (2022, unknown): “Application of these methods in speech synthesis has not been extensivelyexploredyet,restrictedprimarilytovoiceconversion andspeakeradaptation[32],[33],[34],[35],[36].”
STYLETTS-VC: ONE-SHOT VOICE CONVERSION BY KNOWLEDGE TRANSFER FROM STYLE-BASED TTS MODELS (2022, methodology): “We use two adversarial objectives: the original cross-entropy loss function for adversarial training following [20] and the additional feature-matching loss following [27]”
PHONEME-LEVEL BERT FOR ENHANCED PROSODY OF TEXT-TO-SPEECH WITH GRAPHEME PREDICTIONS (2023, background): “instructed the subjects to listen and rate them without telling them the model labels [20, 21].”