Zero-Shot Voice Style Transfer with Only Autoencoder Loss

By Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, Mark Hasegawa-Johnson. 2019

555 citations recorded by the source corpus.

Non-parallel many-to-many voice conversion, as well as zero-shot voice conversion, remain under-explored areas. Deep style transfer algorithms, such as generative adversarial networks (GAN) and conditional variational autoencoder (CVAE), are being applied as new solutions in this field. However, GAN training is sophisticated and difficult, and there is no strong evidence that its generated speech is of good perceptual quality. On the other hand, CVAE training is simple but does not come with the distribution-matching property of a GAN. In this paper, we propose a new style transfer scheme that involves only an autoencoder with a carefully designed bottleneck. We formally show that this scheme can achieve distribution-matching style transfer by training only on a self-reconstruction loss. Based on this scheme, we proposed AUTOVC, which achieves state-of-the-art results in many-to-many voice conversion with non-parallel data, and which is the first to perform zero-shot voice conversion.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Zhou Zhao
7. Lei He
8. Yi Ren

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Sheng Zhao
4. Lei He
5. Frank K. Soong
6. Tie-Yan Liu
7. Rongjie Huang
8. Yi Ren

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “…timbre, and acoustic details and reconstruct speech waveform with these disentangled representations, leveraging information bottleneck [16, 17], various supervised losses, and adversarial training [18] to enhance disentanglement; 2) we propose a factorized diffusion model, which generates…”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “…into content, prosody, and timbre representations with the reconstruction loss of the autoencoder and a carefully designed bottleneck [44]: 1) we feed the mel-spectrogram into the prosody encoder, and we also introduce carefully-tuned dimension reduction and phoneme-level…”
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, neutral): “For a deeper analysis, following [46], we compared the transfer between male, female and mixed gender speakers individually.”
• Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, methodology): “[36] proposed to perform zero-shot voice conversion by utilizing the pre-trained speaker recognition network and information bottleneck by carefully designing the bottleneck of an auto-encoder.”
• ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, methodology): “A mel-spectrogram is widely used as an intermediate representation in recent TTS [1, 2, 3, 4, 5] and VC [6, 7, 8] systems owing to its compactness and expressiveness.”
• Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme (2021, methodology): “Whereas in the pioneering AutoVC model (Qian et al., 2019) only speaker embedding from the pre-trained speaker verification network was used as conditioning, several other models improved on AutoVC enriching conditioning with phonetic features such as pitch and loudness (Qian et al., 2020;…”
• STYLETTS-VC: ONE-SHOT VOICE CONVERSION BY KNOWLEDGE TRANSFER FROM STYLE-BASED TTS MODELS (2022, background): “To convert an unseen speaker’s voice into another speaker’s voice unseen during training, the model needs to learn a shared representation of speech across all potential sources and target speakers [1].”
• Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, methodology): “This includes the model using auto-encoders [15], variational auto-encoders [16],”
• MakeItTalk: Speaker-Aware Talking Head Animation (2020, methodology): “To extract the speaker-agnostic content representation of the audio, we use AutoVC encoder from Qian et al. [2019].”
• WG-WaveNet: Real-Time High-Fidelity Speech Synthesis without GPU (2020, methodology): “Recently, neural network-based models have achieved state-ofthe-art performance in speech tasks such as text-to-speech and voice conversion [1, 2, 3, 4].”
• StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion (2021, methodology): “AUTO-VC is trained with one-hot embedding for 1M steps.”
• CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech (2020, background): “Therefore, we introduced a temporal bottleneck [20], which reduces the amount of information flowing from the reference encoder to the decoder along the time”
• Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Inspired by the voice conversion model AutoVC [165], Qian et al.”
• Deep Learning for Text Style Transfer: A Survey (2020, background): “…different modalities, including images (Gatys, Ecker, and Bethge 2016; Zhu et al. 2017; Chen et al. 2017b), text, voice (Gao, Singh, and Raj 2018; Qian et al. 2019; Yuan et al. 2021), handwriting (Azadi et al. 2018; Zhang and Liu 2013), and videos (Ruder, Dosovitskiy, and Brox 2016; Chen et al.…”
• Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “[230] Encoder-decoder speech spectrogram ▪ VCTK corpus ▪ Prosody flipping between the source and the target.”
• Enhancing the Stability of LLM-based Speech Generation Systems through Self-Supervised Representations (2024, background): “…speaker and non-speaker information is a challenge and there are multiple proposals in the literature: a bottleneck based on information theory by (Qian et al., 2019), a variational autoencoder (Saito et al., 2018), generative adversarial networks (GAN) by (Kaneko et al., 2019; Kameoka et al.,…”
• PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, background): “Given that it is impossible to comprehensively describe speech with fine-grained details (Yang et al., 2022; Qian et al., 2019; 2020) using text prompts alone, there exists the one-to-many problem in the text prompt based TTS system.”
• Towards Robust Neural Vocoding for Speech Generation: A Survey (2019, background): “Most speech generation models, such as text-to-speech [1, 2, 3, 4] and voice conversion [5, 6, 7, 8], do not output waveform directly.”