Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion

By Sicheng Yang, Methawee Tantrawenith, Hao-Wen Zhuang, Zhiyong Wu, Aolan Sun, Jianzong Wang, Ning Cheng, Huaizhen Tang, Xintao Zhao, Jie Wang and 1 more. 2022

45 citations recorded by the source corpus.

One-shot voice conversion (VC) with only a single target speaker's speech for reference has become a hot research topic. Existing works generally disentangle timbre, while information about pitch, rhythm and content is still mixed together. To perform one-shot VC effectively with further disentangling these speech components, we employ random resampling for pitch and content encoder and use the variational contrastive log-ratio upper bound of mutual information and gradient reversal layer based adversarial mutual information learning to ensure the different parts of the latent space containing only the desired disentangled representation during training. Experiments on the VCTK dataset show the model achieves state-of-the-art performance for one-shot VC in terms of naturalness and intellgibility. In addition, we can transfer characteristics of one-shot VC on timbre, pitch and rhythm separately by speech representation disentanglement. Our code, pre-trained models and demo are available at https://im1eon.github.io/IS2022-SRDVC/.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Lei He
3. Sheng Zhao
4. Tao Qin
5. Yanqing Liu
6. Tie-Yan Liu
7. Jinyu Li
8. Yichong Leng

In Signal Processing

1. Xu Tan
2. Lei He
3. Sheng Zhao
4. Yanqing Liu
5. Jinyu Li
6. Yichong Leng
7. Jiang Bian
8. Zeqian Ju

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, methodology): “Inspired by [47], we adopt adversarial classifier with the gradient reversal layer (GRL) [48] to eliminate undesired information in latent space.”
• PromptTTS 2: Describing and Generating Voices with Text Prompt (2023, background): “Given that it is impossible to comprehensively describe speech with fine-grained details (Yang et al., 2022; Qian et al., 2019; 2020) using text prompts alone, there exists the one-to-many problem in the text prompt based TTS system.”