An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep Learning

By Berrak Sisman, Junichi Yamagishi, Simon King, Haizhou Li. 2020

426 citations recorded by the source corpus.

Speaker identity is one of the important characteristics of human speech. In voice conversion, we change the speaker identity from one to another, while keeping the linguistic content unchanged. Voice conversion involves multiple speech processing techniques, such as speech analysis, spectral conversion, prosody conversion, speaker characterization, and vocoding. With the recent advances in theory and practice, we are now able to produce human-like voice quality with high speaker similarity. In this article, we provide a comprehensive overview of the state-of-the-art of voice conversion techniques and their performance evaluation methods from the statistical approaches to deep learning, and discuss their promise and limitations. We will also report the recent Voice Conversion Challenges (VCC), the performance of the current state of technology, and provide a summary of the available resources for voice conversion research.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Wei-Ning Hsu
5. Frank K. Soong
6. Yossi Adi
7. Junichi Yamagishi
8. Emmanuel Dupoux

In Signal Processing

1. Xu Tan
2. Wei-Ning Hsu
3. Tao Qin
4. Yossi Adi
5. Tie-Yan Liu
6. Frank K. Soong
7. Junichi Yamagishi
8. Haizhou Li

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Due to page limitations, we only reviewed core algorithms of TTS; readers can refer to other papers for TTS related problems and applications, such as voice conversion [302], singing voice synthesis [112, 211, 35], talking face synthesis [36], etc.”
• Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “Another line of works consists of modifying the parameters of the speech signal (Polyak et al., 2020b,a; Sisman et al., 2020; Robinson et al., 2019; Zhou et al., 2021b), i.”
• Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “VC has various applications in real life including expressive voice synthesis, personalized speech speaking assistance, vocally impaired people, voice dubbing for the entertainment industry, and many others [185].”
• Creating, Using, Misusing, and Detecting Deep Fakes (2022, background): “VC predates deep fakes by 65 years, although recent efforts have incorporated deep learning (Sisman et al. 2020).”
• Many-to-Many Voice Transformer Network (2020, background): “Interested readers are referred to an excellent review article [26] on the recent trends in VC research.”
• Graphspeech: Syntax-Aware Graph Attention Network for Neural Speech Synthesis (2020, background): “The recent advances have enabled many applications [3] such as smart voice assistants, dubbing of movies and games, online education, and smart home.”
• Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion (2022, background): “Voice conversion (VC) is a speech task that studies how to convert one’s voice to sound like that of another while preserving the linguistic content of the source speaker [1, 2].”
• Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “For example, they are used as the backbone module in text-to-speech (TTS) [5–10], voice conversion [11, 12], speech-to-speech translation (S2ST) [13–15], speech enhancement (SE) [16–19], speech restoration [20, 21], and speech coding [22–25].”
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, background): “In a recent VC review paper [48], it was shown that a sufficient amount of efforts has been dedicated to transferring knowledge from ASR and TTS to improving various aspects of VC, regardless of using a seq2seq model or not.”
• An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era (2022, background): “Two recent comprehensive reviews of VC can be found in [85] and [86].”
• Expressive TTS Training With Frame and Style Reconstruction Loss (2020, methodology): “One is the Griffin Lim [44] algorithm, another is via a neural vocoder [5], [45], [46], [50].”