Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme

By Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, Jiansheng Wei. 2021

193 citations recorded by the source corpus.

Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying the target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to state-of-the-art one-shot voice conversion approaches. Moreover, focusing on real-time applications, we investigate general principles which can make diffusion models faster while keeping synthesis quality at a high level. As a result, we develop a novel Stochastic Differential Equations solver suitable for various diffusion model types and generative tasks as shown through empirical studies and justify it by theoretical analysis.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Navonil Majumder
2. Jun Zhu
3. Soujanya Poria
4. Deepanway Ghosal
5. Chongxuan Li
6. Fan Bao
7. Jiacheng Sun
8. P. Heng

In Signal Processing

1. Navonil Majumder
2. Minki Kang
3. Stan Z. Li
4. Jun Zhu
5. P. Heng
6. Soujanya Poria
7. Zhangyang Gao
8. Linquan Liu

Citations in the corpus

• DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, neutral): “Training-free samplers include adopting implicit [19] or analytical [21] generation process, advanced differential equation (DE) solvers [3, 20, 22, 23, 24] and dynamic programming [18].”
• Grad-StyleSpeech: Any-Speaker Adaptive Text-to-Speech Synthesis with Diffusion Models (2022, methodology): “We use the Maximum Likelihood SDE solver [23] for faster sampling and take 100 denoising steps.”
• EdiTTS: Score-based Editing for Controllable Text-to-Speech (2021, methodology): “1, where j ∈ [1, 9] is the positional difference as defined above.”
• Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, background): “…of applications, such as controllable generation (Choi et al., 2021; Meng et al., 2021; Sinha et al., 2021; Nichol et al., 2021), voice conversion (Popov et al., 2021), image super-resolution (Sa-haria et al., 2021; Li et al., 2021), image-to-image translation (Sasaki et al., 2021), shape…”
• AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models (2023, unknown): “Speech voice conversion [2, 35, 44, 43] and singing voice conversion [7, 28] aim to modify the timbre without changing the speech or singing content.”
• A Survey on Generative Diffusion Models (2022, methodology): “These methods form the foundation for general waveform generation, and additional features like loudness, melody, and phonetic posteriorgram enable controllable style generation [187], [190], [191], [215].”
• Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model (2023, background): “Recent years have seen a surge in diffusion models as a leading approach for generating high-quality speech [2, 16, 27, 28, 10, 8].”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “We selected open-source SOTA methods for comparison, including HierSpeech++ [Lee et al., 2023] and DiffVC [Popov et al., 2021].”