DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

By Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Zhou Zhao. 2021

362 citations recorded by the source corpus.

Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing. In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs. To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively. The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger.

Subfields: Artificial Intelligence, Generative Models, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Zhou Zhao
4. Tie-Yan Liu
5. Yi Ren
6. Rongjie Huang
7. Jinglin Liu
8. Jinyu Li

In Generative Models

1. Zhou Zhao
2. Yi Ren
3. Rongjie Huang
4. Jinglin Liu
5. Jia-Bin Huang
6. Ming-Hsuan Yang
7. K. Yu
8. Zhijun Liu

In Speech & Audio

1. Xu Tan
2. Yi Ren
3. Zhou Zhao
4. Jinglin Liu
5. Rongjie Huang
6. Jinyu Li
7. Sheng Zhao
8. Chenpeng Du

In Signal Processing

1. Xu Tan
2. Zhou Zhao
3. Yi Ren
4. Rongjie Huang
5. Jinglin Liu
6. Jinyu Li
7. Chenpeng Du
8. K. Yu

Citations in the corpus

• NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (2024, background): “Generation Methods in TTS. Previous works have demonstrated that NAR-based models [3, 33, 34, 7, 5, 11] enjoy better robustness and generation speed than AR-based models, because they explicitly model the duration and predict all features simultaneously.”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “…2023b) noisy speech speech Target Speech Extraction (TSE) ♣ (Wang et al., 2018) mixed speech, speaker prompt speech Singing Voice Synthesis (SVS) (Liu et al., 2022) phoneme (with duration), speaker prompt, MIDI singing Text-to-Sound (TT-Sound) (Yang et al., 2023c) textual description sounds…”
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding (2023, methodology): “Additional techniques, such as normalizing flow (Valle et al. 2020; Kim et al. 2020) and diffusion models (Liu et al. 2022; Popov et al. 2021; Liu, Guo, and Yu 2023), have been introduced to generate the mel-spectrogram.”
• Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, methodology): “Text-to-speech models usually generate mel-spectrogram from text [59, 2, 35, 48, 29, 47, 36, 22] and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [41, 31, 62, 20], or directly generate waveform from text in an end-to-end manner [45, 15, 30”
• Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024, neutral): “We compare the generated audio samples with other systems, including 1) GT, the ground-truth audio; 2) YourTTS (Casanova et al., 2022), Gen-erSpeech (Huang et al., 2022b), VALL-E (Wang et al., 2023a) for English zero-shot TTS; 3) YourTTS (Casanova et al., 2022) for zero-shot multilingual TTS; 4) NAN”
• ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “For sampling, we use 64 steps in Grad-TTS and 128 steps in DiffSpeech, respectively, following [13] and [29]. denoising diffusion probabilistic models.”
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, neutral): “L G ] 6 M a y 2023 Preprint DDIM (order = 1) (Song et al., 2021a) DPM-Solver (order = 2) (Lu et al., 2022) PNDM (order = 2) (Liu et al., 2022b) DPM-Solver-3 (order = 3) (Lu et al., 2022) DEIS-1 (order = 2) (Zhang & Chen, 2022) † DDIM (thresholding) (Saharia et al., 2022b) DEIS-2 (order = 3) (Zhang…”
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “…processing and generation, replicating this success for audio modality (speech (Ren et al., 2020; Huang et al., 2022a), music (Huang et al., 2021; Liu et al., 2022a), sound (Yang et al., 2022; Huang et al., 2023a), and talking head (Wu et al., 2021; Ye et al., 2023)) is limited, while it is…”
• Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, background): “While several open-source SVS databases designed for English [29, 30] or other languages [31] have been released and also lots of research has been conducted towards Mandarin singing voice synthesis [24, 32, 33], no accessible public Mandarin SVS corpus with high quality exists, thereby limiting the”
• DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, background): “In several studies [1], [2], [3], [4], [5], [6], [7], prosody modeling has been leveraged to synthesize speech that closely resembles human expression.”
• PortaSpeech: Portable and High-Quality Generative Text-to-Speech (2021, background): “1 Introduction Recently, deep learning-based text-to-speech (TTS) has attracted a lot of attention in speech community [2, 15, 17, 21, 23, 25, 26, 30, 36].”
• Pseudo Numerical Methods for Diffusion Models on Manifolds (2022, methodology): “Some works apply DDPMs to various types of data successfully, including text-to-speech (Chen et al., 2020; Lam et al., 2021), singing voice (Liu et al., 2021), 3D Point Cloud (Luo & Hu, 2021), text generation (Austin et al., 2021).”
• SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, methodology): “The mel-spectrogram is processed with a convolutional network, and is used as an additional input to the DPM denoising network (cid:15) [6, 24, 30].”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “These TTS models have been extended to more complex scenarios, including multiple speakers, emotions, and styles for expressive and diverse voice generation [14, 30, 4].”
• SingGAN: Generative Adversarial Network For High-Fidelity Singing Voice Generation (2021, methodology): “DiffSinger [18] proposes a conditional denoising diffusion probabilisticmodel (DDPM) and generates high-fidelity samples via iterative refinement.”