Infergrad: Improving Diffusion Models for Vocoder by Considering Inference in Training

By Zehua Chen, Xu Tan, Ke Wang, Shifeng Pan, Danilo P. Mandic, Lei He, Sheng Zhao. 2022

37 citations recorded by the source corpus.

Denoising diffusion probabilistic models (diffusion models for short) require a large number of iterations in inference to achieve the generation quality that matches or surpasses the state-of-the-art generative models, which invariably results in slow inference speed. Previous approaches aim to optimize the choice of inference schedule over a few iterations to speed up inference. However, this results in reduced generation quality, mainly because the inference process is optimized separately, without jointly optimizing with the training process. In this paper, we propose InferGrad, a diffusion model for vocoder that incorporates inference process into training, to reduce the inference iterations while maintaining high generation quality. More specifically, during training, we generate data from random noise through a reverse process under inference schedules with a few iterations, and impose a loss to minimize the gap between the generated and ground-truth data samples. Then, unlike existing approaches, the training of InferGrad considers the inference process. The advantages of InferGrad are demonstrated through experiments on the LJSpeech dataset showing that InferGrad achieves better voice quality than the baseline WaveGrad under same conditions while maintaining the same voice quality as the baseline but with 3x speedup (2 iterations for InferGrad vs 6 iterations for WaveGrad).

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Zhang
2. Nima Mesgarani
3. Cong Han
4. Shinji Watanabe
5. Yonghui Wu
6. Jinyu Li
7. Heiga Zen
8. Wei Han

In Signal Processing

1. Yu Zhang
2. Nima Mesgarani
3. Cong Han
4. Shinji Watanabe
5. Jinyu Li
6. William Chan
7. Heiga Zen
8. Wei Han

Citations in the corpus

• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29].”
• AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “In the realm of speech or audio synthesis, these models have been explored for both mel-spectrogram generation [31], [47] and waveform generation [48], [49], [50].”
• Noise2Music: Text-conditioned Music Generation with Diffusion Models (2023, background): “Some relevant examples are provided in the context of the text-to-audio task, in which text-conditioned spectrogram generation and spectrogram-conditioned audio has been intensively studied (Popov et al., 2021; Chen et al., 2021; Kong et al., 2021; Wu & Shi, 2021; Chen et al., 2022).”
• A Survey on Generative Diffusion Models (2022, background): “InferGrad[239] improved the diffusion-based text-to-speech model for small inference steps.”
• SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping (2022, background): “Since it worked well regardless of the choice of noise schedule, SpecGrad should be able to enhance the performance of other DDPM-based neural vocoders that use extended noise schedules [25, 26].”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “To reduce the inference iterations while maintaining the generation quality, InferGrad [9] proposes to incorporate the inference process into training with an additional loss.”
• Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “0E-01], respectively, because the output quality was better than the schedules used in the original paper [49].”