Bilateral Denoising Diffusion Models

By Max W. Y. Lam, Jun Wang, Rongjie Huang, Dan Su, Dong Yu. 2021

47 citations recorded by the source corpus.

Denoising diffusion probabilistic models (DDPMs) have emerged as competitive generative models yet brought challenges to efficient sampling. In this paper, we propose novel bilateral denoising diffusion models (BDDMs), which take significantly fewer steps to generate high-quality samples. From a bilateral modeling objective, BDDMs parameterize the forward and reverse processes with a score network and a scheduling network, respectively. We show that a new lower bound tighter than the standard evidence lower bound can be derived as a surrogate objective for training the two networks. In particular, BDDMs are efficient, simple-to-train, and capable of further improving any pre-trained DDPM by optimizing the inference noise schedules. Our experiments demonstrated that BDDMs can generate high-fidelity samples with as few as 3 sampling steps and produce comparable or even higher quality samples than DDPMs using 1000 steps with only 16 sampling steps (a 62x speedup).

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Signal Processing

Citations in the corpus

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, background): “With these important improvements, DPMs show great potential in various applications, including speech synthesis (Chen et al., 2020; Kong et al., 2020; Popov et al., 2021; Lam et al., 2021), controllable generation (Choi et al.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Recent studies have improved DPMs further in both theory and practice [25,29,31].”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Samplers that need extra training or optimization include knowledge distillation [13, 14], learning the noise level or variance [15, 16, 33], and learning the noise schedule or sample trajectory [17, 18].”
ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech (2022, methodology): “Diffusion process Similar as previouswork [8, 22, 43], we define the data distribution as q(x0).”
Pseudo Numerical Methods for Diffusion Models on Manifolds (2022, methodology): “Some works apply DDPMs to various types of data successfully, including text-to-speech (Chen et al., 2020; Lam et al., 2021), singing voice (Liu et al., 2021), 3D Point Cloud (Luo & Hu, 2021), text generation (Austin et al., 2021).”
Bespoke Solvers for Generative Flow Models (2023, methodology): “…JVP for their method; Watson et al. (2021) (DDSS) optimized a perceptual loss considering a family of generalized Gaussian diffusion models; Lam et al. (2021) improved the denoising process using bilateral filters, thereby indirectly affecting the efficiency of the ODE solver; Duan et al.…”
Diffusion Models in Vision: A Survey (2022, background): “[90] propose to learn the noise scheduling for sampling.”
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “There has been an increasing interest in self-supervised learning in the machine learning (Zhang et al., 2022a; Lam et al., 2021; Zhang et al., 2023b; 2022b) and multimodal processing community (Xia et al., 2022; Zhang et al., 2023a; Zhang et al.; Huang et al., 2022b;a).”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, unknown): “Samplers based on extra training include model distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021), learning reverse process variances (San-Roman et al., 2021; Nichol & Dhariwal, 2021; Bao et al., 2022a), and learning sampling steps (Lam et al., 2021; Watson et al., 2022).”
Infergrad: Improving Diffusion Models for Vocoder by Considering Inference in Training (2022, unknown): “In BDDMs [21], they also trained an extra scheduling network that was used to optimize the choice of inference schedule.”
GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “Recently, non-autoregressive TTS models [23, 12, 18, 17] have been proposed and significantly speed up mel-spectrogram generation, and researchers have experimented on style modeling and transferring in parallel: Meta-StyleSpeech [31] generally adopts a speech encoding network for multi-speaker TTS ”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, methodology): “In the realm of speech or audio synthesis, these models have been explored for both mel-spectrogram generation [31], [47] and waveform generation [48], [49], [50].”
Diffusion Recommender Model (2023, background): “Approaches along this line usually focus on developing noise scheduling [1, 18], estimation object (e.”