Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

By Fan Bao, Chongxuan Li, Jun Zhu. 2022

419 citations recorded by the source corpus.

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20x to 80x speed up.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In General Social Sciences

In Generative Models

Citations in the corpus

Elucidating the Design Space of Diffusion-Based Generative Models (2022, neutral): “Previous results [3, 24, 46, 48] suggest that non-zero β ( t ) is helpful, but as far as we can tell, the implicit choice for β ( t ) in Song et al. [48] enjoys no special properties.”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, methodology): “However, these methods still require ∼ 50 function evaluations [21] to generate high-quality samples (comparable to those generated by plain samplers in about 1000 function evaluations), thereby are still time-consuming.”
Improving and generalizing flow-based generative models with minibatch optimal transport (2023, neutral): “• FM and CFM are equivalently computationally efficient per iteration and OT-CFM comes with a low ( < 1”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “For example, OPT (Zhang et al., 2022), BLOOM (Scao et al., 2022), GLM (Zeng et al., 2023), LLaMA (Touvron et al., 2023a;b), and Falcon (Penedo et al., 2023) all raised great attention and are been widely deployed.”
A Survey on Generative Diffusion Models (2022, methodology): “Analytic-DPM [82] and extended Analytic-DPM [83] jointly propose optimal reverse solutions under correction for each state.”
Post-Training Quantization on Diffusion Models (2022, positive): “Previously DM acceleration methods only focus on the former [2, 4, 18, 24, 29, 40], but overlook the latter.”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, neutral): “Denoising learning mechanisms [27, 44] and improvements in sampling efficiency [4, 40, 41, 55, 56] have been continuously optimized to generate high-quality images.”
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, neutral): “Intrinsically, this is because such models characterize the joint distribution by the chain rule of probability, motivating research on developing other types of generative models for text.”
Subspace Diffusion Generative Models (2022, background): “Other works [13, 3, 10, 22, 17, 11, 2] focus on reducing the number of steps in the discrete-time Markov chain formulation.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[121] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, background): “, 2020a) and DPM-Solver (Lu et al., 2022a;b; Bao et al., 2022) are often adopted to speed up the sampling process of diffusion models.”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, neutral): “…either discretizing diffusion SDEs (Ho et al., 2020; Song et al., 2021b; Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) or discretizing diffusion ODEs (Song et al., 2021b;a; Liu et al., 2022b; Zhang & Chen, 2022; Lu et al., 2022).”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “AnalyticDPM [124] provides more efficient inference by estimating the analytic form of optimal model reverse variance and KL-divergence w.”
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, methodology): “As shown in Table 10, our method still outperforms the strong baseline Analytic-DPM.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “[121] 2021 Discrete Reverse Variance Learning Reverse Variance Learning Bao et al.[8] 2021 Discrete Nichol et al.”
A Continuous Time Framework for Discrete Denoising Models (2022, background): “[32] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.”