Learning to Efficiently Sample from Diffusion Probabilistic Models

By Daniel Watson, Jonathan Ho, Mohammad Norouzi, William Chan. 2021

163 citations recorded by the source corpus.

Denoising Diffusion Probabilistic Models (DDPMs) have emerged as a powerful family of generative models that can yield high-fidelity samples and competitive log-likelihoods across a range of domains, including image and speech synthesis. Key advantages of DDPMs include ease of training, in contrast to generative adversarial networks, and speed of generation, in contrast to autoregressive models. However, DDPMs typically require hundreds-to-thousands of steps to generate a high fidelity sample, making them prohibitively expensive for high dimensional problems. Fortunately, DDPMs allow trading generation speed for sample quality through adjusting the number of refinement steps as a post process. Prior work has been successful in improving generation speed through handcrafting the time schedule by trial and error. We instead view the selection of the inference time schedules as an optimization problem, and introduce an exact dynamic programming algorithm that finds the optimal discrete time schedules for any pre-trained DDPM. Our method exploits the fact that ELBO can be decomposed into separate KL terms, and given any computation budget, discovers the time schedule that maximizes the training ELBO exactly. Our method is efficient, has no hyper-parameters of its own, and can be applied to any pre-trained DDPM with no retraining. We discover inference time schedules requiring as few as 32 refinement steps, while sacrificing less than 0.1 bits per dimension compared to the default 4,000 steps used on ImageNet 64x64 [Ho et al., 2020; Nichol and Dhariwal, 2021].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Efficiency & Compression

Citations in the corpus

Image Super-Resolution via Iterative Refinement (2021, background): “That said, recent and concurrent work proposes alternative approaches that can result in higher quality fast samplers for diffusionmodels [73], [74], [75], [76].”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, methodology): “Moreover, a DPM (including Analytic-DPM) with OT does not necessarily lead to a better FID score (Watson et al., 2021) (see Appendix G.5 for a comparison of ET and OT in Analytic-DPM).”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “In other words, the training procedure of D θ should not dictate σ ( t ) , s ( t ) , and { t i } , nor vice versa; from the viewpoint of the sampler, D θ is simply a black box [52, 53].”
A Survey on Neural Speech Synthesis (2021, background): “Thus, a lot of works on diffusion models [307, 181, 378, 171] are investigating how to reduce inference time while maintaining generation quality.”
Palette: Image-to-Image Diffusion Models (2021, methodology): “This is an active area of research, and several new techniques [Jolicoeur-Martineau et al. 2021; Nichol and Dhariwal 2021; Watson et al. 2021] have been proposed to reduce the number of refinement steps significantly.”
Subspace Diffusion Generative Models (2022, background): “Other works [13, 3, 10, 22, 17, 11, 2] focus on reducing the number of steps in the discrete-time Markov chain formulation.”
A Survey on Generative Diffusion Models (2022, methodology): “This section highlights four recent developments for enhancing diffusion models: 1) Sampling Acceleration techniques (Section III-A) to speed up the standard ODE/SDE simulation; 2) New Forward Processes(SectionIII-B)forimprovedBrownianmotioninpixel space;3)LikelihoodOptimizationtechniques(SectionIII”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, methodology): “Recently, techniques to accelerate sampling from pre-trained SGMs have been proposed (San-Roman et al., 2021; Watson et al., 2021; Kong & Ping, 2021; Song et al., 2021a).”
Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models (2022, methodology): “Several works explore to ﬁnd short trajectories by applying searching algorithms, e.g., grid search (Chen et al., 2020), dynamic programming (Watson et al., 2021) and differentiable search (Anonymous, 2022a).”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[115] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan.”
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality (2022, background): “Watson et al. (2021) proposed a dynamic programming algorithm that chooses log-likelihood optimal strides, but find that log-likelihood reduction has a mismatch with FID scores, particularly with in the very few step regime, also falling behind DDIM(η = 0) in this front.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “There are also works [125, 126] which directly figure out optimal sampling trajectories via dynamic programming.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “A simple approach is to skip steps at inference: cosine schedules [162] spend more time where larger visual changes are made reducing the impact of skipping; another approach is to use dynamic programming to find what steps should be taken to minimise ELBO based on a computation budget [242].”
Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling (2021, methodology): “…complements recent techniques to speed up existing SGM methods that rely on either different noise schedules (Nichol and Dhariwal, 2021; San-Roman et al., 2021; Watson et al., 2021), alternative discretizations (Jolicoeur-Martineau et al., 2021a) or knowledge distillation (Luhman and Luhman, 2021).”
Fast Sampling of Diffusion Models with Exponential Integrator (2022, methodology): “One category of methods modify/optimize the forward noising process such that backward denoising process can be more efficient [23, 30, 36, 3].”
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference (2023, methodology): “Training-based approaches include optimized discretization (Watson et al., 2021), truncated diffusion (Lyu et al., 2022; Zheng et al., 2022), neural operator (Zheng et al., 2023) and distillation (Salimans & Ho, 2022; Meng et al., 2023).”
Pseudo Numerical Methods for Diffusion Models on Manifolds (2022, background): “Some works search for better variance schedules, including Nichol & Dhariwal (2021) and Watson et al. (2021).”