DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models

By Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu. Machine Intelligence Research, 2022

1,074 citations recorded by the source corpus.

Diffusion probabilistic models (DPMs) have achieved impressive success in high-resolution image synthesis, especially in recent large-scale text-to-image generation applications. An essential technique for improving the sample quality of DPMs is guided sampling, which usually needs a large guidance scale to obtain the best sample quality. The commonly-used fast sampler for guided sampling is denoising diffusion implicit models (DDIM), a first-order diffusion ordinary differential equation (ODE) solver that generally needs 100 to 250 steps for high-quality samples. Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before. In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows larger. To further speed up guided sampling, we propose DPM-Solver++, a high-order solver for the guided sampling of DPMs. DPM-Solver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution. We further propose a multistep variant of DPM-Solver++ to address the instability issue by reducing the effective step size. Experiments show that DPM-Solver++ can generate high-quality samples within only 15 to 20 steps for guided sampling by pixel-space and latent-space DPMs.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/baa4f95081e9663fb045d145acc70049ace16ac9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Probabilistic weather forecasting with machine learning (2024, positive): “We use the second-order DPMSolver++2S solver 53 , augmented with the stochastic churn (again making use of P noise ) and noise inflation techniques used in ref.”
Bespoke Solvers for Generative Flow Models (2023, positive): “(Zhang et al., 2023) further introduced refined error conditions to fulfill desired order conditions and achieve better sampling, while Lu et al. (2022b) adapted the method to guided sampling.”
One-Step Diffusion with Distribution Matching Distillation (2023, methodology): “Our method showcases superior performance over StyleGAN-T [67], surpasses all other diffusion acceleration methods, including advanced diffusion solvers [46, 90], and diffusion distillation techniques such as Latent Consistency Models [48, 49], UFOGen [83], and InstaFlow [43].”
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation (2022, methodology): “DPM Solver [26, 27] solved the ordinary differential equation of the reverse process of DPMs [42], gave high-order approximated solutions of these equations, and got high-quality results with only about 10-20 evaluations.”
Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models (2023, background): “Over the recent years, several samplers have been proposed for sampling from the trained diffusion models [68, 48, 26, 27, 15].”
Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, neutral): “Intrinsically, this is because such models characterize the joint distribution by the chain rule of probability, motivating research on developing other types of generative models for text.”
Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “Related methods include simulating the diffusion process in fewer steps by generalizing it to a non-Markovian process [42], adjusting the variance schedule [30], and the use of high-order solvers to approximate diffusion generation [22, 1, 24, 25].”
The operational medium-range deterministic weather forecasting can be extended beyond a 10-day lead time (2023, neutral): “Under these prior conditions, we employ the diffusion model 22,37,38 , to estimate the potential solution space of the forecasts.”
Continuous diffusion for categorical data (2022, methodology): “Based on results in the image domain, it may be possible to reduce this further with stochastic samplers 11 , or more advanced sampling algorithms (Lu et al., 2022a,b).”
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, methodology): “In Table 4, We compute the FID score between the 1k samples from the 100 prompts and a 50k reference batch, which is sampled by 50-step DPM-Solver++ [28] with 500 images per prompt.”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “…classifier-free guidance based conditional diffusion models with a single inference step [19] • DPM++(2M): An improved version of the second-order DPM sampler for classifier-free guidance based conditional diffusion models [19], which is a second order method using two model predictions per step.”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “The first kind focuses on fast post-hoc samplers [19, 20, 29, 60, 61, 62].”
A Survey on Generative Diffusion Models (2022, methodology): “For instance, advanced ODE/SDE solvers have been proposed to expedite the sampling process [28], [29], [30], while model distillation strategies have been employed [14], [31] to achieve the same goal.”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “Algorithm 1 Flow-DPM-Solver (Modified from DPM-Solver++) Require: initial value x T , time steps { t i } Mi =0 , data prediction model x θ , velocity prediction model v θ , time-step shift factor s ▷ Hyper-parameter and Time-step transformation end if 15: end for 16: return ˜ x t M Multi-Caption Aut”
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference (2023, methodology): “For LCM, we use three possible ODE solvers here: DDIM (Song et al., 2020a), DPM-Solver (Lu et al., 2022a), DPM-Solver++ (Lu et al., 2022b), and we compare their performance in Sec 5.2.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “MUSE [10] scales MaskGIT’s architecture to 3 billion parameters and merges it with the T5 language model [50], setting new benchmarks in text-to-image synthesis.”
Composer: Creative and Controllable Image Synthesis with Composable Conditions (2023, methodology): “Sampling algorithms such as DDIM (Song et al., 2020a) and DPM-Solver (Lu et al., 2022a;b; Bao et al., 2022) are often adopted to speed up the sampling process of diffusion models.”