Gotta Go Fast When Generating Data with Score-Based Models

By Alexia Jolicoeur-Martineau, Ke Li, Remi Piche-Taillefer, Tal Kachman, Ioannis Mitliagkas. 2021

265 citations recorded by the source corpus.

Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly - they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, methodology): “Jolicoeur-Martineau et al. (2021) introduce an advanced SDE solver to simulate the reverse process in a more efficient way.”
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Recent studies have improved DPMs further in both theory and practice [25,29,31].”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “Figure 4 shows that our stochastic sampler outperforms previous samplers [24, 36, 48] by a significant margin, especially at low step counts.”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, methodology): “The compared methods include the Euler-Maruyama discretization for diffusion SDEs [3], the adaptive step size solver for diffusion SDEs [20] and the RK methods for diffusion ODEs [3, 28] in Eq.”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…including unconditional image generation [Song and Ermon, 2019, Ho et al., 2020, Song et al., 2021, Nichol and Dhariwal, 2021], shape generation [Cai et al., 2020], text-to-speech [Chen et al., 2021, Kong et al., 2020] and single image super-resolution [Saharia et al., 2021, Li et al., 2021].”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, unknown): “In essence, all training-free sampling methods for DPMs can be understood as either discretizing diffusion SDEs (Ho et al., 2020; Song et al., 2021b; Jolicoeur-Martineau et al., 2021; Tachibana et al., 2021; Kong & Ping, 2021; Bao et al., 2022b; Zhang et al., 2022) or discretizing diffusion ODEs…”
Fast Sampling of Diffusion Models with Exponential Integrator (2022, background): “Another category of methods speed up the numerical solver for stochastic differential equations (SDEs) or ordinary differential equations (ODEs) associated with the DMs [14, 30, 33].”
Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling (2021, methodology): “, 2021), alternative discretizations (Jolicoeur-Martineau et al., 2021a) or knowledge distillation (Luhman and Luhman, 2021).”
Subspace Diffusion Generative Models (2022, methodology): “Distillation [16] 2.57 CLD-SGM [4] 2.23 LSGM [21] 2.10 Adaptive solver [8] 2 Fisher divergence threshold”
A Continuous Time Framework for Discrete Denoising Models (2022, background): “For example, in continuous state spaces, continuous time samplers that greatly reduce the sampling time have been devised [14, 15, 16, 17] as well as ones that improve sample quality [4, 18].”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “, 2021), as well as acceleration of numerical schemes (Poli et al., 2020; Jolicoeur-Martineau et al., 2021).”
SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models (2021, unknown): “Though the diffusion model is relatively slow in inference, it has great potential to generate images much faster, according to recent works [57–59], which focus on boosting the inference speed of diffusion models.”
Post-Training Quantization on Diffusion Models (2022, unknown): “[14] introduce an advanced SDE solver to accelerate the reverse process via an adaptively larger sampling rate.”
On Distillation of Guided Diffusion Models (2022, methodology): “For instance, denoising diffusion implicit models (DDIM [38]), probability flow sampler [40], fast SDE integrators [8] have been proposed to improve the sampling speed of diffusion models.”
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference (2023, methodology): “Various approaches have been proposed, including training-free methods such as ODE solvers (Song et al., 2020a; Lu et al., 2022a;b), adaptive step size solvers (Jolicoeur-Martineau et al., 2021), predictor-corrector methods (Song et al., 2020b).”