On Distillation of Guided Diffusion Models

By Chenlin Meng, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, Tim Salimans. Computer Vision and Pattern Recognition, 2022

836 citations recorded by the source corpus.

Classifier-free guided diffusion models have recently been shown to be highly effective at high-resolution image generation, and they have been widely used in large-scale diffusion frameworks including DALL.E 2, Stable Diffusion and Imagen. However, a downside of classifier-free guided diffusion models is that they are computationally expensive at inference time since they require evaluating two diffusion models, a class-conditional model and an unconditional model, tens to hundreds of times. To deal with this limitation, we propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from: Given a pre-trained classifier-free guided model, we first learn a single model to match the output of the combined conditional and unconditional models, and then we progressively distill that model to a diffusion model that requires much fewer sampling steps. For standard diffusion models trained on the pixel-space, our approach is able to generate images visually comparable to that of the original model using as few as 4 sampling steps on ImageNet $64\times 64$ and CIFAR-10, achieving FID/IS scores comparable to that of the original model while being up to 256 times faster to sample from. For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps, accelerating inference by at least 10-fold compared to existing methods on ImageNet $256\times 256$ and LAION datasets. We further demonstrate the effectiveness of our approach on text-guided image editing and inpainting, where our distilled model is able to generate high-quality results using as few as 2–4 denoising steps.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/625d57bd52c60cd79aa4add6c4420dc2ad3b808a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Statistical and Nonlinear Physics

Citations in the corpus

Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (2022, methodology): “It is worth mentioning that alternative methods, such as (Meng et al., 2022), have been proposed to reduce the sampling time, which do not come with such disadvantages.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “However, despite advances in model distillation [49,52,73] and accelerated sampling [4,11,12,36,45,46,80,96,100,108], DMs generally require repeated evaluations of a computationally demanding large U-Net.”
Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, methodology): “Distillation Sampling Distillation models (Salimans & Ho, 2021; Meng et al., 2023) successfully amortize the sampling cost by directly estimating the integral of Eq.”
Bespoke Solvers for Generative Flow Models (2023, neutral): “Distillation does not guarantee sampling from the pre-trained model’s distribution, but, when given access to the training data during distillation training, it is shown to empirically generate samples of comparable quality to the original model (Salimans & Ho, 2022; Meng et al., 2023).”
Imagen Video: High Definition Video Generation with Diffusion Models (2022, methodology): “See Meng et al. (2022) for more details.”
DeepCache: Accelerating Diffusion Models for Free (2023, methodology): “Efforts to accelerate these models have focused on two main strategies: reducing the number of sampling steps, as seen in approaches [36, 41, 54, 60], and decreasing the model inference overhead per step through methods like model pruning, distillation, and quantization [10, 13, 23].”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Approaches that use knowledge distillation [164, 179, 233] can significantly improve the sampling speed of diffusion models.”
Consistency Models (2023, neutral): “, 2022), and distillation techniques (Luhman & Luhman, 2021; Salimans & Ho, 2022; Meng et al., 2022; Zheng et al., 2022).”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “To solve this limitation, we distill the combined output for unconditional and conditional inputs into a single student model [60].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Potential solutions may include faster sampling techniques [89, 146] and training-based distillations [91, 93, 101, 103, 110, 124].”
Adversarial Diffusion Distillation (2023, methodology): “The adversarial loss forces the model to directly generate samples that lie on the manifold of real images at each forward pass, avoiding blurriness and other artifacts typically observed in other distillation methods [43].”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, neutral): “Naturally, a large body of work focuses on speeding up the sampling of diffusion models — both via improved samplers [55, 11, 66, 51] and distilled models that are trained to match the sample quality of their teacher models in fewer steps [44, 35, 34, 58].”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “As a result, prior works [3, 21, 22] have proposed employing knowledge distillation on these models to reduce the required sampling steps and accelerate their inference.”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, neutral): “Distil-lation techniques train a student model to generate samples in fewer steps by mimicking the behavior of a teacher diffusion model [29, 52, 59, 65, 69, 70, 76, 94, 100].”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, methodology): “It’s the most widely used weighting strategy [33, 22, 12, 41].”
Directly Fine-Tuning Diffusion Models on Differentiable Rewards (2023, neutral): “Interestingly, the model with guidance performs similarly; possibly ﬁne-tuning compensates for the lack of guidance similarly to how guided diffusion models can be distilled into unguided ones (Meng et al., 2023).”
Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference (2023, methodology): “In our implementation, we follow the same training procedure in (Meng et al., 2023) except the difference of computation resources.”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, methodology): “To alleviate the training overhead, progressive distillation strategies are usually adopted [25, 35].”