Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed

By Eric Luhman, Troy Luhman. 2021

392 citations recorded by the source corpus.

Iterative generative models, such as noise conditional score networks and denoising diffusion probabilistic models, produce high quality samples by gradually denoising an initial noise vector. However, their denoising process has many steps, making them 2-3 orders of magnitude slower than other generative models such as GANs and VAEs. In this paper, we establish a novel connection between knowledge distillation and image generation with a technique that distills a multi-step denoising process into a single step, resulting in a sampling speed similar to other single-step generative models. Our Denoising Student generates high quality samples comparable to GANs on the CIFAR-10 and CelebA datasets, without adversarial training. We demonstrate that our method scales to higher resolutions through experiments on 256 x 256 LSUN. Code and checkpoints are available at https://github.com/tcl9876/Denoising_Student

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

In Efficiency & Compression

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “[39] add an adversarial objective, [5] introduce non-Markovian diffusion processes that allow to trade off synthesis speed, quality, and sample diversity, [40] learn a sequence of conditional energy-based models for denoising, [41] distill the iterative sampling process into single shot synthesis, a”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “One promising work in this direction is from Luhman and Luhman [37], who explore a way to distill the DDIM sampling process into a single step model.”
Noise Estimation for Generative Diffusion Models (2021, background): “In (Luhman and Luhman 2021), authors propose a distilation framework that can sample in one iteration.”
Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models (2022, methodology): “In addition to these trajectory searching techniques, Luhman & Luhman (2021) compress the reverse denoising process into a single step model; San-Roman et al. (2021) dynamically adjust the trajectory during inference.”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Samplers that need extra training or optimization include knowledge distillation [13, 14], learning the noise level or variance [15, 16, 33], and learning the noise schedule or sample trajectory [17, 18].”
Bilateral Denoising Diffusion Models (2021, neutral): “A more recent work [34] explored a student-teacher method to distill the DDIMs sampling process into a single-step model.”
One-Step Diffusion with Distribution Matching Distillation (2023, methodology): “Alternatively, diffusion distillation has emerged as a promising avenue for further boosting speed [3, 16, 42, 47, 51, 65, 75, 82, 91].”
Gradient-based Constrained Sampling from Language Models (2022, background): “Further improvements may also be achieved by adapting more sophisticated gradient based methods for faster convergence (Girolami and Calderhead, 2011) or techniques from diffusion models in image generation (Luhman and Luhman, 2021).”
Diffusion-GAN: Training GANs with Diffusion (2022, methodology): “For this reason, a variety of methods have been proposed to reduce the generation cost of diffusion-based generative models (Kong and Ping, 2021; Luhman and Luhman, 2021; Pandey et al., 2022; San-Roman et al., 2021; Song et al., 2021a; Xiao et al., 2021; Zheng et al., 2022).”
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow (2022, positive): “Quantitatively, we claim a state-of-the-art result of FID (4.85) and recall (0.51) on CIFAR10 for one-step fast diffusion/ﬂow models [5, 48, 91, 99, 47].”
SinSR: Diffusion-Based Image Super-Resolution in a Single Step (2023, background): “4. the ordinary differential equation (ODE) of the inference process makes this scheme less attractive on a large-scale dataset [24].”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “However, despite advances in model distillation [49,52,73] and accelerated sampling [4,11,12,36,45,46,80,96,100,108], DMs generally require repeated evaluations of a computationally demanding large U-Net.”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, neutral): “Samplers based on extra training include model distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021), learning reverse process variances (San-Roman et al., 2021; Nichol & Dhariwal, 2021; Bao et al., 2022a), and learning sampling steps (Lam et al., 2021; Watson et al., 2022).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Approaches that use knowledge distillation [148, 203] can significantly improve the sampling speed of diffusion models.”
Bespoke Solvers for Generative Flow Models (2023, methodology): “Current approaches for efficient sampling of flow models divide into two main groups: (i) Distil-lation : where the pre-trained model is fine-tuned to predict either the final sampling (Luhman & Luhman, 2021) or some intermediate solution steps (Salimans & Ho, 2022) of the ODE.”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, methodology): “To alleviate this issue, multiple methods have been proposed, including knowledge distillation (Luhman & Luhman, 2021), learning an adaptive noise schedule (San-Roman et al., 2021), introducing non-Markovian diffusion processes (Song et al., 2021a; Kong & Ping, 2021), and using better SDE solvers…”
A Survey on Generative Diffusion Models (2022, background): “TRACT [60], Denoising Student [61], and Consistency Models [14] extended this effect, increasing acceleration rates to 64 and 1024, by directly estimating clean data from noisy samples at time T .”