Pseudo Numerical Methods for Diffusion Models on Manifolds

By Luping Liu, Yi Ren, Zhijie Lin, Zhou Zhao. 2022

905 citations recorded by the source corpus.

Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples. However, DDPMs require hundreds to thousands of iterations to produce final samples. Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g., Improved Denoising Diffusion Probabilistic Models) or the denoising equation (e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceleration methods cannot maintain the quality of samples and even introduce new noise at a high speedup rate, which limit their practicability. To accelerate the inference process while keeping the sample quality, we provide a fresh perspective that DDPMs should be treated as solving differential equations on manifolds. Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs). Specifically, we figure out how to solve differential equations on manifolds and show that DDIMs are simple cases of pseudo numerical methods. We change several classical numerical methods to corresponding pseudo numerical methods and find that the pseudo linear multi-step method is the best in most situations. According to our experiments, by directly using pre-trained models on Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules. Our implementation is available at https://github.com/luping-liu/PNDM.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Generative Models

Citations in the corpus

Elucidating the Design Space of Diffusion-Based Generative Models (2022, methodology): “Improving the output quality and/or decreasing the computational cost of sampling are common topics in diffusion model research (e.g., [10, 24, 30, 31, 32, 36, 43, 50, 52, 53, 56]).”
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, methodology): “3.4 for using DPM-Solver in discrete-time DPMs, and then compare DPM-Solver with other discrete-time training-free samplers, including DDPM [2], DDIM [19], Analytic-DDPM [21], Analytic-DDIM [21], PNDM [22], FastDPM [38] and Itô-Taylor [24].”
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, methodology): “Generally, fast samplers such as DDIM [21], PNDM [36] and DPM-Solver [37, 38], are adopted in the inference stage to accelerate the generation process.”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, methodology): “For this, they collect a dataset of noise-image pairs ( z, y ) where the image y is generated using the teacher diffusion model, and a deterministic sampler [48,49,52], starting from the noise map z .”
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2022, background): “5 and use 50 inference steps with PNDM [13].”
One-Step Diffusion with Distribution Matching Distillation (2023, methodology): “Concretely, we build a paired dataset D = { z, y of random Gaussian noise images z and the corresponding outputs y , obtained by sampling the pre-trained diffusion model µ base using a deterministic ODE solver [31, 41, 72].”
Token Merging: Your ViT But Faster (2022, methodology): “5, we show the results of applying our method off-the-shelf and during MAE fine-tuning using ViT-L from Spatiotemporal MAE compared to the relevant state-of-the-art on Kinetics-400 classification: Swin (Liu et al., 2022c) pretrained on ImageNet-21k, MViTv2 (Li et al., 2022) pretrained with MaskFeats”
DIRE for Diffusion-Generated Image Detection (2023, methodology): “Since then, more and more researchers turn their attention to diffusion models for improving the architectures [10, 37], accelerating sampling speed [32, 42, 23, 24], exploring downstream tasks [16, 31, 1, 33], and etc .”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “We also tried and found no benefit from gamma noise [Nachmani et al., 2021], L1 loss, and F-PNDM sampling [Liu et al., 2022].”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “To accelerate the inference, a series of post-hoc samplers have been proposed [18, 19, 20].”
DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “With the changing economic and geopolitical landscape within the transport and logistics industry, Last Mile Delivery (LMD) has become the most expensive portion of the logistics fulfillment chain, representing over 41% of overall supply chain costs [50].”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, background): “This is also reminiscent of our approach, but their model requires multi-view data for training, is only tested on synthetic data, and requires to explicitly sample multiple views for reconstruction (in our case they remain implicit).”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, background): “Other studies propose fast sampling methods that require only a dozen steps [49, 29, 32, 24] to generating high-quality images.”
Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis (2022, methodology): “Similarly, in the T2I subdomain, Liu et al. (2021b) and Nichol et al. (2021) apply a noisy CLIP model to measure the cosine similarity between text prompts and noisy images.”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, methodology): “Samples are drawn from these diffusion models using classifier(-free) guidance [13, 28] as well as various sampling algorithms that use deterministic [15, 34, 43, 46, 70, 89] or stochastic [4, 5, 16, 90] iterative updates.”