Improving Diffusion Model Efficiency Through Patching

By Troy Luhman, Eric Luhman. 2022

23 citations recorded by the source corpus.

Diffusion models are a powerful class of generative models that iteratively denoise samples to produce data. While many works have focused on the number of iterations in this sampling procedure, few have focused on the cost of each iteration. We find that adding a simple ViT-style patching transformation can considerably reduce a diffusion model's sampling time and memory usage. We justify our approach both through an analysis of the diffusion model objective, and through empirical experiments on LSUN Church, ImageNet 256, and FFHQ 1024. We provide implementations in Tensorflow and Pytorch.

Subfields: Artificial Intelligence, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

In Efficiency & Compression

Citations in the corpus

Scalable Adaptive Computation for Iterative Generation (2022, background): “While recent work (Luhman & Luhman, 2022) has explored pixel-level diffusion with Transformers, they have not been shown to attain strong performance or scale to large inputs.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Furthermore, diffusion models have close connections with other research area, such as robust learning [101, 156, 205], representative learning [1, 131, 232, 249] and reinforcement learning [92].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, methodology): “[48] have also trained two diffusion models for two-stage denoising for image generation, but their models are trained over images of different resolutions and separately from the beginning.”
Stable Bias: Evaluating Societal Representations in Diffusion Models (2023, unknown): “…diffusion models is that they are simpler to train than previous generations of generative models owing to their likelihood-based loss function, whose mode-covering characteristic [38] is also one of the main reasons why model outputs are so diverse compared to other classes of generative models.”