Elucidating the Design Space of Diffusion-Based Generative Models
By Tero Karras, Miika Aittala, Timo Aila, Samuli Laine. 2022
3,661 citations recorded by the source corpus.
We argue that the theory and practice of diffusion-based generative models are currently unnecessarily convoluted and seek to remedy the situation by presenting a design space that clearly separates the concrete design choices. This lets us identify several changes to both the sampling and training processes, as well as preconditioning of the score networks. Together, our improvements yield new state-of-the-art FID of 1.79 for CIFAR-10 in a class-conditional setting and 1.97 in an unconditional setting, with much faster sampling (35 network evaluations per image) than prior designs. To further demonstrate their modular nature, we show that our design changes dramatically improve both the efficiency and quality obtainable with pre-trained score networks from previous work, including improving the FID of a previously trained ImageNet-64 model from 2.07 to near-SOTA 1.55, and after re-training with our proposed improvements to a new SOTA of 1.36.
Subfields: Biophysics, Computer Vision and Pattern Recognition, Generative Models, Model Architectures, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biophysics
- Prafulla Dhariwal
- Yang Song
- Jonathan Ho
- Diederik P. Kingma
- Alex Nichol
- Arash Vahdat
- Ruiqi Gao
- Jiaming Song
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Chen Change Loy
In Generative Models
- Arash Vahdat
- Stefano Ermon
- Karsten Kreis
- Richard Zhang
- Eli Shechtman
- Jun-Yan Zhu
- Jiaming Song
- Prafulla Dhariwal
In Model Architectures
- Shuicheng Yan
- Luc Van Gool
- Diederik P. Kingma
- Pan Zhou
- Quoc V. Le
- Ruiqi Gao
- Mohammad Norouzi
- Prafulla Dhariwal
In Signal Processing
- Jonathan Ho
- David J. Fleet
- Diederik P. Kingma
- Arash Vahdat
- Mohammad Norouzi
- Yang Song
- Prafulla Dhariwal
- Stefano Ermon
Citations in the corpus
- Scalable Diffusion Models with Transformers (2022, background): “Improvements to DDPMs over the past two years have largely been driven by improved sampling techniques [19, 55, 27], most notably classifier-free guidance [21], reformulating diffusion models to predict noise instead of pixels [19] and using cascaded pipelines where low-resolution base diffusion…”
- Scalable Adaptive Computation for Iterative Generation (2022, methodology): “Common diffusion models for images and videos can be roughly divided into pixel diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020; Dhariwal & Nichol, 2022; Ho et al., 2022a; Karras et al., 2022) and latent diffusion models (Rombach et al., 2022).”
- Accurate structure prediction of biomolecular interactions with AlphaFold 3 (2024, methodology): “We use a relatively standard diffusion approach 33 in which the diffusion model is trained to receive ‘noised’ atomic coordinates and then predict the true coordinates.”
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “We follow the EDM [50] formulation with the denoiser K(s; t, σ) preconditioned as:”
- Fast Sampling of Diffusion Models with Exponential Integrator (2022, methodology): “Let us consider VESDE introduced in Karras et al. (2022) where The VESDE has a simple ODE formulation, Eq.”
- DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “Typical diffusion models [100, 102, 40, 103, 85, 56] operate in the continuous domain, progressively adding Gaussian noise to the clean data in the forward process, and learning to remove noises in the reverse process in a discrete-time framework.”
- SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “Our approach involves utilizing the widely used EDM [18] framework, incorporating a simplified diffusion loss for fine-tuning, as followed in SVD [2].”
- CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model (2023, methodology): “More specifically, we choose the Euler solver, and the 𝑁 discrete-time points are designed as the same setting in EDM [14].”
- Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “Many works improve diffusion-based image synthesis by changing the parameterization [14, 27, 46], introducing advanced sampling methods [52, 24, 22, 47, 20], designing more powerful architectures [3, 15, 57, 30], or conditioning on additional information [25].”
- Consistency Models (2023, positive): “Training Details In both consistency distillation and progressive distillation, we distill EDMs (Karras et al., 2022).”
- One-Step Diffusion with Distribution Matching Distillation (2023, methodology): “Our work builds upon leading diffusion models [31, 63] and introduces a simple distillation pipeline that reduces the multi-step generative process to a single forward pass.”
- Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, neutral): “Then, the parameters θ of the diffusion model f θ are optimized to minimize the denoising score matching objective function: where c is optional conditions, and the target y can be ϵ , α τ ϵ − σ τ x 0 [45], or x 0 [23], depending on the selected denoising process.”
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference (2023, methodology): “(Song et al., 2023) used Euler (Song et al., 2020b) or Heun solver (Karras et al., 2022) as the numerical ODE solver.”
- PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, background): “…be interpreted as score matching [80], and generate samples via an iterative denoising procedure that may use stochastic updates [6, 5, 15, 94] which solve stochastic differential equations (SDEs) or deterministic updates [74, 43, 41, 93, 30, 14] which solve ordinary differential equations (ODEs).”
- Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, methodology): “The optimal choice of γ depends on practical usage and empirical configuration (Karras et al., 2022; Xu et al., 2023).”
- Mean Flows for One-step Generative Modeling (2025, positive): “This procedure involves solving stochastic differential equations (SDE), which can be reformulated as probability flow ordinary differential equations (ODE) [45, 22].”
- Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, positive): “ELBO [Kingma et al., 2021, Song et al., 2021a] 1 IDDPM ( (cid:15) -prediction with ’cosine’ schedule) [Nichol and Dhariwal, 2021] sech ( λ/ 2) EDM [Karras et al., 2022] (Appendix D.1) N ( λ ;2 .”