simple diffusion: End-to-end diffusion for high resolution images

By Emiel Hoogeboom, Jonathan Heek, Tim Salimans. International Conference on Machine Learning, 2023

453 citations recorded by the source corpus.

Currently, applying diffusion models in pixel space of high resolution images is difficult. Instead, existing approaches focus on diffusion in lower dimensional spaces (latent diffusion), or have multiple super-resolution levels of generation referred to as cascades. The downside is that these approaches add additional complexity to the diffusion framework. This paper aims to improve denoising diffusion for high resolution images while keeping the model as simple as possible. The paper is centered around the research question: How can one train a standard denoising diffusion models on high resolution images, and still obtain performance comparable to these alternate approaches? The four main findings are: 1) the noise schedule should be adjusted for high resolution images, 2) It is sufficient to scale only a particular part of the architecture, 3) dropout should be added at specific locations in the architecture, and 4) downsampling is an effective strategy to avoid high resolution feature maps. Combining these simple yet effective techniques, we achieve state-of-the-art on image generation among diffusion models without sampling modifiers on ImageNet.

Subfields: Biophysics, Modeling and Simulation, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/6e3a3b7a8a0376d867cad72eedf2f9b746f29a33

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Modeling and Simulation

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “Recent works [44] show that it is crucial to adopt the noise schedule when training image diffusion models, shifting towards more noise for higher-resolution images.”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “We also disable self-attention at 32 × 32 resolution, similar to many prior works [21, 25, 46].”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, unknown): “In recent years, image generation has experienced remarkable progress, driven by the significant advancements in both transformers [19, 62, 66, 10, 67, 68] and diffusion models [16, 55, 29, 49, 21].”
Text-to-3D using Gaussian Splatting (2023, background): “One of the most successful applications of diffusion models lies in text-to-image generation, where they have shown remarkable progress in generating realistic images from text prompts [1, 26, 55].”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, background): “Diffusion Models and Flow-based Models Diffusion models [16, 17, 25, 26, 27, 28, 29, 30, 31] have achieved unprecedented results in various generative modeling tasks, including image/video generation [9, 32, 33, 34, 35], audio generation [36], point cloud generation [37, 38, 39, 40], biological…”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, unknown): “…the U-Net (Ronneberger et al., 2015) denoising backbone with a Transformer (Peebles & Xie, 2022; Jabri et al., 2023) or a hybrid of both (Hoogeboom et al., 2023), making the distinctions between diffusion and language models in visual generation more blurred, with a key distinction…”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Similar to prior work [10, 67], we found it important to shift the noise schedule towards high noise levels as we move from the pre-trained image diffusion model to our multi-view diffusion model that captures data of higher dimensionality.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “We used a cosine β -noise schedule [43].”
FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, unknown): “Expressing the log-SNR for arbitrary µ and σ gives The α -shifted log-SNR [12, 18] is obtained as Comparing Equation (9) and Equation (10), we identify µ = log α for σ = 1 .”
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers (2024, background): “Params sian [62], though alternatives of these choices exist [27, 54].”
Improved Techniques for Training Consistency Models (2023, methodology): “For ImageNet 64 ˆ 64 , we use a dropout rate of 0.2, but only apply them to convolutional layers whose the feature map resolution is smaller or equal to 16 ˆ 16 , following the configuration in Hoogeboom et al. (2023).”
Photorealistic Video Generation with Diffusion Models (2023, background): “Recently, the community has started adopting transformers as the denoising backbone for diffusion models in place of U-Net [12, 38, 50, 53, 87].”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, unknown): “• Simple diffusion (Hoogeboom et al., 2023) proposes a diffusion model for high-resolution image generation by exploring various techniques to simplify a noise schedule and architectures.”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, unknown): “For class-conditional generation on ImageNet 128x128, we heavily followed the setting of simple diffusion [Hoogeboom et al., 2023].”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “In contrast to previous work that trains diffusion models directly on ImageNet data (e.g., [12, 26, 28]), here we leverage a large-scale text-to-image diffusion model [50] as a foundation, in part to explore the potential beneﬁts of pre-training on a larger, generic corpus.”
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, methodology): “• Architecture: During the exploration stage of this work, we brieﬂy experimented with transformer-based architectures such as UViT [16] and DiT [33], but found no immediate beneﬁt.”