Scalable Diffusion Models with Transformers
By William S. Peebles, Saining Xie. IEEE International Conference on Computer Vision, 2022
6,663 citations recorded by the source corpus.
We explore a new class of diffusion models based on the transformer architecture. We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops. We find that DiTs with higher Gflops—through increased transformer depth/width or increased number of input tokens—consistently have lower FID. In addition to possessing good scalability properties, our largest DiT-XL/2 models outperform all prior diffusion models on the class-conditional ImageNet 512×512 and 256×256 benchmarks, achieving a state-of-the-art FID of 2.27 on the latter.
Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Generative Models, Model Architectures, Signal Processing
View Paper: https://arxiv.org/abs/2212.09748
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Xiaogang Wang
- Jitendra Malik
- Trevor Darrell
- Andrew Zisserman
- Andrea Vedaldi
In Radiology, Nuclear Medicine and Imaging
- Nicholas Matthew Boffi
- Eric Vanden-Eijnden
- Michael S. Albergo
- Diederik P. Kingma
- Ruiqi Gao
- Georgios Pavlakos
- Ming-Hsuan Yang
- Jitendra Malik
In Generative Models
- Richard Zhang
- Eli Shechtman
- Andreas Blattmann
- Robin Rombach
- Jun-Yan Zhu
- Stefano Ermon
- Ming-Hsuan Yang
- David J. Fleet
In Model Architectures
In Signal Processing
- Stefano Ermon
- Robin Rombach
- David J. Fleet
- Richard Zhang
- Ming-Hsuan Yang
- Andreas Blattmann
- Patrick Esser
- Eli Shechtman
Citations in the corpus
- Scalable Adaptive Computation for Iterative Generation (2022, background): “Concurrent work (Peebles & Xie, 2022) has shown Trans-formers may be more tenable when combined with latent diffusion i.e. by downsampling inputs with large-scale pre-trained VAEs, but reliance on uniform computation limits gracefully scaling to larger data.”
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “For example, based on DiT [2], OpenSora 2 , OpenSoraPlan [3] and recent works [4, 5] utilize the collected million-scale text-video datasets to reproduce Sora.”
- Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “Although recent studies show that transformer architectures are also capable of serving as the noise estimation backbone [32], convolutional UNets are still the de facto choice of architecture today.”
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “Model As shown in Fig.1, we use latent Diffusion Transformer (DiT) [41] as backbone.”
- SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Recent advancements have also explored transformer-based architectures [67, 44, 6, 7].”
- AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, methodology): “…schedules used in our experiments, Schedule A is the schedule for pre-training Stable Diffusion; Schedule B is our choice, which is different from the schedule of SD in how the beta sequence is computed; Schedule C is used in DDPM [5] and DiT [18] and differs more from SD’s pre-training schedule.”
- Analyzing and Improving the Training Dynamics of Diffusion Models (2023, neutral): “Model averaging [27, 49, 57, 68, 72] is an indispensable technique in all high-quality image synthesis methods [2, 11, 23, 29, 31, 33, 46, 48, 54, 60, 63, 65].”
- Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, methodology): “Diffusion Classifier can be used to extract a zero-shot classifier from a text-to-image model like Stable Diffusion [64], to extract a standard classifier from a class-conditional diffusion model like DiT [57], and so on.”
- Text-to-3D using Gaussian Splatting (2023, background): “After DDPM [27, 66] highlights the effectiveness of diffusion models in capturing real-world image data, a plethora of research has emerged to improve the inherent challenges, including fast sampling [5, 39, 65] and architecture improvements [6, 20, 28, 36, 48, 49].”
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (2023, methodology): “• Architecture: During the exploration stage of this work, we briefly experimented with transformer-based architectures such as UViT [16] and DiT [33], but found no immediate benefit.”
- Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, neutral): “In order to better utilize camera information in our pipeline, we implement an adaptive layer norm (adaLN) to modulate the DINOv2 features with the camera features, which is similar to [21, 48].”
- Structured 3D Latents for Scalable and Versatile 3D Generation (2024, neutral): “Since previous works have not utilized this dataset for training, we leverage it as our testing dataset to evaluate the generalization of our model.”
- Simplified and Generalized Masked Diffusion for Discrete Data (2024, neutral): “For ImageNet 64 × 64 , we reduced the number of residual blocks (in one side of the U-Net structure) from 64 to 48 and added a 12-layer diffusion transformer [75] with 768 hidden dimension and 12 heads in the middle.”
- SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning (2024, neutral): “5-VL [2], GLM4-V [19], InternVL3 [65], and DiT models [45] , can all satisfy to this trend.”
- Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, neutral): “The transition from early 3D Convolutional Neural Networks [30] to the now-common Transformer architectures [94, 67] has led to the development of classic shape generation networks, enhancing performance.”
- SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “Stable Diffusion 3 (SD3) (Esser et al., 2024) further advanced the field by proposing the Multi-modal Diffusion Transformer (MM-DiT), which effectively integrates text and image modalities.”