All are Worth Words: A ViT Backbone for Diffusion Models

By Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, Jun Zhu. Computer Vision and Pattern Recognition, 2022

631 citations recorded by the source corpus.

Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a simple and general ViT-based architecture (named U-ViT) for image generation with diffusion models. U-ViT is characterized by treating all inputs including the time, condition and noisy image patches as tokens and employing long skip connections between shallow and deep layers. We evaluate U-ViT in unconditional and classconditional image generation, as well as text-to-image generation tasks, where U-ViT is comparable if not superior to a CNN-based U-Net of a similar size. In particular, latent diffusion models with U-ViT achieve record-breaking FID scores of 2.29 in class-conditional image generation on ImageNet 256×256, and 5.48 in text-to-image generation on MS-COCO, among methods without accessing large external datasets during the training of generative models. Our results suggest that, for diffusion-based image modeling, the long skip connection is crucial while the down-sampling and upsampling operators in CNN-based U-Net are not always necessary. We believe that U-ViT can provide insights for future research on backbones in diffusion models and benefit generative modeling on large scale cross-modality datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/897f3bb5eacaa80359e81ff33378e1110e20ae95

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024, positive): “The success of diffusion transformers on images (Bao et al., 2023a; Peebles & Xie, 2023a; Bao et al., 2023b) and videos (Bao et al., 2024) suggests that diffusion models can be scaled up by incorporating transformers.”
Text-to-3D using Gaussian Splatting (2023, background): “After DDPM [27, 66] highlights the effectiveness of diffusion models in capturing real-world image data, a plethora of research has emerged to improve the inherent challenges, including fast sampling [5, 39, 65] and architecture improvements [6, 20, 28, 36, 48, 49].”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “MUSE [10] scales MaskGIT’s architecture to 3 billion parameters and merges it with the T5 language model [50], setting new benchmarks in text-to-image synthesis.”
Efficient Diffusion Training via Min-SNR Weighting Strategy (2023, methodology): “Recently, researchers have also explored the use of Vision Transform-ers [13] as an alternative, with U-ViT [2] borrowing the skip connection design from UNet [42] and DiT [38] leveraging Adaptive LayerNorm and discovering that the zero initialization strategy is critical for achieving…”
RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, positive): “Afterward, all of them are concatenated together in the length direction to achieve in-context conditioning (Peebles & Xie, 2023; Bao et al., 2023), resulting in an input token sequence of length 1 + T a + 1 + 1 .”
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation (2024, methodology): “In the realm of diffusion models, DiT [35] and UViT [2] pioneer the use of Transformer architecture.”
Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding (2024, methodology): “Our diffusion model is parameterized with a transformer [33, 23, 4].”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “DiT [45] and U-ViT [5] employ more scalable transformers to model diffusion, achieving superior results.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Recent advancements have also explored transformer-based architectures [67, 44, 6, 7].”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, positive): “More recently, LlamaGen (Sun et al., 2024) has demonstrated that large language model architecture like Llama (Touvron et al., 2023) can also autoregressively model image tokens, which can accordingly obtain decent performance in class-conditional image generation.”
Large Language Diffusion Models (2025, neutral): “(1), rather than a unique result of the ARMs in Eq.”
Latte: Latent Diffusion Transformer for Video Generation (2024, positive): “Conversely, on the one hand, DiT (Pee-bles and Xie, 2023) and U-ViT (Bao et al, 2023) adapt the architecture of ViT (Dosovitskiy et al, 2021) into diffusion models for image generation and achieves great performance.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Along this line, DiT [4] and U-ViT [54] are among the first works to employ vision transformers for latent diffusion models.”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, neutral): “Specifically, we consider the following four types of approaches: (a) Pixel diffusion : ADM (Dhariwal & Nichol, 2021), VDM ++ (Kingma & Gao, 2024), Simple diffusion (Hoogeboom et al., 2023), CDM (Ho et al., 2022), (b) Latent diffusion with U-Net : LDM (Rombach et al., 2022), (c) Latent diffusion wit”
HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “…of diffusion based visual generation, Li et.al [49] study the scaling properties on UNet, while transformer based works such as DiT [65], U-ViT [3], Lumina-T2X [23], and SD3 [21] only study the scaling behavior between sample quality and network complexity, leaving the power-laws about the…”
PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (2023, methodology): “The Diffusion Transformer (DiT) (Peebles & Xie, 2023) and its variant (Bao et al., 2023; Zheng et al., 2023) follow the step to further replace the Convolutional-based U-Net (Ronneberger et al., 2015) backbone with Transformers.”