Masked Diffusion Transformer is a Strong Image Synthesizer

By Shanghua Gao, Pan Zhou, Mingg-Ming Cheng, Shuicheng Yan. IEEE International Conference on Computer Vision, 2023

312 citations recorded by the source corpus.

Despite its success in image synthesis, we observe that diffusion probabilistic models (DPMs) often lack contextual reasoning ability to learn the relations among object parts in an image, leading to a slow learning process. To solve this issue, we propose a Masked Diffusion Transformer (MDT) that introduces a mask latent modeling scheme to explicitly enhance the DPMs’ ability to contextual relation learning among object semantic parts in an image. During training, MDT operates in the latent space to mask certain tokens. Then, an asymmetric masking diffusion transformer is designed to predict masked tokens from unmasked ones while maintaining the diffusion generation process. Our MDT can reconstruct the full information of an image from its incomplete contextual input, thus enabling it to learn the associated relations among image tokens. Experimental results show that MDT achieves superior image synthesis performance, e.g., a new SOTA FID score in the ImageNet data set, and has about 3× faster learning speed than the previous SOTA DiT. The source code is released at https://github.com/sail-sg/MDT.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ca21f28e2a0a8205038301d8385151ab7ca2a050

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “Like Masked AutoEncoder (MAE) [33], Masked Diffusion Transformer (MDT) [55] incorporates mask latent modeling into the diffusion process to explicitly enhance contextual relation learning among object semantic parts in image synthesis.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, unknown): “MUSE [10] scales MaskGIT’s architecture to 3 billion parameters and merges it with the T5 language model [50], setting new benchmarks in text-to-image synthesis.”
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, unknown): “Previous great work MDT [12] or REPA [45] achieved similar convergence performance with the help of Mask Image Modeling (MIM) and representation alignment.”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, unknown): “Put simply, ‘EDM-monotonic’ equals the EDM weighting, except that it is constant to the left of its peak; see Figure 3.”
Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation (2023, unknown): “For instance, when evaluating on the ImageNet dataset, a gold standard benchmark for image generation, the best language model (Lee et al., 2022) underperforms the diffusion model (Gao et al., 2023) by a substantial 48% margin (FID 3.41 vs .”
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation (2024, methodology): “Subsequent works, including DiffiT [16], SiT [29], and FiT [27], have improved upon DiT’s architecture, while [13, 54] enhance training efficiency through masked modeling techniques.”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Multiple classes of generative models have utilized Transformers [73] as back-bone, such as, Generative adversarial networks [42, 47, 85], autoregressive [10, 11, 20, 21, 27, 59, 74, 77, 78, 80, 81] and diffusion [2, 22, 41, 50, 53, 87] models.”
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers (2024, methodology): “Several works have built on DiT and have made improvements by modifying the architecture to internally include masked prediction layers [22, 70]; these choices are orthogonal to the transition from DiT to SiT studied in this work; they may be fruitfully combined in future work.”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, unknown): “They have been extended to perform diffusion to generate masked patches conditioned on unmasked ones [62, 19].”
An Image is Worth 32 Tokens for Reconstruction and Generation (2024, unknown): “In recent years, image generation has experienced remarkable progress, driven by the significant advancements in both transformers [19, 62, 66, 10, 67, 68] and diffusion models [16, 55, 29, 49, 21].”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, unknown): “…2022), (c) Latent diffusion with transformer+U-Net hybrid models : U-ViT-H/2 (Bao et al., 2023), DiffiT (Hatamizadeh et al., 2024), and MDTv2-XL/2 (Gao et al., 2023), and (d) Latent diffusion with transformers : MaskDiT (Zheng et al., 2024), SD-DiT (Zhu et al., 2024), DiT (Peebles & Xie, 2023),…”