Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think

By Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, Saining Xie. International Conference on Learning Representations, 2024

621 citations recorded by the source corpus.

Recent studies have shown that the denoising process in (generative) diffusion models can induce meaningful (discriminative) representations inside the model, though the quality of these representations still lags behind those learned through recent self-supervised learning methods. We argue that one main bottleneck in training large-scale diffusion models for generation lies in effectively learning these representations. Moreover, training can be made easier by incorporating high-quality external visual representations, rather than relying solely on the diffusion models to learn them independently. We study this by introducing a straightforward regularization called REPresentation Alignment (REPA), which aligns the projections of noisy input hidden states in denoising networks with clean image representations obtained from external, pretrained visual encoders. The results are striking: our simple strategy yields significant improvements in both training efficiency and generation quality when applied to popular diffusion and flow-based transformers, such as DiTs and SiTs. For instance, our method can speed up SiT training by over 17.5$\times$, matching the performance (without classifier-free guidance) of a SiT-XL model trained for 7M steps in less than 400K steps. In terms of final generation quality, our approach achieves state-of-the-art results of FID=1.42 using classifier-free guidance with the guidance interval.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f5ac95fa58815a35622a60f42a5729f518e57cdf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jingfeng Yao
2. Xinggang Wang

Citations in the corpus

• Mean Flows for One-step Generative Modeling (2025, neutral): “Orthogonal improvements, such as REPA [51], are applicable, which are left for future work.”
• Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, positive): “Previous great work MDT [12] or REPA [45] achieved similar convergence performance with the help of Mask Image Modeling (MIM) and representation alignment.”