Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
By Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel and 7 more. International Conference on Machine Learning, 2024
4,349 citations recorded by the source corpus.
Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available.
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/41a66997ce0a366bba3becf7c3f37c9aebb13fbd
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Robin Rombach
- Andreas Blattmann
- Tim Dockhorn
- Mike Zheng Shou
- Axel Sauer
- Dominik Lorenz
- Yi Jiang
- Bingyue Peng
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Alan L. Yuille
- Bernt Schiele
Citations in the corpus
- VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “…et al., 2024; Chen et al., 2023d; Liu et al., 2024c; Chen et al., 2023b; Lin et al., 2023c; Young et al., 2024) and photorealistic image generation (Esser et al., 2024; Saharia et al., 2022; Ramesh et al., 2021) have approached near-human capabilities, catalyzing major break-throughs in sectors…”
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, positive): “Different from DDPM, flow matching models (Esser et al., 2024; Lipman et al., 2022; Liu et al., 2023) deal with the continuous time-dependent process with a data x ∗ ∼ p ( x ) and a Gaussian noise ϵ ∼ N ( 0 , I ) on t ∈ [0 , 1] : where α t and σ t are a decreasing and increasing function of t…”
- Flow-GRPO: Training Flow Matching Models via Online RL (2025, positive): “M) [4] from 63% to 95% on GenEval, outperforming the state-of-the-art GPT-4o [18] model.”
- Structured 3D Latents for Scalable and Versatile 3D Generation (2024, neutral): “Similar to the challenges reported in SD3 [9], we encounter training instability caused by the ex-ploding norms of queries and keys within the multi-head attention blocks.”
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “RL has proven to be an effective learning paradigm in text and image processing [Schulman et al., 2017, Rafailov et al., 2024, Sutton et al., 1999, Esser et al., 2024, Wallace et al., 2023].”
- Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, neutral): “Similarly, due to the SD3-Medium weight initialization, which is infamous for its human structure, our method achieves a relatively low score in human action, which could be addressed by switching to other base models or training from scratch.”
- OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, positive): “These approaches yield suboptimal results when applied to newer Diffusion Transformer (DiT) models [40] due to fundamental architectural differences, despite the lat-ter’s superior generation capabilities [4, 9, 24].”
- Emu3: Next-Token Prediction is All You Need (2024, methodology): “Following SD3 [22], we evaluate text-to-image generation capability of Emu3 on the GenEval benchmark [26].”
- VACE: All-in-One Video Creation and Editing (2025, neutral): “Beyond the early foundational pre-trained models for text-to-image [7, 16, 33] or text-to-video [9, 22, 64] generation in the field, there has been a proliferation of down-stream tasks and applications, such as repainting [3, 82], editing [4, 42, 68, 70, 75], controllable generation [30, 76], frame…”
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, methodology): “Then, we employ the image recaptioning model CogVLM (Wang et al., 2023a) used in Stable Diffusion 3 (Esser et al., 2024) and CogView3 (Zheng et al., 2024a) to create dense image captions for each of the frames within a video.”
- Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding (2024, methodology): “Diffusion-based text-to-image generative models, such as DALL-E [5], Stable Diffusion [24, 10] and Pixart [7], have shown the ability to generate images with unprecedented quality.”
- Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model (2024, background): “While Transfusion does lag behind SD 3 [Esser et al., 2024a], this model leveraged synthetic image captions through backtranslation [Betker et al., 2023], which enhances its GenEval performance by 6.5% absolute (0.433 → 0.498) at smaller scale; for simplicity, our experimental setup only included…”
- FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space (2025, positive): “2 FLUX.1 FLUX.1 is a rectified flow transformer [12, 30, 32] trained in the latent space of an image autoen-coder [42].”
- Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “Further, Show-o, with only 1.3B parameters, achieves comparable performance to models with two times larger number of parameters such as SDXL (2.6B) and SD3 (2B).”
- DanceGRPO: Unleashing GRPO on Visual Generation (2025, neutral): “Recent advances in generative models—particularly diffusion models [1–4] and rectified flows [5–7]—have transformed visual content creation by improving output quality and versatility in image and video generation.”
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (2025, neutral): “Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. capability is to increase the feature dimension of visual to-kens, which effectively expands the information capacity of the latent representation.”
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “…model surpasses the recently popular diffusion models Diffusion Transformer (DiT), which serves as the precursor to the latest Stable-Diffusion 3 [18] and SORA [8], in multiple dimensions: 1) In image generation diversity and quality (FID and IS), VAR with 2B parameters consistently performs…”
- SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “The most commonly used autoen-coders in previous latent diffusion works (Peebles & Xie, 2023; Bao et al., 2022; Cai et al., 2024; Esser et al., 2024; Dai et al., 2023; Chen et al., 2024b;a) feature a down-sampling factor of F = 8 , mapping images from pixel space R H × W × 3 to latent space R H 8 ×…”
- One Step Diffusion via Shortcut Models (2024, neutral): “…2020; Song et al., 2020) and flow-matching (Lipman et al., 2022; Liu et al., 2022) have seen remarkable success in modelling diverse images (Rombach et al., 2022; Esser et al., 2024), video (Ho et al., 2022; Bar-Tal et al., 2024), audio (Kong et al., 2020), and proteins (Abramson et al., 2024).”
- Mean Flows for One-step Generative Modeling (2025, positive): “Since its introduction, Flow Matching has seen widespread adoption in modern generative modeling [10, 33, 35].”