PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis

By Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu and 1 more. International Conference on Learning Representations, 2023

927 citations recorded by the source corpus.

The most advanced text-to-image (T2I) models require significant training costs (e.g., millions of GPU hours), seriously hindering the fundamental innovation for the AIGC community while increasing CO2 emissions. This paper introduces PIXART-$\alpha$, a Transformer-based T2I diffusion model whose image generation quality is competitive with state-of-the-art image generators (e.g., Imagen, SDXL, and even Midjourney), reaching near-commercial application standards. Additionally, it supports high-resolution image synthesis up to 1024px resolution with low training cost, as shown in Figure 1 and 2. To achieve this goal, three core designs are proposed: (1) Training strategy decomposition: We devise three distinct training steps that separately optimize pixel dependency, text-image alignment, and image aesthetic quality; (2) Efficient T2I Transformer: We incorporate cross-attention modules into Diffusion Transformer (DiT) to inject text conditions and streamline the computation-intensive class-condition branch; (3) High-informative data: We emphasize the significance of concept density in text-image pairs and leverage a large Vision-Language model to auto-label dense pseudo-captions to assist text-image alignment learning. As a result, PIXART-$\alpha$'s training speed markedly surpasses existing large-scale T2I models, e.g., PIXART-$\alpha$ only takes 10.8% of Stable Diffusion v1.5's training time (675 vs. 6,250 A100 GPU days), saving nearly \$300,000 (\$26,000 vs. \$320,000) and reducing 90% CO2 emissions. Moreover, compared with a larger SOTA model, RAPHAEL, our training cost is merely 1%. Extensive experiments demonstrate that PIXART-$\alpha$ excels in image quality, artistry, and semantic control. We hope PIXART-$\alpha$ will provide new insights to the AIGC community and startups to accelerate building their own high-quality yet low-cost generative models from scratch.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8fafd95a6ffbecf9c1b5f4542ac4b78a00602551

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation (2024, methodology): “Models are finetuned on the PixArt-α ’s 256px pre-trained checkpoint with the position embedding interpolation trick [5].”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “Consequently, mainstream text-to-image diffusion models [6, 12, 22] have adopted the DiT architecture due to its effectiveness.”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “…transformers akin to GPT-2 and VQ-GAN [49, 19], with the only modification of substituting traditional layer normalization for adaptive normalization (AdaLN) – a choice motivated by its widespread adoption and proven effectiveness in visual generative models [33, 34, 32, 57, 56, 29, 46, 12].”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Recently, OpenSora 2 , trained based on a pretrained T2I model [25] and a large text-to-video dataset, aims to reproduce Sora.”
Qwen-Image Technical Report (2025, neutral): “…both text-to-image generation (T2I) (Rombach et al., 2021; Wu et al., 2022; Liang et al., 2022; OpenAI, 2023; Podell et al., 2023; Chen et al., 2024c; Li et al., 2024b; Esser et al., 2024; BlackForest, 2024; Gao et al., 2025; Gong et al., 2025; Cai et al., 2025) and image editing…”
VACE: All-in-One Video Creation and Editing (2025, neutral): “Beyond the early foundational pre-trained models for text-to-image [7, 16, 33] or text-to-video [9, 22, 64] generation in the field, there has been a proliferation of down-stream tasks and applications, such as repainting [3, 82], editing [4, 42, 68, 70, 75], controllable generation [30, 76], frame…”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, methodology): “In the example at the top of Fig.6, Chameleon, SEED-X, and Show-o all provide comprehensive descriptions of the image’s main content.”
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, methodology): “Speciﬁcally, at each step of diffusion, we will make two predictions using positive prompts pos and negative prompts neg , and take the fusion of these two results as the ﬁnal output z t − 1 : , where H ( · ) is our diffusion model with adaptor, σ t is the variance of the noise at time-step t , and…”
OminiControl: Minimal and Universal Control for Diffusion Transformer (2024, neutral): “Diffusion models have emerged as a powerful framework for image generation[17, 44], demonstrating success across diverse tasks including text-to-image synthesis [4, 44, 49], image-to-image translation [48], and image editing [1, 35].”
Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, positive): “Based on these improvements, Pixart-α (Chen et al., 2024b), Pixart-Σ (Chen et al., 2024a), Stable diffusion 3 (Esser et al., 2024) show pure transformers can be scaled up for challenging text-to-image generation, and CMD (Yu et al., 2024), WALT (Gupta et al., 2024), and Sora (Brooks et al., 2024) de”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, methodology): “5, v2 and XL [40] model, Composable v2 [35], Structured v2 [19], Attn-Exct v2 [11], GORS [27], PixArt-α [12] and DALL-E 2 [45].”
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “With the rapid development of generative models, particularly diffusion models [26, 47], numerous breakthroughs have been achieved in fields such as image generation [11, 12, 16, 19, 39, 41, 43, 57] and video generation [10, 24, 27, 45, 50], as well as in recognition and detection tasks [13, 37].”
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, methodology): “To integrate the text condition into autoregressive models, we use FLAN-T5 XL [Chung et al. 2024] as the text encoder, the encoded text feature is projected by an additional MLP [Chen et al. 2023b,c] and is used as prefilling token embedding in autoregressive models.”
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, neutral): “In the field of visual generation, diffusion-based approaches [9, 20, 20, 64] have seen notable success.”
LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “Recently, DiT (Peebles & Xie, 2023) has started to replace UNet to become the novel architecture for text-to-image diffusion models (Chen et al., 2024; Esser et al., 2024).”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “On one hand, there is a growing consensus among researchers regarding several key points: (1) Replace U-Net with Transformer architectures (Chen et al., 2024b;a; Esser et al., 2024; Labs, 2024), (2) Using Vision Language Models (VLM) for auto-labelling images (Chen et al., 2024b; OpenAI, 2023; Zhuo…”
Structured 3D Latents for Scalable and Versatile 3D Generation (2024, neutral): “More Implementation Details”