Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
By Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan and 6 more. 2022
1,537 citations recorded by the source corpus.
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements. See https://parti.research.google/ for high-resolution images.
Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Jun-Yan Zhu
- Stefano Ermon
- Arash Vahdat
- Richard Zhang
- Eli Shechtman
- Karsten Kreis
- Alexei A. Efros
- Jiaming Song
In Multimodal Learning
In Natural Language Generation
- William Yang Wang
- Jason Baldridge
- Mohit Bansal
- Ruslan Salakhutdinov
- Lior Wolf
- Percy Liang
- Ron Mokady
- Devi Parikh
Citations in the corpus
- Visual Instruction Tuning (2023, background): “…foundation vision models [27, 16], with strong capabilities in open-world visual understanding such as classification [40, 21, 57, 54, 39], detection [29, 62, 33], segmentation [25, 63, 58] and captioning [50, 28], as well as visual generation and editing [42, 43, 56, 15, 44, 30].”
- Fast Inference from Transformers via Speculative Decoding (2022, background): “Large autoregressive models, notably large Transformers (Vaswani et al., 2017), are much more capable than smaller models, as is evidenced countless times in recent years e.g., in the text or image domains, like GPT-3 (Brown et al., 2020), LaMDA (Thoppilan et al., 2022), Parti (Yu et al., 2022), and”
- Scalable Diffusion Models with Transformers (2022, background): “They have also been trained on discrete codebooks [59] as both autoregressive models [11, 47] and masked generative models [4, 14]; the former has shown excellent scaling behavior up to 20B parameters [62].”
- Visual Programming: Compositional visual reasoning without training (2022, methodology): “Text to image generation has made impressive strides over the last few years with models like DALL-E [23], Parti [32], and Stable Diffusion [25].”
- LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Another recent success of multimodal learning is in image generation, where DALL-E [59] and later models [52, 60, 64, 66, 90] demonstrated the potential of text-guided image generation by producing high-quality images specific to the provided text.”
- Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “, 2021), image synthesis (Esser et al., 2020; Ramesh et al., 2021; Yu et al., 2022) and protein folding (Jumper et al.”
- BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, background): “In CV, pre-trained models that learn from hundreds of millions of image-caption pairs have achieved top performance on image-to-text benchmarks [7–9], while delivering remarkably photo-realistic results in text-to-image tasks [10–14].”
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “…[Vaswani et al., 2017] trained to predict sequences of discrete tokens — have been subsequently extended to continuous, natural signals with images [Yu et al., 2022b] or audio waveforms [Lakhotia et al., 2021, Kreuk et al., 2022, Wang et al., 2023] being converted into a stream of discrete…”
- High-Fidelity Audio Compression with Improved RVQGAN (2023, background): “This approach has shown success across image [44, 43, 31], audio [5, 40], video and music [9, 1] domains.”
- AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “This approach has been used to generate high-resolution images [11], [12], [42] and long videos [43].”
- MusicLM: Generating Music From Text (2023, background): “In contrast, and inspired by progress in text-to-image generation (Ramesh et al., 2021; 2022; Saharia et al., 2022; Yu et al., 2022), recent work has explored generating audio from sequence-wide, high-level captions (Yang et al., 2022; Kreuk et al., 2022) such as “whistling with wind blowing” .”
- Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “, 2022), multimodal applications (Yu et al., 2022), retrieval (Tay et al.”
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models (2023, background): “However, autoregressive models often require large parameters and computing resources to generate high-quality images, as seen in Parti [19].”
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Image generation language models can generate images autoregres-sively [74] or via masked prediction [12, 13].”
- Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, positive): “Both for ours and the competing models, we generate samples based on prompts from the commonly used PartiPrompts benchmark [64].”
- Shifted Diffusion for Text-to-image Generation (2022, neutral): “[7, 20–23, 26, 32].”
- Underspecification in Scene Description-to-Depiction Tasks (2022, background): “E 2) (Ramesh et al., 2022), Imagen (Saharia et al., 2022), and Parti (Yu et al., 2022)—the papers introducing these models rely on the broadest task formulation, wherein the model takes a textual prompt of any kind and produces an image of any kind.”
- Text-to-image Diffusion Models in Generative AI: A Survey (2023, methodology): “scale data for text-to-image generation, with representative methods including DALL-E [11] from OpenAI and Parti [14] from Google.”