Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

By Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan and 6 more. 2022

1,537 citations recorded by the source corpus.

We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements. See https://parti.research.google/ for high-resolution images.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Ming-Hsuan Yang
  3. Dacheng Tao
  4. Trevor Darrell
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrea Vedaldi
  8. Andrew Zisserman

In Generative Models

  1. Jun-Yan Zhu
  2. Stefano Ermon
  3. Arash Vahdat
  4. Richard Zhang
  5. Eli Shechtman
  6. Karsten Kreis
  7. Alexei A. Efros
  8. Jiaming Song

In Multimodal Learning

  1. Mohit Bansal
  2. Zhe Gan
  3. Jianfeng Gao
  4. Devi Parikh
  5. Aniruddha Kembhavi
  6. Chunyuan Li
  7. Lijuan Wang
  8. Jaemin Cho

In Natural Language Generation

  1. William Yang Wang
  2. Jason Baldridge
  3. Mohit Bansal
  4. Ruslan Salakhutdinov
  5. Lior Wolf
  6. Percy Liang
  7. Ron Mokady
  8. Devi Parikh

Citations in the corpus