Zero-Shot Text-to-Image Generation
By Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever. 2021
6,590 citations recorded by the source corpus.
Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.
Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Andrea Vedaldi
In Generative Models
- Jun-Yan Zhu
- Stefano Ermon
- Richard Zhang
- Eli Shechtman
- Arash Vahdat
- Daniel Cohen-Or
- Alexei A. Efros
- Karsten Kreis
In Multimodal Learning
Citations in the corpus
- Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “Most recently, BEiT [2] proposes to predict discrete tokens [37, 43].”
- Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, methodology): “Like GLIDE and DALL-E, unCLIP is not directly trained on the MS-COCO training set, but can still generalize to the validation set zero-shot.”
- Evaluating Large Language Models Trained on Code (2021, background): “…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
- GSPMD: General and Scalable Parallelization for ML Computation Graphs (2021, background): “GSPMD as a shared, robust mechanism for different parallelism patterns is particularly relevant moving forward because the ML community is increasingly investing into multimodality, where text, image and audio are combined into a single model [31].”
- Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “(4) We try to standardize pre-training data to ImageNet-1k, however BEiT uses the DALL · E [32] discrete VAE (dVAE), which was trained on ∼ 250 million proprietary and undisclosed images, as an image tokenizer.”
- BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Refer to (Ramesh et al., 2021) for more training details of the image tokenizer.”
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (2021, methodology): “We also compute FID against a subset of the MS-COCO validation set that has been purged of all images similar to images in our training set, as done by Ramesh et al. (2021).”
- Generative Adversarial Networks (2021, neutral): “GANs have emerged as a useful technique for producing realistic data in a variety of disciplines ranging from computer vision and graphics to NLP and audio synthesis.”
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “1 LAnguage-Free traIning for Text-to-image gEneration supervised text-to-image generation [53, 56, 59] and zeroshot text-to-image generation [7, 38].”
- LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs (2021, methodology): “Multi-modal language-vision models demonstrated recently strong transfer capability to novel datasets in absense of per-sample labels [1, 2, 3].”
- Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “C V ] 5 J un 2 02 2 Remarkable image generation results have been achieved by pre-quantizing images into discrete latent variables and modeling them autoregressively, including VQVAE (Oord et al., 2017), DALLE (Ramesh et al., 2021) and VQGAN (Esser et al., 2021).”
- Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “[1, 89, 65]), and the shape and structure of a task’s inputs and outputs may place strong constraints on how such a system processes data, making it difficult to adapt to new settings.”
- CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, background): “, 2020), with the recent DALL-E model (Ramesh et al., 2021) showcasing the benefit of scaling text-to-image synthesis networks to a large capacity.”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, methodology): “, DALL-E [74] and CLIP [73] over image-text pairs).”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “To evaluate the performance, we conduct a human evaluation to make comparisons between various methods, similar to previous works [27, 39].”
- High-Resolution Complex Scene Synthesis with Transformers (2021, methodology): “Recent years have brought unprecedented advances to conditional image generation with neural networks, e.g . in class-conditional [4, 24], text-conditional [23, 34] or pixel-wise conditional settings [20, 33, 30].”
- Diffusion Models Beat GANs on Image Synthesis (2021, background): “These models can be used in a variety of ways, such as generating images from text prompts [78, 56] or learning useful feature representations [18, 10].”