Generating Images from Captions with Attention

By Elman Mansimov, Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov. 2015

495 citations recorded by the source corpus.

Motivated by the recent progress in generative models, we introduce a model that generates images from natural language descriptions. The proposed model iteratively draws patches on a canvas, while attending to the relevant words in the description. After training on Microsoft COCO, we compare our model with several baseline generative models on image generation and retrieval tasks. We demonstrate that our model produces higher quality samples than other approaches and generates images with novel scene compositions corresponding to previously unseen captions in the dataset.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Image Transformer (2018, background): “In future work we would like to explore a broader variety of conditioning information including free-form text, as previously proposed (Mansimov et al., 2015), and tasks combining modalities such as language-driven editing of images.”
Zero-Shot Text-to-Image Generation (2021, background): “Modern machine learning approaches to text to image synthesis started with the work of Mansimov et al. (2015), who showed that the DRAW Gregor et al. (2015) generative model, when extended to condition on image captions, could also generate novel visual scenes.”
Conditional Image Generation with PixelCNN Decoders (2016, background): “Another promising direction of this work would be to model images based on an image caption instead of class label [15, 19].”
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “Preliminary text-to-image methods conditioned RNN-based DRAW [18] on text [40].”
D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “However, downstream applications of generative models are often based on various conditioning signals, such as labels [61], text descriptions [60], reward values [101], or similarity with existing data [45].”
CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “The attempts to teach machines text-to-image generation can be traced to the early times of deep generative models, when Mansimov et al. [35] added text information to DRAW [20].”
Generative Adversarial Text to Image Synthesis (2016, methodology): “GAN-CLS generates sharper and higher-resolution samples that roughly correspond to the query, but AlignDRAW samples more noticably reﬂect single-word changes in the selected queries from that work.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “It is worth mentioning that the alignDRAW [15] also used LAPGAN [3] to scale the image to a higher resolution.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Another direction that has recently seen large progress in multimodal learning is text-guided image generation [47, 62, 95].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Autoregressive models [39], GANs [79, 84], VQ-VAE Transformer-based methods [55, 22], and diffusion models have seen remarkable progress in text-to-image [59, 43, 59], including the concurrent”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, methodology): “[17] built an AlignDRAW model by learning to estimate alignment between text and the generating canvas.”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “These networks have been used in a wide variety of conditional image generation schemes such as text and spatial structure to image (Mansimov et al., 2015; Reed et al., 2016b;a; Wang & Gupta, 2016).”
Autoencoding beyond pixels using a learned similarity metric (2015, methodology): “Mansimov et al. (2015) append a GAN-based sharpening step to their generative model.”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “There has been much research effort for this task through different approaches, such as variational inference [17, 6], approximate Langevin process [24], conditional PixelCNN via maximal likelihood estimation [26, 24], and conditional generative adversarial networks [23, 22, 32, 33].”
Learning What and Where to Draw (2016, background): “Variational Autoencoders also show some promise for conditional image synthesis, in particular recurrent versions such as DRAW [Gregor et al., 2015, Mansimov et al., 2016].”
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks (2016, background): “These models can also be augmented semantic attributes [35], image captions [24], 3D data [5,17], spatial/temporal status [11,13,27] etc.”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “To generate interpretable images with richer information, the VAE is extended to be conditioned on captions [29] and graphics code [30].”