Learning What and Where to Draw

By Scott E. Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, Honglak Lee. 2016

641 citations recorded by the source corpus.

Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers. While existing models can synthesize images based on global constraints such as a class label or caption, they do not provide control over pose or object location. We propose a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location. We show high-quality 128 x 128 image synthesis on the Caltech-UCSD Birds dataset, conditioned on both informal text descriptions and also object location. Our system exposes control over both the bounding box around the bird and its constituent parts. By modeling the conditional distributions over part locations, our system also enables conditioning on arbitrary subsets of parts (e.g. only the beak and tail), yielding an efficient interface for picking part locations.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Zero-Shot Text-to-Image Generation (2021, methodology): “…model architecture with modifications like multi-scale generators (Zhang et al., 2017; 2018), integrating attention and auxiliary losses (Xu et al., 2018), and leveraging additional sources of conditioning information beyond just text (Reed et al., 2016a; Li et al., 2019; Koh et al., 2021).”
Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “The image-conditional models have tackled image prediction from a normal map [52], future frame prediction [37], product photo generation [56], and image generation from sparse annotations [28, 45] (c.”
Unrolled Generative Adversarial Networks (2016, methodology): “The flexibility of the GAN framework has also enabled a number of successful extensions of the technique, for instance for structured prediction (Reed et al., 2016a;b; Odena et al., 2016), training energy based models (Zhao et al., 2016), and combining the GAN loss with a mutual information loss…”
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, background): “Since previous generative models failed to add the location information, Reed et al. proposed GAWWN [21] to encode localization constraints.”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “Inception scores by state-of-the-art GAN models [20, 18, 36, 37, 16] and our AttnGAN on CUB and COCO test sets.”
Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis (2017, background): “It has also been empirically shown [2–4, 8] that adding auxiliary tasks seems to reduce such a tendency and improve convergence stability; side information is therefore often used as either additional information to the discriminator/generator [4, 5, 8], as a quantity to be reconstructed by the dis”
Conditional Image Synthesis with Auxiliary Classifier GANs (2016, background): “Richer side information such as image captions and bounding box localizations may improve sample quality further (Reed et al., 2016a;b).”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “Several other works propose using hierarchical models that generate images by explicitly modeling the location and semantics of objects [46, 70, 71, 72].”
Self-Attention Generative Adversarial Networks (2018, background): “GANs have achieved great success in various image generation tasks, including image-to-image translation [9, 40, 29, 14], image super-resolution [12, 28] and text-to-image synthesis [24, 23, 37].”
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks (2016, methodology): “Two state-of-the-art methods on text-to-image synthesis, GAN-INT-CLS [26] and GAWWN [24], are compared.”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “These networks have been used in a wide variety of conditional image generation schemes such as text and spatial structure to image (Mansimov et al., 2015; Reed et al., 2016b;a; Wang & Gupta, 2016).”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “There has been much research effort for this task through different approaches, such as variational inference [17, 6], approximate Langevin process [24], conditional PixelCNN via maximal likelihood estimation [26, 24], and conditional generative adversarial networks [23, 22, 32, 33].”
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks (2017, methodology): “However, as mentioned by its authors, GAWWN fails to generate any plausible images when it is only conditioned on text descriptions [29].”
Improving GANs Using Optimal Transport (2018, methodology): “To further demonstrate the effectiveness of the proposed method on conditional image synthesis, we compare OT-GAN with state-of-the-art models on text-to-image generation (Reed et al., 2016b;a; Zhang et al., 2017).”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “[36] synthesize 128×128 images of birds and people conditioned on text descriptions and on spatial constraints such as bounding boxes or keypoints.”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, background): “Recently, approaches based on conditional Generative Adversarial Network (GAN) [7] have shown promising results in text-toimage synthesis [21, 23, 34, 6, 4].”
Invertible Conditional GANs for image editing (2016, background): “Additionally, the most advanced and recent work on cGANs trains a model to generate realistic images from text descriptions and landmarks [11].”
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence (2018, methodology): “MHP dataset also comes with sentence descriptions provided by [27].”
Few-Shot Adversarial Domain Adaptation (2017, background): “For example, [43, 44, 64] concatenate the input text to the penultimate layers of the discriminators.”