Generative Adversarial Text to Image Synthesis
By Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee. 2016
3,433 citations recorded by the source corpus.
Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal. However, in recent years generic and powerful recurrent neural network architectures have been developed to learn discriminative text feature representations. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to generate highly compelling images of specific categories, such as faces, album covers, and room interiors. In this work, we develop a novel deep architecture and GAN formulation to effectively bridge these advances in text and image modeling, translating visual concepts from characters to pixels. We demonstrate the capability of our model to generate plausible images of birds and flowers from detailed text descriptions.
Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Yi Yang
- Bernt Schiele
In Generative Models
- Jun-Yan Zhu
- Richard Zhang
- Stefano Ermon
- Eli Shechtman
- Alexei A. Efros
- Ming-Hsuan Yang
- Yoshua Bengio
- Daniel Cohen-Or
In Multimodal Learning
Citations in the corpus
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “This can be implemented with a conditional denoising autoencoder ǫθ(zt, t, y) and paves the way to controlling the synthesis process through inputs y such as text [66], semantic maps [32, 59] or other image-to-image translation tasks [33].”
- Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, methodology): “This attention mechanism can also be straightforwardly applied to the multiscale PixelCNN architecture of Reed et al. (2017). In that model, pixel factors P (xt|x<t, ft(s, x<t)) are simply replaced by pixel group factors P (xg|x<g, fg(s, x<g)), where g indexes a set of pixels and < g indicates all p”
- Zero-Shot Text-to-Image Generation (2021, methodology): “…model architecture with modifications like multi-scale generators (Zhang et al., 2017; 2018), integrating attention and auxiliary losses (Xu et al., 2018), and leveraging additional sources of conditioning information beyond just text (Reed et al., 2016a; Li et al., 2019; Koh et al., 2021).”
- Conditional Image Generation with PixelCNN Decoders (2016, background): “Another promising direction of this work would be to model images based on an image caption instead of class label [15, 19].”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, neutral): “Text-conditioned GANs provided additional improvement [48].”
- Generative Adversarial Networks (2021, neutral): “This approach has potential applications in fashion design, interior design, and other areas where the ability to produce images from textual descriptions can be useful [20].”
- CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders (2021, background): “In recent years, focus has been on methods that aim to learn a direct text-to-image mapping function, often in the form of a conditional GAN (Goodfellow et al., 2014a; Mirza & Osindero, 2014; Reed et al., 2016; Frolov et al., 2021).”
- D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation (2021, background): “These properties c include (but are not limited to2) labels [61], text descriptions [60, 76], noisy or partial observations [11, 5, 47, 24], and manipulation constraints [69].”
- CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “Reed et al. [42] fed the text embeddings to both generator and discriminator as extra inputs.”
- Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, methodology): “, class information [3], [5], [6], [9], [36], source image [10], [12], [13], [37], text description [16], [17], [38], [39], scene graph [18], [40], etc) as input.”
- Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Researchers have explored various models for generating images based on text [18,41,49, 52].”
- Image-to-Image Translation with Conditional Adversarial Networks (2016, background): “Prior and concurrent works have conditioned GANs on discrete labels [38, 21, 12], text [43], and, indeed, images.”
- Unrolled Generative Adversarial Networks (2016, methodology): “The flexibility of the GAN framework has also enabled a number of successful extensions of the technique, for instance for structured prediction (Reed et al., 2016a;b; Odena et al., 2016), training energy based models (Zhao et al., 2016), and combining the GAN loss with a mutual information loss…”
- Improving Text-to-Image Synthesis Using Contrastive Learning (2021, background): “Recently, a great number of studies [25, 45, 46, 42, 13, 47, 23, 48, 43, 18, 22, 17, 1, 12, 6, 19, 33, 4, 24, 32, 14, 36, 44] present promising results on the text-to-image synthesis task, most of which make use of GANs as the back-bone model.”
- StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Additional supervision was used in other works [37, 25, 20] to further improve the image quality.”
- Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Generative Adversarial Networks (GANs) [12] have shown promising results on text-to-image generation [44, 61, 62], using a conditional GAN formulation [11].”
- DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-To-Image Synthesis (2019, methodology): “Dataset GAN-INT-CLS [20] GAWWN [21] StackGAN [30] PPGN [16] AttnGAN [28] DM-GAN”
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “Inception scores by state-of-the-art GAN models [20, 18, 36, 37, 16] and our AttnGAN on CUB and COCO test sets.”