Text to Image Synthesis Using Generative Adversarial Networks

By Cristian Bodnar. 2018

38 citations recorded by the source corpus.

Generating images from natural language is one of the primary applications of recent conditional generative models. Besides testing our ability to model conditional, highly dimensional distributions, text to image synthesis has many exciting and practical applications such as photo editing or computer-aided content creation. Recent progress has been made using Generative Adversarial Networks (GANs). This material starts with a gentle introduction to these topics and discusses the existent state of the art models. Moreover, I propose Wasserstein GAN-CLS, a new model for conditional image generation based on the Wasserstein distance which offers guarantees of stability. Then, I show how the novel loss function of Wasserstein GAN-CLS can be used in a Conditional Progressive Growing GAN. In combination with the proposed loss, the model boosts by 7.07% the best Inception Score (on the Caltech birds dataset) of the models which use only the sentence-level visual semantics. The only model which performs better than the Conditional Wasserstein Progressive Growing GAN is the recently proposed AttnGAN which uses word-level visual semantics as well.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, background): “convert sketches to images) [24], image blending (mixing selected parts of two images to get a new one) [25], image inpainting (restoring missing pieces of an image) [26] and unconditional [27] and conditional synthesis [28].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, positive): “GANs have already been used successfully in a variety of applications, including image captioning[26, 126], image to image translation [56], text to image synthesis [15, 115], and text generation [36, 145].”
Generative Adversarial Network in Medical Imaging: A Review (2018, positive): “We also expect to witness the utility of GANs in medical image synthesis from text descriptions (Bodnar, 2018), especially for rare cases, so as to ﬁll in the gap of training samples required for training supervised neural networks for medical image classi-ﬁcation tasks.”
Perceptual Pyramid Adversarial Networks for Text-to-Image Synthesis (2019, background): “Nevertheless, these problems are entirely different because text-to-image and image-to-text conversions are highly different cross-modal problems (Bodnar 2018).”