Semantic Image Synthesis via Adversarial Learning

By Hao Dong, Simiao Yu, Chao Wu, Yike Guo. 2017

278 citations recorded by the source corpus.

In this paper, we propose a way of synthesizing realistic images directly with natural language description, which has many useful applications, e.g. intelligent image manipulation. We attempt to accomplish such synthesis: given a source image and a target text description, our model synthesizes images to meet two requirements: 1) being realistic while matching the target text description; 2) maintaining other image features that are irrelevant to the text description. The model should be able to disentangle the semantic information from the two modalities (image and text), and generate new images from the combined semantics. To achieve this, we proposed an end-to-end neural architecture that leverages adversarial learning to automatically learn implicit loss functions, which are optimized to fulfill the aforementioned two requirements. We have evaluated our model by conducting experiments on Caltech-200 bird dataset and Oxford-102 flower dataset, and have demonstrated that our model is capable of synthesizing realistic images that match the given descriptions, while still maintain other features of original images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Some methods [10, 31, 27] use a GAN-based encoderdecoder architecture, to disentangle the semantics of both input images and text descriptions.”
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Compared to L1 loss, which often leads to blurry images [21, 22], the adversarial loss [16] has become a popular choice for many image-to-image tasks [10, 24, 25, 31, 40, 45, 53, 58, 64].”
Controllable Text-to-Image Generation (2019, background): “Recently, thanks to the success of generative adversarial networks (GANs) [4, 6, 15] in generating realistic images, text-to-image generation has made remarkable progress [16, 25, 27] by implementing conditional GANs (cGANs) [5, 16, 17], which are able to generate realistic images conditioned on giv”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, background): “Later on, [7] proposes to bypass the difficulty of learning mappings from text to RGB images Generator”
Generative Adversarial Networks: A Survey and Taxonomy (2019, background): “These properties have allowed GANs to enjoy success especially in the computer vision ﬁeld e.g., plausible image generation [29]–[33], image to image translation [2], [34]–[40], image super-resolution [26], [41]–[44] and image completion [45]–[49].”
Cycle-Consistent Inverse GAN for Text-to-Image Synthesis (2021, methodology): “[8] use an encoder-decoder architecture to take the original images as well as the textual descriptions as the input, and output the manipulated images, which is supervised by a discriminator.”
StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation (2021, background): “Text-based image manipulation methods aim to generate images that contain visual attributes corresponding to the given text input without changing irrelevant attributes [7, 39, 19, 16, 17].”
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021, background): “Text-guided image manipulation [10, 40, 69, 31, 61] semantically edits an image, where the text description and the original image serve as control signals.”
Sound-Guided Semantic Image Manipulation (2021, methodology): “Several studies [10, 26, 29] employed the GAN-based encoder-decoder structure to preserve the features of the image while presenting image manipulations corresponding to the text description.”
DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction (2018, background): “Additional prior information can be discrete labels, text and images [55], [56].”
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, background): “Text-guided image manipulation [10, 41, 69, 31, 62] semantically edits an image, where the text description and the original image serve as control signals.”
Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesis (2019, methodology): “Recent GAN-based text-toimage synthesis [28, 42, 9, 44, 40, 16, 10] design the generative procedure only using semantics from the input text.”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, methodology): “Various methods have been studied, proposing many different text-adaptive architectures and loss functions to enforce better semantic alignment between the input text and generated image, resulting in significant improvements in editing quality and correctness [55,60,51,27,11,61,33,35,17,28,7,23,40]”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “[7] propose an encoderdecoder architecture to modify an image according to a given text.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “On the forefront of image manipulation there are also many works that address text-guided image manipulation [174, 175, 176, 177, 178, 179], which might be a more ﬂexible interface for users than, e.g., editing semantic maps or (a limited amount of) labels.”
ManiGAN: Text-Guided Image Manipulation (2019, methodology): “Our model is evaluated on the CUB bird [34] and more complicated COCO [18] datasets, comparing with two state-of-the-art approaches SISGAN [6] and TAGAN [23] on image manipulation using natural language descriptions.”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, methodology): “Prior work [9, 40, 39, 17, 18] repurposes close-domain image caption datasets [26, 38, 32] for image editing.”