Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language

By Seonghyeon Nam, Yunji Kim, Seon Joo Kim. 2018

225 citations recorded by the source corpus.

This paper addresses the problem of manipulating images using natural language description. Our task aims to semantically modify visual attributes of an object in an image according to the text describing the new visual appearance. Although existing methods synthesize images having new attributes, they do not fully preserve text-irrelevant contents of the original image. In this paper, we propose the text-adaptive generative adversarial network (TAGAN) to generate semantically manipulated images while preserving text-irrelevant contents. The key to our method is the text-adaptive discriminator that creates word level local discriminators according to input text to classify fine-grained attributes independently. With this discriminator, the generator learns to generate images where only regions that correspond to the given text is modified. Experimental results show that our method outperforms existing methods on CUB and Oxford-102 datasets, and our results were mostly preferred on a user study. Extensive analysis shows that our method is able to effectively disentangle visual attributes and produce pleasing outputs.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Multimodal Learning

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, methodology): “Some methods [10, 31, 27] use a GAN-based encoderdecoder architecture, to disentangle the semantics of both input images and text descriptions.”
Controllable Text-to-Image Generation (2019, methodology): “To address the issue, we propose a novel word-level discriminator inspired by [12] to explore the correlation between image subregions and each word; see Fig.”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “The text-adaptive GAN [224] facilitates the semantic modification of input images of birds and flowers based on natural language.”
M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers (2021, background): “Text-guided image manipulation [10, 41, 69, 31, 62] semantically edits an image, where the text description and the original image serve as control signals.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, methodology): “Closest to ours are the TA-GAN [38] and ManiGAN [27] schemes that take the image caption as input to describe attributes for conditional image manipulation.”
Adversarial Text-to-Image Synthesis: A Review (2021, background): “On the forefront of image manipulation there are also many works that address text-guided image manipulation [174, 175, 176, 177, 178, 179], which might be a more ﬂexible interface for users than, e.g., editing semantic maps or (a limited amount of) labels.”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, background): “ [62], ACGAN [63], TACGAN [64], SISGAN Human pose synthesis:PG[1262[65]], Deformable-GAN [66] Stenographic applications: S-GAN [67], SS-GAN [68], Stegano-GAN [69] Image manipulation: IGAN [70], TAGAN [71], IAN [72], AttGAN [73], DGAN [234] Visual saliency prediction: Sal-GAN [74], SalCapsule-CGAN [7”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “Previous methods based on generative adversarial networks (GANs) [18, 15, 37, 22] have achieved some success on domain-specific datasets (e.”
StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation (2021, background): “Text-based image manipulation methods aim to generate images that contain visual attributes corresponding to the given text input without changing irrelevant attributes [7, 39, 19, 16, 17].”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Image Manipulation take a different path from the earlier benchmark approaches about image generation, the TAGAN (Nam et al., 2018) was introduced to generate semantically manipulated images while preserving text-irrelevant contents.”
UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis (2021, background): “Text-guided image manipulation [10, 40, 69, 31, 61] semantically edits an image, where the text description and the original image serve as control signals.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, background): “There has been remarkable progress since the use of conditional GANs in both text-guided image generation [35,47,48,49], and editing [7,20,27].”
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU (2023, background): “Generative Adversarial Networks (GANs) have introduced numerous applications across various domains, including image blending [67], 3D object generation [68], face aging [69], medicine [70, 71], steganography [72], image manipulation [73], text transfer [74], language and speech synthesis [75], traf”
Language-Driven Image Style Transfer (2021, methodology): “Our LDIST task is different from the general language-based image-editing (LBIE) [25, 26, 30, 28], which usually alters objects or properties of object in an image.”