ManiGAN: Text-Guided Image Manipulation

By Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip H. S. Torr. 2019

320 citations recorded by the source corpus.

The goal of our paper is to semantically edit parts of an image matching a given text that describes desired attributes (e.g., texture, colour, and background), while preserving other contents that are irrelevant to the text. To achieve this, we propose a novel generative adversarial network (ManiGAN), which contains two key components: text-image affine combination module (ACM) and detail correction module (DCM). The ACM selects image regions relevant to the given text and then correlates the regions with corresponding semantic words for effective manipulation. Meanwhile, it encodes original image features to help reconstruct text-irrelevant contents. The DCM rectifies mismatched attributes and completes missing contents of the synthetic image. Finally, we suggest a new metric for evaluating image manipulation results, in terms of both the generation of new attributes and the reconstruction of text-irrelevant contents. Extensive experiments on the CUB and COCO datasets demonstrate the superior performance of the proposed method.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Generative Models

Citations in the corpus

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “ManiGAN [22] introduces a novel text-image combination module, which produces high-quality images.”
GAN Inversion: A Survey (2021, methodology): “[8], [71] utilize attention mechanisms to learn image-text matching by correlating finegrained word-level information with the intermediate feature maps, which enhances the detailed attribute modification guided by natural language descriptions.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “To fuse the vision processing ability into such LLMs, several challenges are lying ahead since it is hard to train either large language models or vision models, and the well-designed instructions [4, 55, 21] and cumbersome conversions [30, 52] are required to connect different modalities.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, methodology): “oviding a more explainable search experience that allows users to inspect the result before the retrieval. The closet problem to ours on conditional image generation is text-guided image manipulation [37, 27]. In this setting, a reference image is used to augment the text description such that the g”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, methodology): “Prior work [9, 40, 39, 17, 18] repurposes close-domain image caption datasets [26, 38, 32] for image editing.”
CLIPstyler: Image Style Transfer with a Single Text Condition (2021, methodology): “ManiGAN [14] proposed a modules for simultaneously embedding the text and image features.”
Describe What to Change: A Text-guided Unsupervised Image-to-image Translation Approach (2020, background): “Nam et al. [38] and Li et al. [26] introduced instead TAGAN and ManiGAN that modify fine-grained attributes such as the colours of birds’ wings through text.”
Sound-Guided Semantic Image Manipulation (2021, methodology): “Several studies [10, 26, 29] employed the GAN-based encoder-decoder structure to preserve the features of the image while presenting image manipulations corresponding to the text description.”
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, methodology): “Figure 5: Given video frames (adopted from (Li et al., 2020)) as input, the Video Storytelling Model generates a textual story in sequence.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, background): “There has been remarkable progress since the use of conditional GANs in both text-guided image generation [38,50,51,52], and editing [9,22,29].”
More Control for Free! Image Synthesis with Semantic Diffusion Guidance (2021, methodology): “Various methods have been studied, proposing many different text-adaptive architectures and loss functions to enforce better semantic alignment between the input text and generated image, resulting in significant improvements in editing quality and correctness [55,60,51,27,11,61,33,35,17,28,7,23,40]”
Text-Guided Neural Image Inpainting (2020, methodology): “Image manipulation researchers also explored changing image content according to the text guidance [15, 23].”
PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization (2021, background): “Recent advances on image manipulation techniques [1]–[4] enable easy editing of raw images, such as removing unwanted objects [5]–[8], face swapping [2], attribute changing [9], etc.”
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing (2023, methodology): “Previous methods based on generative adversarial networks (GANs) [18, 15, 38, 22] have achieved some success on domain-specific datasets ( e.g ., face datasets), but they have limited applicability and generality.”
DiffEdit: Diffusion-based semantic image editing with mask guidance (2022, unknown): “Some approaches involve training an end-to-end architecture with a proxy objective before being adapted to editing at inference time, based on GANs (Li et al., 2020b;a; Ma et al., 2018; Alami Mejjati et al., 2018; Mo et al., 2018) or transformers (Wang et al., 2022a; Brown et al., 2022; Issenhuth…”
Language-Driven Image Style Transfer (2021, methodology): “Inspired by text-to-image generation [19, 20, 21], previous works [22, 23, 24, 25, 26, 27, 30] perform LBIE by conditional GAN.”