Text as Neural Operator:Image Manipulation by Text Instruction

By Tianhao Zhang, Hung-Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, Irfan Essa. 2020

41 citations recorded by the source corpus.

n recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community. The input to conditional image generation has evolved from image-only to multimodality. In this paper, we study a setting that allows users to edit an image with multiple objects using complex text instructions to add, remove, or change the objects. The inputs of the task are multimodal including (1) a reference image and (2) an instruction in natural language that describes desired modifications to the image. We propose a GAN-based method to tackle this problem. The key idea is to treat text as neural operators to locally modify the image feature. We show that the proposed model performs favorably against recent strong baselines on three public datasets. Specifically, it generates images of greater fidelity and semantic relevance, and when used as a image query, leads to better retrieval performance.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Huan Sun
2. Kai Zhang
3. Yu Su
4. Lingbo Mo
5. Wenhu Chen

In Generative Models

1. Wenhu Chen

In Multimodal Learning

1. Huan Sun
2. Yu Su
3. Wenhu Chen

Citations in the corpus

• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “To fuse the vision processing ability into such LLMs, several challenges are lying ahead since it is hard to train either large language models or vision models, and the well-designed instructions [4, 55, 21] and cumbersome conversions [30, 52] are required to connect different modalities.”
• MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing (2023, background): “Instruction-guided editing, as initially proposed in various studies [11, 13, 42], enables users to edit images without requiring elaborate descriptions or region masking.”