Text-Guided Neural Image Inpainting

By Lisai Zhang, Qingcai Chen, Baotian Hu, Shuoran Jiang. 2020

67 citations recorded by the source corpus.

Image inpainting task requires filling the corrupted image with contents coherent with the context. This research field has achieved promising progress by using neural image inpainting methods. Nevertheless, there is still a critical challenge in guessing the missed content with only the context pixels. The goal of this paper is to fill the semantic information in corrupted images according to the provided descriptive text. Unique from existing text-guided image generation works, the inpainting models are required to compare the semantic content of the given text and the remaining part of the image, then find out the semantic content that should be filled for missing part. To fulfill such a task, we propose a novel inpainting model named Text-Guided Dual Attention Inpainting Network (TDANet). Firstly, a dual multimodal attention mechanism is designed to extract the explicit semantic information about the corrupted regions, which is done by comparing the descriptive text and complementary image areas through reciprocal attention. Secondly, an image-text matching loss is applied to maximize the semantic similarity of the generated image and the text. Experiments are conducted on two open datasets. Results show that the proposed TDANet model reaches new state-of-the-art on both quantitative and qualitative measures. Result analysis suggests that the generated images are consistent with the guidance text, enabling the generation of various results by providing different descriptions. Codes are available at https://github.com/idealwhite/TDANet

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Adversarial Text-to-Image Synthesis: A Review (2021, background): “On the forefront of image manipulation there are also many works that address text-guided image manipulation [174, 175, 176, 177, 178, 179], which might be a more ﬂexible interface for users than, e.g., editing semantic maps or (a limited amount of) labels.”
Text as Neural Operator:Image Manipulation by Text Instruction (2020, background): “Other works include text-guided image inpainting [45, 76, 79] which use image caption to inpaint incomplete images.”
SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model (2022, unknown): “TDANet [29] proposes a dual attention mechanism to exploit the text features about the masked region by comparing text with the corrupted image and its counterpart.”
Prompt Engineering for Text-Based Generative Art (2022, background): “Some GAN-based systems also allow image inpainting [36] which offers a level of interactivity beyond mere textual input prompts.”