Coherent Semantic Attention for Image Inpainting

By Hongyu Liu, Bin Jiang, Yi Xiao, Chao Yang. 2019

400 citations recorded by the source corpus.

The latest deep learning-based approaches have shown promising results for the challenging task of inpainting missing regions of an image. However, the existing methods often generate contents with blurry textures and distorted structures due to the discontinuity of the local pixels. From a semantic-level perspective, the local pixel discontinuity is mainly because these methods ignore the semantic relevance and feature continuity of hole regions. To handle this problem, we investigate the human behavior in repairing pictures and propose a fined deep generative model-based approach with a novel coherent semantic attention (CSA) layer, which can not only preserve contextual structure but also make more effective predictions of missing parts by modeling the semantic relevance between the holes features. The task is divided into rough, refinement as two steps and we model each step with a neural network under the U-Net architecture, where the CSA layer is embedded into the encoder of refinement step. Meanwhile, we further propose consistency loss and feature patch discriminator to stabilize the network training process and improve the details. The experiments on CelebA, Places2, and Paris StreetView datasets have validated the effectiveness of our proposed methods in image inpainting tasks and can obtain images with a higher quality as compared with the existing state-of-the-art approaches. The codes and pre-trained models will be available at https://github.com/KumapowerLIU/CSA-inpainting.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Alternative attention mechanisms were suggested in [28, 47, 65].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Although some multi-modal unpaired image-to-image translation methods propose to encode the style from another reference image (Huang et al., 2018; Liu et al., 2019b), this unreasonably assumes that the style representations are entirely independent of the conditional input and hence compromises…”
High-Fidelity Pluralistic Image Completion with Transformers (2021, methodology): “In recent years, CNN-based solutions [22, 14, 18, 27, 36, 21] started to dominate this field.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, background): “Compared with attention models introduced in recent works [32, 15], DSA has two key advantages: first, it is more efficient and can capture more comprehensive contextual information; second, we impose an oracle supervision signal to ensure that the attention scores produced by DSA are reasonable.”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, methodology): “These two-stage methods [29,24,26,40,27,41,21] typically generate an intermediate image with recovered structures in the first stage (i.”
Implicit Neural Representations with Periodic Activation Functions (2020, methodology): “Other work explores neural networks with periodic activations for simple classification tasks [22–24] and recurrent neural networks [25–29].”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, methodology): “Deep-learning based inpainting Deep-learning based inpainting methods often use deep neural networks and GANs to adversarially generate pixels of missing parts [6, 15, 20, 25, 13].”
Recurrent Feature Reasoning for Image Inpainting (2020, methodology): “For Coarse-To-Fine, we use the data reported in the CSA paper [14] because the official code is unavailable.”
Diversity-Generated Image Inpainting with Style Extraction (2019, neutral): “However, similar to other existing methods [1, 20, 35, 36, 38], classic image completion methods attempt to inpaint missing regions of the original image in a deterministic manner, thus only producing a single result.”
Text-Guided Neural Image Inpainting (2020, methodology): “There have been many methods proposed for generating semantically coherent content, such as integrating context features [11, 19], enhance convolution layers [18, 41].”
Intermediate Layer Optimization for Inverse Problems using Deep Generative Models (2021, background): “Finally, there is signiﬁcant prior work on deep learning meth-ods that do not rely on pre-trained generators, see e.g. (Lucas et al., 2018; Yu et al., 2019; Liu et al., 2019; Sun & Chen, 2020; Sun et al., 2020; Yang et al., 2019; Tian et al., 2020; Tripathi et al., 2018).”
Image Inpainting: A Review (2019, background): “7809 [33] Paris StreetView Random region 32.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “Besides, more sophisticated networks or learning strategies are proposed to generate high-quality images, including global and local discrimination [22], contextual attention [35, 61, 65, 66], partial [33] and gated [67] convolution, etc.”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, background): “There also exists another trend to combine the advantages of deep generative and traditional patch-based methods for image inpainting [35, 30, 24, 15], delivering inpainting contents with both realistic textures and plausible semantics.”