Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations

By Hongyu Liu, Bin Jiang, Yibing Song, Chao Yang. 2020

315 citations recorded by the source corpus.

Deep encoder-decoder based CNNs have advanced image inpainting methods for hole filling. While existing methods recover structures and textures step-by-step in the hole regions, they typically use two encoder-decoders for separate recovery. The CNN features of each encoder are learned to capture either missing structures or textures without considering them as a whole. The insufficient utilization of these encoder features limit the performance of recovering both structures and textures. In this paper, we propose a mutual encoder-decoder CNN for joint recovery of both. We use CNN features from the deep and shallow layers of the encoder to represent structures and textures of an input image, respectively. The deep layer features are sent to a structure branch and the shallow layer features are sent to a texture branch. In each branch, we fill holes in multiple scales of the CNN features. The filled CNN features from both branches are concatenated and then equalized. During feature equalization, we reweigh channel attentions first and propose a bilateral propagation activation function to enable spatial equalization. To this end, the filled CNN features of structure and texture mutually benefit each other to represent image content at all feature levels. We use the equalized feature to supplement decoder features for output image generation through skip connections. Experiments on the benchmark datasets show the proposed method is effective to recover structures and textures and performs favorably against state-of-the-art approaches.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Other popular choices are Wasserstein adversarial losses with gradient-penalized discriminators [56, 54] and spectral-normalized discriminators [32, 57, 27, 61].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, neutral): “…like edges or contours (Altinel et al., 2018; Ding et al., 2018; Iizuka et al., 2017; Jiang et al., 2019; Lahiri et al., 2020; Li et al., 2020; Liu et al., 2018; 2019a; 2020; Nazeri et al., 2019; Ren et al., 2019; Wang et al., 2018; Xie et al., 2019; Xiong et al., 2019; Yan et al., 2018; Yu…”
Palette: Image-to-Image Diffusion Models (2021, background): “features [Iizuka et al. 2017; Kim et al. 2021; Liu et al. 2020; Nazeri et al. 2019; Yi et al. 2020; Yu et al. 2018b, 2019], and they lack diversity in their outputs [Zhao et al.”
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models (2023, background): “Inpainting (Liu et al., 2020; Nazeri et al., 2019) is the task of ﬁlling masked regions of an audio with new content since parts of the audio are corrupted or undesired.”
High-Fidelity Pluralistic Image Completion with Transformers (2021, methodology): “To demonstrate the superiority, we compare our method with state-of-the-art deterministic [25, 20, 37] and pluralistic [41] image completion approaches on multiple datasets.”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, methodology): “Follow-up works have produced impressive results in recent years [15,20,30,34,50].”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, methodology): “MED [14] attempts to correlate structure and texture generation, while the shared generator is inadequate for generating sharp edges and clear textures (e.g., facades in Figure 5).”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “Afterwards, numerous variants [34, 57, 64, 69] of the U-Net structure [45] have been developed for image completion.”