Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

By Yanhong Zeng, Jianlong Fu, Hongyang Chao, Baining Guo. 2019

471 citations recorded by the source corpus.

High-quality image inpainting requires filling missing regions in a damaged image with plausible content. Existing works either fill the regions by copying high-resolution patches or generating semantically-coherent patches from region context, while neglecting the fact that both visual and semantic plausibility are highly-demanded. In this paper, we propose a Pyramid-context Encoder Network (denoted as PEN-Net) for image inpainting by deep generative models. The proposed PEN-Net is built upon a U-Net structure with three tailored components, ie., a pyramid-context encoder, a multi-scale decoder, and an adversarial training loss. First, we adopt a U-Net as backbone which can encode the context of an image from high-resolution pixels into high-level semantic features, and decode the features reversely. Second, we propose a pyramid-context encoder, which progressively learns region affinity by attention from a high-level semantic feature map, and transfers the learned attention to its adjacent high-resolution feature map. As the missing content can be filled by attention transfer from deep to shallow in a pyramid fashion, both visual and semantic coherence for image inpainting can be ensured. Third, we further propose a multi-scale decoder with deeply-supervised pyramid losses and an adversarial loss. Such a design not only results in fast convergence in training, but more realistic results in testing. Extensive experiments on a broad range of datasets shows the superior performance of the proposed network.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Simple convolutional discriminators [34, 52] or PatchGAN discriminators [18, 59, 36, 28] were used to implement adversarial losses.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…et al., 2019; Wang et al., 2018; Xie et al., 2019; Xiong et al., 2019; Yan et al., 2018; Yang et al., 2017; 2019a; Yu et al., 2018; Yu et al., 2019; Zeng et al., 2019; Lahiri et al., 2020; Zhao et al., 2020a; Li et al., 2020; Zhou et al., 2020), among which (Liu et al., 2018; Yu et al., 2019)…”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “Performances of PEN-Net [9] and StructureFlow [13] are also evaluated using their official released codes and models.”
Image Inpainting: A Review (2019, background): “[32] built a pyramidal-context architecture called PEN-NET for high-quality image inpainting.”
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling (2020, background): “[41] propose to use region affinity from a high-level feature map to guide the completion of missing regions”
Diversity-Generated Image Inpainting with Style Extraction (2019, background): “Recently, some works [34, 36] used spatial attention to recover the lost area using the surrounding image features as reference.”
Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting (2020, methodology): “We compared our method with other state-of-the-art learning-based inpainting methods including Global-local GAN [15], DeepFillV1 [36], DeepFillV2 [35], PENNet [37] and Partial Convolution [19].”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, background): “Follow-up works have produced impressive results in recent years [15,20,30,34,50].”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “Recently, deep learning methods [8,18,33,35,48,50,52,53,55,59,28,57,61] dramatically boosted the quality, in terms of both visual quality and semantic coherence.”
Learning to Incorporate Structure Knowledge for Image Inpainting (2020, methodology): “Besides, under the scenario with irregular masks, although models such as CA, GL, and PEN-Net can deal with irregular holes (like ﬁlling irregular holes with multiple regular patches), they usually show inferior performance since not particularly trained on irregular masks.”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, background): “In order to overcome these problems, researchers have done a lot of work, such as [27, 23, 26, 22, 13, 16, 12, 25].”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, methodology): “inherit the discriminator from PatchGAN [24] due to its great success in image translation [7, 17, 13, 21, 25, 45].”
Learning Oracle Attention for High-Fidelity Face Completion (2020, methodology): “In each row from left to right: (a) the result taken from the paper of PEN-Net [34].”
Learning Joint Spatial-Temporal Transformations for Video Inpainting (2020, methodology): “To eﬀectively model long-range correspondences, recent models have adopted attention modules and show promising results in image and video synthesis [21,34,36].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “…[48], such as unconditional image synthesis [21, 23, 32, 33, 34, 49, 61], image translation [8, 12, 27, 29, 46, 47, 58, 67, 73, 84, 102, 103], text-to-image translation [62, 90, 95], image processing [17,18,28,36,39,41,42,44,72,78,89,94], and video synthesis [2,10,35,45,50,60,63,69,82,83,101].”
Image Fine-grained Inpainting (2020, unknown): “This problem has been mitigated by GAN [6] or its improved version WGANGP [7] that is frequently utilized in image inpainting methods [20, 9, 29, 17, 30, 24, 28, 26, 33, 31].”
Learning Texture Transformer Network for Image Super-Resolution (2020, methodology): “To learn a better representation across different scales, inspired by [25, 37], we propose a cross-scale feature integration module (CSFI) to exchange information among the features of different scales.”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, methodology): “We follow [42] to involve over 35 volunteers for evaluating the results on CelebA, Place2 and Paris StreetView datasets.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “Afterwards, numerous variants [34, 57, 64, 69] of the U-Net structure [45] have been developed for image completion.”
Image Inpainting with Learnable Feature Imputation (2020, background): “There exist several studies proposing methods to ensure global and local consistency; using several discriminators to focus on different scales [16,25], specific modules to connect spatially distant features [39,44,45,47], patch-based discriminators [48,49], multi-column generators [40], or progress”