Foreground-Aware Image Inpainting

By Wei Xiong, Jiahui Yu, Zhe L. Lin, Jimei Yang, Xin Lu, Connelly Barnes, Jiebo Luo. 2019

375 citations recorded by the source corpus.

Existing image inpainting methods typically fill holes by borrowing information from surrounding pixels. They often produce unsatisfactory results when the holes overlap with or touch foreground objects due to lack of information about the actual extent of foreground and background regions within the holes. These scenarios, however, are very important in practice, especially for applications such as distracting object removal. To address the problem, we propose a foreground-aware image inpainting system that explicitly disentangles structure inference and content completion. Specifically, our model learns to predict the foreground contour first, and then inpaints the missing region using the predicted contour as guidance. We show that by such disentanglement, the contour completion model predicts reasonable contours of objects, and further substantially improves the performance of image inpainting. Experiments show that our method significantly outperforms existing methods and achieves superior inpainting results on challenging cases with complex compositions.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Several works suggest two-stage methods that use completion of other structure types as an intermediate step: salient edges in [32], semantic segmentation maps in [44], foreground object contours in [48], gradient maps in [52], and edge-preserved smooth images in [36].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “, 2019), a multiplicity of task-specific approaches have been proposed to specialize GAN frameworks, mostly focused on hand-engineered multi-stage architectures, specialized operations, or intermediate structures like edges or contours (Altinel et al., 2018; Ding et al., 2018; Iizuka et al., 2017; J”
Free-Form Image Inpainting With Gated Convolution (2018, background): “The latter approach [15, 49, 45, 46, 38, 37, 48, 26, 52, 33, 35, 19] can exploit semantics learned from large scale datasets to synthesize contents in nonstationary images in an end-to-end fashion.”
LaFIn: Generative Landmark Guided Face Inpainting (2019, methodology): “For the sake of maintaining the structure of corrupted images, a number of methods, such as [20, 31], try to ﬁrst predict the edge information for corrupted images, then apply it as a condition to guide the inpainting.”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, background): “[34] present foreground-aware inpainting, which involves three stages, i.”
RePaint: Inpainting using Denoising Diffusion Probabilistic Models (2022, background): “…[48] to guide the convolution kernel according to the inpainted mask, Contextual Attention [46] to leverage on global information, Edges maps [9,27,43,44] or Semantic Segmentation maps [14, 31] to further guide the generation, and Fourier Convolutions [40] to include both global and lo-cal…”
Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting (2020, background): “Compared to traditional methods, these techniques have strong ability to learn adaptive and high-level features of disparate semantics and thus are more adept in hallucinating visually plausible contents especially when inpainting structured images like faces [15, 20, 24, 35, 36, 37], objects [18, 2”
Deblurring by Realistic Blurring (2020, background): “have achieved a great success in the areas of object recognition [9, 44, 43, 19, 18] and image reconstruction including video deblurring [45], video dehazing [25], and other GAN-based generation tasks [29, 24, 39, 37].”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “Our inpainting model builds upon the recent two-stage approaches [41, 62, 47] but with two key differences.”
Flow-edge Guided Video Completion (2020, neutral): “, edges) to guide the content [25,33,41].”
Image Fine-grained Inpainting (2020, positive): “[27] inferred the contours of the objects in the image, then used the completed contours as a guidance to complete the image.”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “References [11], [39] disentangled the image inpainting problem into the contour completion module and image completion module.”
Learning to Incorporate Structure Knowledge for Image Inpainting (2020, background): “Besides, Nazeri et al.(2019) and Xiong et al.(2019) have demonstrated that structure priors are beneﬁting to image completion; the other way round, it is more likely to ﬁgure out the complete structures from a relatively intact image compared with a corrupted one.”
Progressive Image Inpainting with Full-Resolution Residual Network (2019, background): “Further, GAN-based works [12, 21, 29, 31–33 , 35] can generate visually plausible”
Recurrent Feature Reasoning for Image Inpainting (2020, neutral): “[26] and Nazeri et al.”
Image Inpainting: A Review (2019, methodology): “Foreground-aware dataset [60] is different from the other’s dataset.”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, methodology): “We therefore adopt the gray-scale image as an additional condition and feed the paired data as the input in the structure branch, as several previous studies do [28, 18].”
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling (2020, methodology): “For fair comparisons with previous methods, we also evaluate on the standard Places2 validation set resized to 256 × 256 with 128 × 128 center square holes as in most previous methods e.g. [18,34,36], and irregular holes as in [35,32].”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, background): “the discriminator from PatchGAN [24] due to its great success in image translation [7], [13], [17], [21], [25], [51].”
Region Normalization for Image Inpainting (2019, methodology): “2014), GAN-based methods (Yeh et al. 2017; Iizuka, Simo-Serra, and Ishikawa 2017; Yu et al. 2018; Xiong et al. 2019; Nazeri et al. 2019) are widely used in image inpainting.”