High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling

By Yu Zeng, Zhe L. Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, Huchuan Lu. 2020

187 citations recorded by the source corpus.

Existing image inpainting methods often produce artifacts when dealing with large holes in real applications. To address this challenge, we propose an iterative inpainting method with a feedback mechanism. Specifically, we introduce a deep generative model which not only outputs an inpainting result but also a corresponding confidence map. Using this map as feedback, it progressively fills the hole by trusting only high-confidence pixels inside the hole at each iteration and focuses on the remaining pixels in the next iteration. As it reuses partial predictions from the previous iterations as known pixels, this process gradually improves the result. In addition, we propose a guided upsampling network to enable generation of high-resolution inpainting results. We achieve this by extending the Contextual Attention module to borrow high-resolution feature patches in the input image. Furthermore, to mimic real object removal scenarios, we collect a large object mask dataset and synthesize more realistic training data that better simulates user inputs. Experiments show that our method significantly outperforms existing methods in both quantitative and qualitative evaluations. More results and Web APP are available at this https URL.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Various shapes of training masks were explored, including random [18], free-form [57] and object-shaped masks [54, 61].”
Using latent space regression to analyze and leverage compositionality in GANs (2021, background): “However, in contrast to inpainting, in which regions have to be filled in otherwise globally consistent images (Pathak et al., 2016; Iizuka et al., 2017; Yu et al., 2018; Zeng et al., 2020), the composition problem involves correcting inconsistencies as well as filling in missing pixels.”
PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization (2021, background): “Recent advances on image manipulation techniques [1]–[4] enable easy editing of raw images, such as removing unwanted objects [5]–[8], face swapping [2], attribute changing [9], etc.”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, unknown): “The deep generative methods [35, 36, 8, 34, 38, 40, 37, 26, 12] are currently dominating, which effectively extract meaningful semantics from damaged images and recover reasonable contents with high visual fidelity, owing to their powerful feature learning ability.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “To allow for high-resolution image inpainting, a few attempts have been made to develop progressively generation systems, such as [18, 32, 41, 71, 72].”