Image Inpainting for Irregular Holes Using Partial Convolutions

By Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro. 2018

2,284 citations recorded by the source corpus.

Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “The subject has been studied in pre-deep learning era [1, 5, 13], and the progress accelerated in recent years through the use of deep and wide neural networks [26, 30, 25] and adversarial learning [34, 18, 56, 44, 57, 32, 54, 61].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “, 2019), a multiplicity of task-specific approaches have been proposed to specialize GAN frameworks, mostly focused on hand-engineered multi-stage architectures, specialized operations, or intermediate structures like edges or contours (Altinel et al., 2018; Ding et al., 2018; Iizuka et al., 2017; J”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “As claimed in [10] this may propagate the noise or errors for most inpainting architectures.”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, background): “[28] introduced “partial convolution” for image inpainting, where convolution weights are normalized by the mask area of the window that the convolution filter currently resides over.”
Free-Form Image Inpainting With Gated Convolution (2018, methodology): “Next, we compare our model with previous state-of-theart methods [15, 23, 49].”
Palette: Image-to-Image Diffusion Models (2021, methodology): “While there are other important prior works on image inpainting, such as [Liu et al. 2018a, 2020; Zheng et al. 2019], we were not able to compare with all of them.”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, methodology): “Figure 6: First column: original image; Second column: defective image ; Third column: inpainted image via partial convolutions [24]; Fourth column: inpainted image using our method.”
Pluralistic Image Completion (2019, methodology): “This was followed by several CNN-based methods, which included combining global and local discriminators as adversarial loss [14], identifying closest features in the latent space of masked images [40], utilizing semantic labels to guide the completion network [36], introducing additional face parsi”
COCO-GAN: Generation by Parts via Conditional Coordinating (2019, background): “The quality outperforms the partial convolution [14].”
StructureFlow: Image Inpainting via Structure-Aware Appearance Flow (2019, neutral): “We use the irregular mask dataset provided by [16].”
Dual Contrastive Loss and Attention for GANs (2021, background): “translation [35, 107, 108, 54, 33, 82, 64, 20, 63], post-processing [46, 71, 44, 45, 77, 62, 102], image manipulation [13, 14, 70, 1, 4, 80], texture synthesis [94, 53, 59], image inpainting [34, 52, 92, 93], and text-to-image generation [68, 99, 100, 74].”
Adversarial Scene Editing: Automatic Object Removal from Weak Supervision (2018, methodology): “We also incorporate the style-loss (Lsty) proposed in [24] to better match the textures in the in-painting output with that of the input image and the total variation loss (Ltv) since it helps produce smoother boundaries between the in-painted region and the original image.”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, methodology): “We use the same training and testing irregular masks as [48], including 55,116 training masks and 12,000 testing masks.”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, methodology): “Analogous to [13], we first use a learning rate of 2× 10−4 for initial training, then finetune the model with a learning rate of 5× 10−5, and freeze the Batch Normalization (BN) parameters of the generator.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “[21], GANs have shown promising results in various areas [48], such as unconditional image synthesis [21, 23, 32, 33, 34, 49, 61], image translation [8, 12, 27, 29, 46, 47, 58, 67, 73, 84, 102, 103], text-to-image translation [62, 90, 95], image processing [17, 18, 28, 36, 39, 41, 42, 44, 72, 78, 89”
Neural Point-Based Graphics (2019, methodology): “Also, highly relevant to our work are methods that successfully apply deep ConvNets for image inpainting tasks [20, 30, 49].”
Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting (2020, methodology): “We compared our method with other state-of-the-art learning-based inpainting methods including Global-local GAN [15], DeepFillV1 [36], DeepFillV2 [35], PENNet [37] and Partial Convolution [19].”
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations (2020, background): “We obtain irregular masks from PConv [20].”
Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation (2019, methodology): “We also adopt skip links, which as [Liu et al., 2018] claimed, may propagate the noises for most inpainting architectures.”