Free-Form Image Inpainting With Gated Convolution

By Jiahui Yu, Zhe L. Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang. 2018

2,051 citations recorded by the source corpus.

We present a generative image inpainting system to complete images with free-form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free-form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch-based GAN loss, named SN-PatchGAN, by applying spectral-normalized discriminator on dense image patches. SN-PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user-guided extension demonstrate that our system generates higher-quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces. Code, demo and models are available at: \url{https://github.com/JiahuiYu/generative_inpainting}.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “Various shapes of training masks were explored, including random [18], free-form [57] and object-shaped masks [54, 61].”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, methodology): “Our method is compared against DeepFillv2 (Yu et al., 2019) and RFR (Li et al.”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “Reconstruction Loss: We employ the widely adopted reconstruction loss [11], [12], [14] Lc over the two output images generated by the regionwise generators.”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “Free-form inpainting method proposed in [52] is perhaps closest in spirit to our scheme.”
MaskGIT: Masked Generative Image Transformer (2022, methodology): “Traditional patch-based methods [3] work well on texture regions, while deep learning based methods [14,38,48,50,54] have been demonstrated to synthesize images requiring better semantic coherence.”
Palette: Image-to-Image Diffusion Models (2021, methodology): “It is important to note that DeepFillv2 and HiFill are not trained on ImageNet, but we report their performance on ImageNet ctest10k primarily for benchmarking purposes.”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, neutral): “We just mention DeepFillv2 (Yu et al. 2018) as a representative state-of-the-art technique.”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, methodology): “Such embedding algorithms can be used to analyze the limitations of GANs [5], do image inpainting [8, 39, 38, 36], local image editing [40, 17], global image transformations such as image morphing and expression transfer [1], and few-shot video generation [35, 34].”
Pluralistic Image Completion (2019, methodology): “This was followed by several CNN-based methods, which included combining global and local discriminators as adversarial loss [14], identifying closest features in the latent space of masked images [40], utilizing semantic labels to guide the completion network [36], introducing additional face parsi”
MaskGAN: Towards Diverse and Interactive Facial Image Manipulation (2019, background): “Because GAN can generate realistic images, it enjoys pervasive applications on tasks such as image-to-image translation [14, 47, 25, 38, 30], image inpainting [24, 44, 45, 15], and virtual try-on [41, 9, 3, 37].”
PEPSI++: Fast and Lightweight Network for Image Inpainting (2019, methodology): “(a) The ground truth (b) The input image of the network (c) Results of the non-generative method, PatchMatch (d) Results of the GatedConv [16] (e) Results of the PEPSI (f) Results of the Diet-PEPSI.”
Single-Shot Freestyle Dance Reenactment (2020, background): “(vii) An inpainting network [49, 48] extracts the background from the training images, as well as from the target image at inference time.”
Gated-SCNN: Gated Shape CNNs for Semantic Segmentation (2019, positive): “[53], on the other hand, proposed the use of convolutions with a soft-gating mechanism for Free-Form Image Inpainting and [46] proposed Gated PixelCNN for conditional image generation.”
SC-FEGAN: Face Editing Generative Adversarial Network With User’s Sketch and Color (2019, methodology): “In the subsequent version, DeepFillv2 [17], free-form masks and a single SN-PatchGAN discriminator replace the rectangular masks and global and local discriminators used in the previous version.”
Image Fine-grained Inpainting (2020, neutral): “e results in Table IV, which validates the A CORRESPONDENCE SUBMITTED TO JOURNAL 8 Input LBAM [38] PICNet [12] DMFN (Ours) Fig. 10. Inpainted images with irregular masks on Paris StreetView. Input GC [16] PICNet [12] DMFN (Ours) Fig. 11. Inpainted images with irregular masks on CelebA-HQ. effectiven”
Implicit Neural Representations with Periodic Activation Functions (2020, methodology): “These inpainting methods are based on convolutional neural networks (CNNs) [27, 38] and generative adversarial networks (GANs) [22, 44, 28, 32].”
Boundless: Generative Adversarial Networks for Image Extension (2019, methodology): “Following the code of DeepFill [48], we concatenate a channel of 1’s to the input of the generator.”
Neural Point-Based Graphics (2019, methodology): “The rendering network in our case has a popular convolutional U-Net architecture [37] with gated convolutions [49].”