Image Inpainting via Generative Multi-column Convolutional Neural Networks

By Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, Jiaya Jia. 2018

338 citations recorded by the source corpus.

In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “In [46], the use of branches in the completion network with varying receptive fields was suggested.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “…Iizuka et al., 2017; Jiang et al., 2019; Lahiri et al., 2020; Li et al., 2020; Liu et al., 2018; 2019a; 2020; Nazeri et al., 2019; Ren et al., 2019; Wang et al., 2018; Xie et al., 2019; Xiong et al., 2019; Yan et al., 2018; Yu et al., 2018; 2019; Yu et al., 2019; Zeng et al., 2019; Zhao et al.,…”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, methodology): “To perceptually enhance image quality, several studies [5]–[7] have attempted to extract features using a pretrained VGG network to reduce”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “ID-MRF loss: We adopt an Implicit Diversified Markov Random Field (ID-MRF) loss [Wang et al. 2018] to reconstruct geometric details.”
Learning to Incorporate Structure Knowledge for Image Inpainting (2020, methodology): “Unlike using an explicit module to propagate information from the surroundings to missing regions, Wang et al.(2018) introduce an implicit diversiﬁed Markov random ﬁelds (ID-MRF) loss which implicit constraints the network to propagates relevant information to the target inpainting areas.”
PEPSI++: Fast and Lightweight Network for Image Inpainting (2019, background): “IMAGE inpainting techniques that attempt to remove an unwanted object or synthesize missing parts of an image have attracted widespread interest in computer vision and graphics communities [1]–[17].”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, background): “As indicated by [14], if the first stage fails to generate correct intermediate clues, the wrong information would be”
Diversity-Generated Image Inpainting with Style Extraction (2019, methodology): “The CNN-based inpainting is extended to the large mask, and a context encoder based on the generation adversarial network (GAN) is proposed for inpainting the learned features [18].”
UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation (2020, background): “In order to overcome these problems, researchers have done a lot of work, such as [27, 23, 26, 22, 13, 16, 12, 25].”
Image Fine-grained Inpainting (2020, positive): “Input Image CE [20] Shift-Net [28] [26] PICNet [33] DMFN (Ours) There are 20 participants invited to accomplish this survey.”
Image Inpainting With Learnable Bidirectional Attention Maps (2019, background): “[28] suggested a generative multi-column CNN incorporating with confidence-driven reconstruction loss and implicit diversified MRF (ID-MRF) term.”
Boundless: Generative Adversarial Networks for Image Extension (2019, background): “Other similar approaches include [26, 43, 47], while [46] train an unconditioned GAN to generate complete images from the target distribution and perform an inference-time optimization to search for the latent code that would produce the closest match to the known pixels of the masked image.”
Learning Oracle Attention for High-Fidelity Face Completion (2020, background): “Recently, convolutional neural network (CNN) based methods have become the mainstream approach to image inpainting [30, 29, 12, 26, 36].”
Prior Guided GAN Based Semantic Inpainting (2020, methodology): “Next, we compare with some of the contemporary ‘singlepass-inference’ baselines of PIC [49], GIP [47] and MCCNN [42].”
Image Inpainting with Learnable Feature Imputation (2020, methodology): “Furthermore, several studies [17,40,41,47] propose to use Wasserstein GAN [2] with gradient penalties [10]; however, the standard gradient penalty causes training instability for image-inpainting models, as we discuss in Section 3.”
Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation (2019, methodology): “To perceptually enhance image quality, several studies [Yang et al., 2017; Song et al., 2017; Wang et al., 2018b] attempted to extract features using pretrained VGG network to reduce the perceptual loss or style loss.”
Aggregated Contextual Transformations for High-Resolution Image Inpainting (2021, background): “The other kind of works stack serialized dilated convolution layers to capture distant contexts [13], [20], [44].”
Contrastive Learning for Diverse Disentangled Foreground Generation (2022, background): “Recently, deep learning methods [8, 18, 33, 34, 46, 48, 50, 51, 53, 57, 28, 55, 59] dramatically boosted the quality, in terms of both visual quality and semantic coherence.”
Wide-Context Semantic Image Extrapolation (2019, methodology): “The IDMRF loss [29, 44] between Ŷ Ω and Y L is defined as”