High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis

By Chao Yang, Xin Lu, Zhe L. Lin, Eli Shechtman, Oliver Wang, Hao Li. 2016

832 citations recorded by the source corpus.

Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal. While these learning-based methods are significantly more effective in capturing high-level features than prior techniques, they can only handle very low-resolution inputs due to memory limitations and difficulty in training. Even for slightly larger images, the inpainted regions would appear blurry and unpleasant boundaries become visible. We propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints, which not only preserves contextual structures but also produces high-frequency details by matching and adapting patches with the most similar mid-layer feature correlations of a deep classification network. We evaluate our method on the ImageNet and Paris Streetview datasets and achieved state-of-the-art inpainting accuracy. We show our approach produces sharper and more coherent results than prior methods, especially for high-resolution images.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, methodology): “Inpainting frameworks often incorporate style [26, 30, 30, 47, 32, 25] and feature matching [51, 44, 32, 16] losses.”
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks (2021, background): “Numerous follow-up works focus on the semantic context and texture, edges and contours, or hand-engineered architectures (Altinel et al., 2018; Ding et al., 2018; Iizuka et al., 2017; Jiang et al., 2019; Jo & Park, 2019; Lahiri et al., 2017; Liu et al., 2019a; Nazeri et al., 2019; Ren et al., 2019; ”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “To perceptually enhance image quality, several studies [6]–[8] attempted to extract features using a pre-trained VGG network to reduce the perceptual loss [9] or style loss [10].”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, methodology): “[49] uses a pre-trained VGG network [42] to improve the output of the context-encoder, by minimizing the feature difference of image background.”
Toward Multimodal Image-to-Image Translation (2017, methodology): “However, the quality has been attained at the expense of multimodality, as the generator learns to largely ignore the random noise vector when conditioned on a relevant context [20, 34, 40, 45, 47, 55].”
Generative Image Inpainting with Contextual Attention (2018, background): “Several works [35, 38] follow ideas from image stylization [5, 24] to formulate the inpainting as an optimization problem.”
Image Inpainting for Irregular Holes Using Partial Convolutions (2018, methodology): “[34] takes the result from Content Encoders as input and then propagates the texture information from non-hole regions to fill the hole regions as postprocessing.”
Deformable Shape Completion with Graph Convolutional Autoencoders (2017, background): “The 3D shape completion task is closely related to the analogous structured prediction task of image inpainting [41, 62].”
Contextual-Based Image Inpainting: Infer, Match, and Translate (2017, methodology): “Quantitative Comparison Table 1 shows numerical comparison result between our approach, CE [27], GLI [14] and NPS [34].”
Pluralistic Image Completion (2019, background): “[39] proposed multi-scale neural patch synthesis, which generates highfrequency details by copying patches from mid-layer features.”
COCO-GAN: Generation by Parts via Conditional Coordinating (2019, background): “In addition, the application is also similar to image restoration [17, 33, 34] or image out-painting [27].”
GP-GAN: Towards Realistic High-Resolution Image Blending (2017, neutral): “Thus our generated images are more natural.”
Image Inpainting via Generative Multi-column Convolutional Neural Networks (2018, methodology): “In fact, adversarial loss has already been used in recent methods [18, 25, 24, 9, 26] to learn multi-modality.”
Swapping Autoencoder for Deep Image Manipulation (2020, methodology): “Many applications have been successfully built with this framework, including image inpainting [65, 33, 83, 58], photo colorization [90, 53, 92, 23], texture and geometry synthesis [93, 20, 81], sketch2photo [71], semantic image synthesis and editing [79, 68, 10, 64].”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, background): “Driven by the progress of convolutional neural networks, CNN-based methods have received considerable attention due to their ability to predict semantically meaningful contents that are not available in the known regions [44, 51, 21, 65, 66].”
Coherent Semantic Attention for Image Inpainting (2019, background): “[36] take the result from context encoders [26] as input and gradually increase the texture details to get high-resolution prediction.”
Image Inpainting via Conditional Texture and Structure Dual Generation (2021, methodology): “Different from the traditional methods [2, 5] that gradually fill in missing areas by searching for the most similar patches from known regions, the deep generative ones [19, 7, 31, 33] capture more high-level semantics and do a better job for images with”
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches (2020, methodology): “Recent models such as Context Encoder [21] and DeepFill [37, 38] build upon the powerful deep neural networks to leverage the extra data for semantic completion, which supports fast intelligent image editing for high-resolution images [33] and free-form masks [38].”
Noisier2Noise: Learning to Denoise From Unpaired Noisy Data (2019, neutral): “Convolutional neural networks have proven to be powerful tools for many image processing tasks, including denoising[13], inpainting[18], and domain translation[5].”