Semantic Image Inpainting with Deep Generative Models

By Raymond A. Yeh, Teck-Yian Lim, Alexander Gerhard Schwing, Mark Hasegawa-Johnson, Minh N. Do. 2016

1,239 citations recorded by the source corpus.

Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Resolution-robust Large Mask Inpainting with Fourier Convolutions (2021, background): “Some approaches apply spatially discounted weighting strategies for a pixel-wise loss [34, 53, 56].”
Regionwise Generative Adversarial Image Inpainting for Large Missing Areas (2019, background): “Subsequently, semantic inpainting [46] was proposed to treat the task as a constrained image generation problem, and attempted to recover the encoding of the corrupted image to the “closest” intact one.”
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning (2019, background): “[50] trains a GAN for image inpainting with uncorrupted data.”
Free-Form Image Inpainting With Gated Convolution (2018, background): “Thus they can synthesize more visually plausible contents especially for images like faces [22, 47], objects [29] and natural scenes [15, 49].”
Style transfer-based image synthesis as an efficient regularization technique in deep learning (2019, methodology): “ image-to-image translation (e.g. convert sketches to images) [24], image blending (mixing selected parts of two images to get a new one) [25], image inpainting (restoring missing pieces of an image) [26] and unconditional [27] and conditional synthesis [28]. Recently, GANs started gaining attention”
GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks (2018, methodology): “…formulation has been built on to address problems such as training stability [16], low resolution [10], and the absence of a true image quality based loss function [2], and applied to tasks such as super resolution [13], reconstructing images from a minimal data [22] and anomaly detection [19].”
Lifelong GAN: Continual Learning for Conditional Image Generation (2019, background): “Recent image-conditioned models have shown promising results for numerous image-to-image translation tasks such as maps → satellite images, sketches → photos, labels → images [12, 41, 40], future frame prediction [32], superresolution [17], and inpainting [36].”
The Perception-Distortion Tradeoff (2017, background): “Specifically, motivated by [20, 31, 46, 34, 32, 14], restoration problems can be approached by modifying the loss of the generator of a GAN to be”
Lifelong Generative Modeling (2017, background): “It has wide ranging applications from image denoising [4, 5], inpainting [6, 7], 5”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “Several approaches such as: Low rank matrix factorization [41], Generative Adversarial Network [42] have successfully addressed the problem.”
Seeing What a GAN Cannot Generate (2019, background): “Later work found that DCGAN left-inverses can be computed to high precision [25, 46], and that inversions of a GAN for glyphs can reveal specific strokes that the generator is unable to generate [9].”
Contextual-Based Image Inpainting: Infer, Match, and Translate (2017, background): “Using deep neural network for image inpainting has also been studied in [35, 27, 34, 33, 14].”