Generative Face Completion

By Yijun Li, Sifei Liu, Jimei Yang, Ming-Hsuan Yang. 2017

631 citations recorded by the source corpus.

In this paper, we propose an effective face completion algorithm using a deep generative model. Different from well-studied background completion, the face completion task is more challenging as it often requires to generate semantically new pixels for the missing key components (e.g., eyes and mouths) that contain large appearance variations. Unlike existing nonparametric algorithms that search for patches to synthesize, our algorithm directly generates contents for missing regions based on a neural network. The model is trained with a combination of a reconstruction loss, two adversarial losses and a semantic parsing loss, which ensures pixel faithfulness and local-global contents consistency. With extensive experimental results, we demonstrate qualitatively and quantitatively that our model is able to deal with a large area of missing pixels in arbitrary shapes and generate realistic face completion results.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Free-Form Image Inpainting With Gated Convolution (2018, background): “Thus they can synthesize more visually plausible contents especially for images like faces [22, 47], objects [29] and natural scenes [15, 49].”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
Generative Image Inpainting with Contextual Attention (2018, methodology): “Global and local Wasserstein GANs Different from previous generative inpainting networks [15, 25, 30] which rely on DCGAN [31] for adversarial supervision, we propose to use a modified version of WGAN-GP [1, 12].”
Image Inpainting for Irregular Holes Using Partial Convolutions (2018, background): “Li et al. [17] and Iizuka et al. [10] extended Content Encoders by deﬁning both global and local discriminators; then Iizuka et al. [10] apply Poisson blending as a post-process.”
Pluralistic Image Completion (2019, methodology): “This was followed by several CNN-based methods, which included combining global and local discriminators as adversarial loss [14], identifying closest features in the latent space of masked images [40], utilizing semantic labels to guide the completion network [36], introducing additional face parsi”
Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart (2018, methodology): “Face completion results comparing with [17]. the number of residual blocks has a negative eﬀect on inpainting, possibly due to a variety of factors such as gradient vanishing, optimization diﬃculty, etc.”
Detecting GAN generated Fake Images using Co-occurrence Matrices (2019, background): “These works focus on generating images of high perceptual quality [10, 11, 12, 13, 14, 15, 4], image-to-image translations [13, 16, 1], domain transfer [17, 18], super-resolution [19], image synthesis and completion [20, 21, 22], and generation of facial attributes and expressions [23, 24, 18, 2].”
Prior Guided GAN Based Semantic Inpainting (2020, methodology): “The first approach has drawn majority of recent attraction [35, 21, 47, 29, 28] due to its fast inference speed and appreciable performance at high resolution.”
Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions (2020, background): “…images [29, 6], image-to-image [45, 26, 61, 9, 42, 30] and text-to-image translations [48, 11, 58, 59], style transfer [27, 60, 61, 25], image inpainting [45, 54, 33, 26, 56], transfer learning [5, 10, 15] or even for training semantic segmentation tasks [35, 53], just to name a few.”
Crowd Counting via Adversarial Cross-Scale Consistency Pursuit (2018, background): “To alleviate this issue and motivated by recent success of generative adversarial networks (GANs) [7, 23, 17, 14, 19], we propose an adversarial loss.”
Partial Convolution based Padding (2018, methodology): “Recent deep learning based methods [18, 14, 9, 30] initialize the holes (regions with missing data) with some constant placeholder values and treat the newly initialized hole region and original non-hole region equally.”
FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors (2017, methodology): “Although our prior estimation network can predict the parsing maps from the LR inputs, for fair comparison, we adopt a recent model GFC [23] to generate the facial parsing maps for the recovered images of all methods, including the bicubic inputs, our coarse SR net, SRResNet, our fine SR net, and ta”
Detecting Overfitting of Deep Generative Networks via Latent Recovery (2019, methodology): “2b) As [17], we corrupt training images by replacing patches of various sizes with fixed Gaussian noise with variance σ(2) d; • Additive Noise (Fig.”
Manipulating Attributes of Natural Scenes via Hallucination (2018, background): “Lastly, building upon conditional GAN model, some image completion works have been proposed to predict missing regions providing global and local context information with multiple discriminator networks [Iizuka et al. 2017; Li et al. 2017].”
Eye In-painting with Exemplar Generative Adversarial Networks (2017, background): “Specifically within this body of work, the applications of GANs [13] to faces are numerous [15, 24, 40].”
Shift-Net: Image Inpainting via Deep Feature Rearrangement (2018, background): “In order to obtain globally consistent result with locally realistic details, global and local discriminators have been proposed in image inpainting [15] and face completion [28].”
SRFeat: Single Image Super-Resolution with Feature Discrimination (2018, methodology): “Thanks to its effectiveness in image generation, GAN has been widely applied to various tasks such as image synthesis, style transfer, image inpainting, and object detection [37, 55, 20, 25, 23, 19, 30, 28].”
High-Fidelity Pluralistic Image Completion with Transformers (2021, background): “In recent years, CNN-based solutions [22, 14, 18, 27, 36, 21] started to dominate this field.”
LaFIn: Generative Landmark Guided Face Inpainting (2019, background): “Compared with the edge [20] and parsing [15] information, for one thing, the landmarks are much neater and more robust, please see Fig.”