Neural Face Editing with Intrinsic Image Disentangling

By Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, Dimitris Samaras. 2017

295 citations recorded by the source corpus.

Traditional face editing methods often require a number of sophisticated and task specific algorithms to be applied one after the other — a process that is tedious, fragile, and computationally intensive. In this paper, we propose an end-to-end generative adversarial network that infers a face-specific disentangled representation of intrinsic face properties, including shape (i.e. normals), albedo, and lighting, and an alpha matte. We show that this network can be trained on in-the-wild images by incorporating an in-network physically-based image formation module and appropriate loss functions. Our disentangling latent representation allows for semantically relevant edits, where one aspect of facial appearance can be manipulated while keeping orthogonal properties fixed, and we demonstrate its use for a number of facial editing applications.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation (2017, methodology): “The idea of conditional image generation has also been successfully applied to domain transfer [8, 27], superresolution imaging[13], and photo editing [2, 26].”
Editing in Style: Uncovering the Local Semantics of GANs (2020, methodology): “Examples of edits accomplished by these techniques include global changes in color, lighting, pose, facial expression, gender, age, hair appearance, eyewear and headwear [2, 19, 25, 30, 31].”
The Sound of Pixels (2018, background): “Our work builds off efforts to learn perceptual models that are “self-supervised” by leveraging natural contextual signals in both images [33,34,35,36] and video [37,38,39,40,41].”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, background): “Previous works on faces [32, 36] and objects [23] have shown success in using a self-supervised reconstruction loss for learning from unlabeled real images.”
CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering (2018, methodology): “[24] also propose a CNN-based method specifically for the domain of facial images, where ground truth geometry can be obtained through model fitting.”
Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz (2017, methodology): “Learning-based Approaches: In addition to optimization-based reconstruction approaches, there are many learning-based methods [72, 41, 27, 20, 56].”
Challenges in Disentangling Independent Factors of Variation (2017, methodology): “GAN is used for disentangling intrinsic image factors (albedo and normal map) in Shu et al. (2017) without using ground truth labelling.”
Visual Object Networks: Image Generation with Disentangled 3D Representations (2018, background): “…inverse graphics” [Yuille and Kersten, 2006, Bever and Poeppel, 2010], researchers have made much progress in recent years on learning to invert graphics engines, many with deep neural networks [Kulkarni et al., 2015b, Yang et al., 2015, Kulkarni et al., 2015a, Tung et al., 2017, Shu et al., 2017].”
Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance (2018, methodology): “Inspired by the success of generative adversarial networks (GANs) [38], we follow previous work [2] where an adversarial loss is adopted to generate visually realistic images: we train the Intrinsic-DAE with an extra adversarial loss term E Adversarial applied on the ﬁnal output.”
A survey on face data augmentation for the training of deep neural networks (2019, background): “For this purpose, [147] introduced a special generative adversarial network”
Mask-Guided Portrait Editing With Conditional GANs (2019, background): “, eyes [13, 41]) or editing attributes of faces [35, 42].”
Learning to reconstruct shape and spatially-varying reflectance from a single image (2018, methodology): “Differentiable rendering layers have been used to aid in the task of learning inverse rendering for problems like face reconstruction [Sengupta et al. 2018; Shu et al. 2017; Tewari et al. 2018] and material capture [Deschaintre et al. 2018; Li et al. 2018; Liu et al. 2017].”
3D-Aware Scene Manipulation via Inverse Graphics (2018, background): “Another line of approaches are built on intrinsic image decomposition [Barrow and Tenenbaum, 1978] and have shown promising results on faces [Shu et al., 2017] and objects [Janner et al.”
Deep Single-Image Portrait Relighting (2019, methodology): “In order to get a full normal image, we solve a Poisson equation to fill in the missing normals for ear, neck, mouth and background region as suggested by [26].”