SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild'

By Soumyadip Sengupta, Angjoo Kanazawa, Carlos Domingo Castillo, David W. Jacobs. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

343 citations recorded by the source corpus.

We present SfSNet, an end-to-end learning framework for producing an accurate decomposition of an unconstrained human face image into shape, reflectance and illuminance. SfSNet is designed to reflect a physical lambertian rendering model. SfSNet learns from a mixture of labeled synthetic and unlabeled real world images. This allows the network to capture low frequency variations from synthetic and high frequency details from real images through the photometric reconstruction loss. SfSNet consists of a new decomposition architecture with residual blocks that learns a complete separation of albedo and normal. This is used along with the original image to predict lighting. SfSNet produces significantly better quantitative and qualitative results than state-of-the-art methods for inverse rendering and independent normal and illumination estimation.

Subfields: Atomic and Molecular Physics, and Optics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/074619ffc19894c13974321d4b31144acc212f91

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Atomic and Molecular Physics, and Optics

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Single-Image Portrait Relighting (2019, methodology): “[26, 30, 8, 29, 22] are all deep learning-based methods.”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “[Richardson et al. 2017; Sengupta et al. 2018] refine 3DMM predictions for higher quality or more detailed results.”
Portrait shadow manipulation (2020, methodology): “General techniques for inverse rendering [Ramamoorthi and Hanrahan 2001; Sengupta et al. 2018] and intrinsic image decomposition [Barron and Malik 2015; Grosse et al.”
Single image portrait relighting (2019, methodology): “For this reason, previous works [Sengupta et al. 2018; Li et al. 2018] have relied heavily on synthetic, often Lambertian, images for use as training data.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, methodology): “As in several recent works [59, 57] we consider a Lambertian reflectance model for human faces and adopt the Spherical Harmonic model to model the effects of illumination on appearance [83, 74, 75].”
Extreme 3D Face Reconstruction: Seeing Through Occlusions (2017, methodology): “Contrary to these methods, deep networks were used in [6, 11, 24, 44, 49, 50, 51] to estimate 3D shapes with an emphasis on unconstrained photos.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, methodology): “2019] or surface normals that add detail to coarse reconstructions [Abrevaya et al. 2020; Sengupta et al. 2018].”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, methodology): “2, IRN has a structure similar to [32], which consists of a convolutional encoder, followed by nine residual blocks and a convolutional decoder for estimating albedo and normals.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “3DMM from I Humans [47] 3DMM, 2DS+KP Pred.”
Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “A number of techniques have been introduced to exploit 2D training data to overcome this limitation, and use key points [26], silhouettes [4, 1, 2, 27], and shading cues [28] for supervision.”
Differentiable surface splatting for point-based geometry processing (2019, methodology): “The differentiable rendering is also relates to shape-from-shading techniques [Langguth et al. 2016; Maier et al. 2017; Sengupta et al. 2018; Shi et al. 2017] that extract shading and albedo information for geometry processing and surface reconstruction.”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, methodology): “[43] proposed to train the SfSNet on combination of labeled synthetic data and unlabeled in-the-wild data to estimate plausible detailed shape in un-Table [15] also tried to predict the displacement map with a conditional GAN based on the 3DMM model, which enables to recover detailed shape from an…”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, background): “A number of recent deep inverse rendering methods have incorporated in-network, differentiable rendering layers that are customized for simple settings: faces [46, 52, 45], planar surfaces [15, 32], single objects [35, 33].”