Single image portrait relighting

By Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E. Debevec, Ravi Ramamoorthi. 2019

306 citations recorded by the source corpus.

Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting: a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset's validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 × 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

A Free Viewpoint Portrait Generator with Dynamic Styling (2020, unknown): “[12, 13, 9, 35, 20, 32].”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, background): “Those conditioned on control vectors are typically limited to a small range of applications such as gaze redirection [26] or relighting [27], [29].”
HyperNeRF (2021, methodology): “This idea has been extended to incorporate tools and principles from the graphics literature, by incorporating reflectance or illumination models into the “translation” process [Meka et al. 2019; Sun et al. 2019], or by using proxy geometries [Aliev et al.”
Portrait shadow manipulation (2020, methodology): “PR-net of Sun et al. (2019) and an ablation of our model with symmetry.”
Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning (2020, background): “GANs have seen heavy use in face image manipulation [34, 19, 49, 44, 36, 45, 53].”
PIE (2020, methodology): “…et al. 2019; Wang et al. 2019a; Zakharov et al. 2019], only enable editing of a single semantic parameter dimension, e.g., scene illumination [Meka et al. 2019; Sun et al. 2019; Zhou et al. 2019], or do not photo-realistically synthesize some important features such as hair [Nagano et al. 2018].”
TediGAN: Text-Guided Diverse Image Generation and Manipulation (2020, unknown): “Those conditioned on control vectors are typically limited to a small range of applications such as gaze redi-rection [13] or relighting [44, 59].”