Learning to predict indoor illumination from a single image

By Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, Jean-François Lalonde. ACM Transactions on Graphics, 2017

398 citations recorded by the source corpus.

We propose an automatic method to infer high dynamic range illumination from a single, limited field-of-view, low dynamic range photograph of an indoor scene. In contrast to previous work that relies on specialized image capture, user input, and/or simple scene models, we train an end-to-end deep neural network that directly regresses a limited field-of-view photo to HDR illumination, without strong assumptions on scene geometry, material properties, or lighting. We show that this can be accomplished in a three step process: 1) we train a robust lighting classifier to automatically annotate the location of light sources in a large dataset of LDR environment maps, 2) we use these annotations to train a deep neural network that predicts the location of lights in a scene from a single limited field-of-view photo, and 3) we fine-tune this network using a small dataset of HDR environment maps to predict light intensities. This allows us to automatically recover high-quality HDR illumination estimates that significantly outperform previous state-of-the-art methods. Consequently, using our illumination estimates for applications like 3D object insertion, produces photo-realistic results that we validate via a perceptual user study.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering

View Paper: https://www.semanticscholar.org/paper/3f319959bdc1ea61c23153bda3c1f974ae8e10e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

NeRD: Neural Reflectance Decomposition from Image Collections (2020, background): “Capturing the geometry and material properties of an object is essential for several computer vision and graphics applications such as view synthesis [10, 54], relighting [5, 10, 21, 22, 30, 55], object insertion [7, 20, 30] etc.”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Inverse rendering tasks are directly related, including single-view [17] and multi-view [57] depth prediction, intrinsic decomposition [34, 11], material classiﬁcation [10] and lighting estimation [20, 21, 33].”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, methodology): “Techniques have also been proposed for single image geometric reconstruction [16] and lighting estimation [21, 17].”
Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, methodology): “The scenes are lit by a randomly selected environment map [8] and rendered using ray tracing.”
OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Barron13 [4] Gardner17 [17] Garon19 [18] Li20 [30] Ours Barron13 [4] - 23.37% 13.25% 13.60% 11.81% Gardner17 [17] 76.63% - 36.25% 39.54% 33.84% Garon19 [18] 86.75% 63.75% - 42.28% 43.47% Li20 [30] 86.40% 60.46% 57.72% - 45.23% Ours 88.19% 66.16% 56.53% 54.77% - Table 3: User study on object…”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, methodology): “In addition, we also compare with recent DL-based methods that estimate only one of the scene attributes, such as albedo [19, 20, 28, 48], lighting [8], and normals [46].”
Learning to reconstruct shape and spatially-varying reflectance from a single image (2018, methodology): “…results for several inverse rendering subproblems including estimating scene geometry [Bansal et al. 2016; Eigen and Fergus 2015], material classes [Bell et al. 2015], illumination [Gardner et al. 2017; Georgoulis et al. 2017; Hold-Geoffroy et al. 2017], and reflectance maps [Rematas et al. 2016].”
Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “To produce a large variety of realistic illumination conditions, we convert images of the Laval Indoor HDR dataset [13] into diffuse SH coefficients, similar to [24].”
Single image portrait relighting (2019, methodology): “Similarly, monocular illumination estimation techniques [Lalonde et al. 2009; Gardner et al. 2017; Hold-Geoffroy et al. 2017] can recover the environment map that illuminated a single image, but this only enables the lighting of 3D geometry that is added to the scene and does not solve the problem…”