Neural Inverse Rendering of an Indoor Scene From a Single Image

By Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, Jan Kautz. 2019

178 citations recorded by the source corpus.

Inverse rendering aims to estimate physical attributes of a scene, e.g., reflectance, geometry, and lighting, from image(s). Inverse rendering has been studied primarily for single objects or with methods that solve for only one of the scene attributes. We propose the first learning based approach that jointly estimates albedo, normals, and lighting of an indoor scene from a single image. Our key contribution is the Residual Appearance Renderer (RAR), which can be trained to synthesize complex appearance effects (e.g., inter-reflection, cast shadows, near-field illumination, and realistic shading), which would be neglected otherwise. This enables us to perform self-supervised learning on real data using a reconstruction loss, based on re-synthesizing the input image from the estimated components. We finetune with real data after pretraining with synthetic data. To this end, we use physically-based rendering to create a large-scale synthetic dataset, named SUNCG-PBR, which is a significant improvement over prior datasets. Experimental results show that our approach outperforms state-of-the-art methods that estimate one or more scene attributes.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Environmental Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Environmental Engineering

In 3D Vision & Pose

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “…the color image. alistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [6, 7, 9, 10, 14, 15, 19, 22, 24, 31, 33, 36, 37, 39, 43, 44, 45, 46, 49, 52, 61, 62, 63, 65, 70, 72, 75, 76, 79, 83, 84].”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, methodology): “This includes real world scans [14, 10], synthetic shape [11] and scene [49, 31, 44] datasets.”
OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, methodology): “On the contrary, both [38] and our network achieve less noisy normal predictions.”
NeRD: Neural Reflectance Decomposition from Image Collections (2020, background): “This casual setup can be extended to estimating the BRDF and shape of objects [7, 8, 10, 39, 60] or scenes [47].”
NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis (2020, background): “The resulting Neural Reﬂectance and Visibility Field (NeRV) is visualized in Figure 3.”
GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “…monocular color images, is a fundamental but challenging problem in 3D computer vision, which plays essential roles in various downstream applications such as autonomous driving [14, 15], 3D surface reconstruction [35, 65, 75], novel view synthesis [33, 43], inverse rendering [57,74], and so on.”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “Normal prediction Figure 23 shows qualitative comparisons with [45] and [61] on three real examples from [45].”