Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image
By Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, Manmohan Chandraker. 2019
329 citations recorded by the source corpus.
We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we obtain a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatially-varying, non-Lambertian surface reflectance. Our novel inverse rendering network incorporates physical insights -- including a spatially-varying spherical Gaussian lighting representation, a differentiable rendering layer to model scene appearance, a cascade structure to iteratively refine the predictions and a bilateral solver for refinement -- allowing us to jointly reason about shape, lighting, and reflectance. Since no existing dataset provides ground truth high quality spatially-varying material and spatially-varying lighting, we propose novel methods to map complex materials to existing indoor scene datasets and a new physically-based GPU renderer to create a large-scale, photorealistic indoor dataset. Experiments show that our framework outperforms previous methods and enables various novel applications like photorealistic object insertion and material editing.
Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
- Jonathan T. Barron
- Andreas Geiger
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthew Tancik
- Angjoo Kanazawa
- Zexiang Xu
- Kalyan Sunkavalli
In Computer Graphics and Computer-Aided Design
- Jonathan T. Barron
- Ben Mildenhall
- Pratul P. Srinivasan
- Andreas Geiger
- Matthew Tancik
- Zexiang Xu
- Angjoo Kanazawa
- Kalyan Sunkavalli
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Andreas Geiger
- Jitendra Malik
- Andrea Vedaldi
- Ming-Hsuan Yang
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
In 3D Vision & Pose
- Jonathan T. Barron
- Zexiang Xu
- Matthew Tancik
- Angjoo Kanazawa
- Ben Mildenhall
- Kalyan Sunkavalli
- Pratul P. Srinivasan
- Noah Snavely
Citations in the corpus
- Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “In response to this challenge, the computer vision community has developed several photorealistic synthetic datasets and interactive simulation environments that have spurred rapid progress towards the goal of holistic indoor scene understanding [5, 6, 8, 9, 13, 14, 17, 20, 22, 29, 31, 34, 35, 37, 4”
- NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis (2020, background): “Most recent single-image inverse rendering methods [23, 25, 45, 49, 56, 60] belong to the first category, and use large datasets of images with labeled geometry and materials to train convolutional neural networks to predict these properties.”
- PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, methodology): “Other than SHs, we also experiment with Spherical Gaussians (SG) [8], a learnable spherical basis which have been used to represent all-frequency lighting [54, 46, 21].”
- Extracting Triangular 3D Models, Materials, and Lighting From Images (2021, methodology): “Recent methods exploit neural networks to predict BRDF from images [15, 20, 34, 35, 39, 45].”
- OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, background): “The best two previous methods [30] (34.”
- PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting (2021, background): “Learning-based single-view methods that recover shape, illumination, and material properties have also been proposed [20, 5].”
- NeRD: Neural Reflectance Decomposition from Image Collections (2020, background): “Capturing the geometry and material properties of an object is essential for several computer vision and graphics applications such as view synthesis [10, 56], relighting [5, 10, 22, 23, 32, 57], object insertion [7, 21, 32] etc .”