Learning to reconstruct shape and spatially-varying reflectance from a single image

By Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, Manmohan Chandraker. ACM Transactions on Graphics, 2018

325 citations recorded by the source corpus.

Reconstructing shape and reflectance properties from images is a highly under-constrained problem, and has previously been addressed by using specialized hardware to capture calibrated data or by assuming known (or highly constrained) shape or reflectance. In contrast, we demonstrate that we can recover non-Lambertian, spatially-varying BRDFs and complex geometry belonging to any arbitrary shape class, from a single RGB image captured under a combination of unknown environment illumination and flash lighting. We achieve this by training a deep neural network to regress shape and reflectance from the image. Our network is able to address this problem because of three novel contributions: first, we build a large-scale dataset of procedurally generated shapes and real-world complex SVBRDFs that approximate real world appearance well. Second, single image inverse rendering requires reasoning at multiple scales, and we propose a cascade network structure that allows this in a tractable manner. Finally, we incorporate an in-network rendering layer that aids the reconstruction task by handling global illumination effects that are important for real-world scenes. Together, these contributions allow us to tackle the entire inverse rendering problem in a holistic manner and produce state-of-the-art results on both synthetic and real data.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/51128b392d1e9f953f1aae5ae035dc97c8b2735f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

NeRD: Neural Reflectance Decomposition from Image Collections (2020, methodology): “[34] is a method that decomposes objects illuminated by environment illumination only into diffuse and specular roughness.”
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting (2021, methodology): “As for the shared specular component, we use the same simplified Disney BRDF model [9, 17] as in prior work [8, 22]:”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, background): “Some recent works also jointly learn some of the intrinsic components of an object, like reflectance and illumination [9, 41], reflectance and shape [22], and normal, BRDF, and distant lighting [34, 23].”
Single image portrait relighting (2019, unknown): “For this reason, previous works [Sengupta et al. 2018; Li et al. 2018] have relied heavily on synthetic, often Lambertian, images for use as training data.”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, methodology): “Inspired by [32, 33], we use a single encoder to capture correlations between material and shape parameters, obtained using four decoders for diffuse albedo (A), roughness (R), normal (N ) and depth (D).”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, background): “[32] and [2] introduce datasets containing full spatially-varying BRDFs, however their models are procedurally generated shapes that do not correspond to real objects.”
NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis (2020, background): “Most recent single-image inverse rendering methods [23, 25, 45, 49, 56, 60] belong to the first category, and use large datasets of images with labeled geometry and materials to train convolutional neural networks to predict these properties.”