Single-image SVBRDF capture with a rendering-aware deep network

By Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, Adrien Bousseau. ACM Transactions on Graphics, 2018

328 citations recorded by the source corpus.

Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, diffuse albedo, specular albedo and specular roughness from a single picture of a flat surface lit by a hand-held flash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading effects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample often offer complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material effects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by defining the loss as a differentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/429aea828954ba77da9cb3a1a8dfd4d9ea4d7101

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Infer Implicit Surfaces without 3D Supervision (2019, background): “Aside from silhouettes, shading and appearances in image space also provides supervision cues for learning fine-grained shape representations in category specific domains such as 3D face reconstruction [31, 32, 33, 34, 35] and material inference [36, 37, 38].”
NeRD: Neural Reflectance Decomposition from Image Collections (2020, background): “Often, planar surfaces under camera flash illumination are considered for single-shot [2, 15, 31, 42], few-shot [2] or multi-shot [3, 9, 16, 17, 19] estimation.”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, background): “Recent advances in 3D face reconstruction [38, 40, 39, 41, 9], material inference [27, 7] and other 3D reconstruction tasks [49, 37, 33, 14, 22, 34] have leveraged some other forms of differentiable rendering layers to obtain gradient flows in the neural networks.”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, background): “In particular, many learning-based techniques [57, 25, 37, 46, 45, 6, 21, 31, 10] have incorporated a differentiable rendering layer to enable an end-to-end architecture for 3D reconstruction and material inference in an unsupervised manner.”
Learning to reconstruct shape and spatially-varying reflectance from a single image (2018, methodology): “Previous inverse rendering methods have utilized 3D shape repositories with homogeneous materials [Liu et al. 2017; Rematas et al. 2016; Shi et al. 2017] or large-scale SVBRDFs with near-planar geometries [Deschaintre et al. 2018; Li et al. 2018].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “Subsequent methods [12, 11, 5, 1] try to estimate materials under natural lighting conditions, assuming the availability of complete geometry information.”
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image (2019, methodology): “The forward image formation model is well understood, and has been used in simple settings like planar scenes and single objects [18, 38, 37, 40].”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, methodology): “[11] propose using a modified U-Net and rendering loss to predict the spatially-varying BRDFs of a flash-lit photograph consisting of only a flat surface.”