Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories

By Jian Shi, Yue Dong, Hao Su, Stella X. Yu. 2016

207 citations recorded by the source corpus.

We focus on the non-Lambertian object-level intrinsic problem of recovering diffuse albedo, shading, and specular highlights from a single image of an object. Based on existing 3D models in the ShapeNet database, a large-scale object intrinsics database is rendered with HDR environment maps. Millions of synthetic images of objects and their corresponding albedo, shading, and specular ground-truth images are used to train an encoder-decoder CNN, which can decompose an image into the product of albedo and shading components along with an additive specular component. Our CNN delivers accurate and sharp results in this classical inverse problem of computer vision. Evaluated on our realistically synthetic dataset, our method consistently outperforms the state-of-the-art by a large margin. We train and test our CNN across different object categories. Perhaps surprising especially from the CNN classification perspective, our intrinsics CNN generalizes very well across categories. Our analysis shows that feature learning at the encoder stage is more crucial for developing a universal representation across categories. We apply our model to real images and videos from Internet, and observe robust and realistic intrinsics results. Quality non-Lambertian intrinsics could open up many interesting applications such as realistic product search based on material properties and image-based albedo/specular editing.

Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, background): “Shi et al. [66] provide photorealistic renderings of individual objects in the ShapeNet dataset [16], where each image is factored into diffuse reﬂectance, diffuse illumination, and a non-diffuse residual term.”
Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…[McCormac et al., 2017, Silberman et al., 2012, Song et al., 2017], many papers start to estimate depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Janner et al., 2017, Shi et al., 2017] with deep networks.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “With the development of depth sensors [23] and larger-scale RGB-D datasets [33,42,43], there have also been papers on estimating depth [6,10], surface normals [1,54], and other intrinsic images [25,41] with deep networks.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “…et al., 2012, Song et al., 2017, McCormac et al., 2017], there have also been papers on estimating depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Shi et al., 2017, Janner et al., 2017] with deep networks.”
CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering (2018, methodology): “Note that models trained on the original SUNCG, Sintel, MIT intrinsics or ShapeNet datasets perform poorly on the SAW test set, indicating the much improved generalization to real scenes of our CGI dataset.”
Self-Supervised Intrinsic Image Decomposition (2017, methodology): “In the absence of any unsupervised training, our model is similar to that of Shi et al. [2017] adapted to predict the full set of intrinsic images.”
NeRD: Neural Reflectance Decomposition from Image Collections (2020, background): “In singleimage estimation, the inherent ambiguity between shape, shading, and reflectance is often tackled with deep learning approaches from labeled data [36, 48, 58], unlabeled [40] and partially labeled data [8, 39, 49, 74].”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, background): “Recent advances in machine learning address this issue by learning the priors of 3D properties, such as shape, reflectance or illumination [2, 4, 43] to reduce the searching space.”
OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets (2020, unknown): “Previous work typically provides material annotations in the form of simple diffuse or Phong reflectance [42, 39].”
Learning to reconstruct shape and spatially-varying reflectance from a single image (2018, methodology): “We also compare with the recent intrinsic image decomposition method of [Shi et al. 2017], which is trained to separate diffuse and specular components from a single image of a ShapeNet object [Chang et al. 2015], rendered under the assumption of a parametric homogeneous BRDF.”
Neural Inverse Rendering of an Indoor Scene From a Single Image (2019, background): “Some recent works also jointly learn some of the intrinsic components of an object, like reflectance and illumination [9, 41], reflectance and shape [22], and normal, BRDF, and distant lighting [34, 23].”
Differentiable surface splatting for point-based geometry processing (2019, methodology): “The differentiable rendering is also relates to shape-from-shading techniques [Langguth et al. 2016; Maier et al. 2017; Sengupta et al. 2018; Shi et al. 2017] that extract shading and albedo information for geometry processing and surface reconstruction.”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, methodology): “However recent advances in skip-connection based convolutional encoder-decoder architectures for image to image translations [24, 10, 40] have also motivated their use in [29].”
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets (2020, methodology): “Previous works typically provide material annotations as simple diffuse or Phong reﬂectance [51, 47], while we provide a physically-based microfacet SVBRDF. Assigning materials to ShapeNet Many ShapeNet CAD models do not have texture coordinates, so we use Blender’s [2] cube projection UV mapping…”