Self-Supervised Intrinsic Image Decomposition

By Michael Janner, Jiajun Wu, Tejas D. Kulkarni, Ilker Yildirim, Josh Tenenbaum. 2017

143 citations recorded by the source corpus.

Intrinsic decomposition from a single image is a highly challenging task, due to its inherent ambiguity and the scarcity of training data. In contrast to traditional fully supervised learning approaches, in this paper we propose learning intrinsic image decomposition by explaining the input image. Our model, the Rendered Intrinsics Network (RIN), joins together an image decomposition pipeline, which predicts reflectance, shape, and lighting conditions given a single image, with a recombination function, a learned shading model used to recompose the original input based off of intrinsic image predictions. Our network can then use unsupervised reconstruction error as an additional signal to improve its intermediate representations. This allows large-scale unlabeled data to be useful during training, and also enables transferring learned knowledge to images of unseen object categories, lighting conditions, and shapes. Extensive experiments demonstrate that our method performs well on both intrinsic image decomposition and knowledge transfer.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…[McCormac et al., 2017, Silberman et al., 2012, Song et al., 2017], many papers start to estimate depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Janner et al., 2017, Shi et al., 2017] with deep networks.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “With the development of depth sensors [23] and larger-scale RGB-D datasets [33,42,43], there have also been papers on estimating depth [6,10], surface normals [1,54], and other intrinsic images [25,41] with deep networks.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “…et al., 2012, Song et al., 2017, McCormac et al., 2017], there have also been papers on estimating depth [Chen et al., 2016, Eigen and Fergus, 2015], surface normals [Bansal and Russell, 2016, Wang et al., 2015], and other intrinsic images [Shi et al., 2017, Janner et al., 2017] with deep networks.”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, methodology): “learning algorithms, such as self-supervised [131] or few-”
3D-Aware Scene Manipulation via Inverse Graphics (2018, background): “Another line of approaches are built on intrinsic image decomposition [Barrow and Tenenbaum, 1978] and have shown promising results on faces [Shu et al., 2017] and objects [Janner et al., 2017].”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “More recently, deep learning-based works have aimed at learning the intrinsic image decomposition from synthetic supervision [47], self supervision [29] or multiview supervision [82].”
CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering (2018, methodology): “Note that models trained on the original SUNCG, Sintel, MIT intrinsics or ShapeNet datasets perform poorly on the SAW test set, indicating the much improved generalization to real scenes of our CGI dataset.”
3D Shape Perception from Monocular Vision, Touch, and Shape Priors (2018, unknown): “Some recent papers have studied the problem of depth and surface normal estimation [18] from a single image.”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, background): “Although our work concentrates on decomposing faces, the problem of inverse rendering for generic objects in a learning based framework has also gained attention in recent years [4, 20, 29, 11].”