Recovering intrinsic images from a single image

By Marshall F. Tappen, William T. Freeman, Edward H. Adelson. 2005

519 citations recorded by the source corpus.

Interpreting real-world images requires the ability distinguish the different characteristics of the scene that lead to its final appearance. Two of the most important of these characteristics are the shading and reflectance of each point in the scene. We present an algorithm that uses multiple cues to recover shading and reflectance intrinsic images from a single image. Using both color information and a classifier trained to recognize gray-scale patterns, given the lighting direction, each image derivative is classified as being caused by shading or a change in the surface's reflectance. The classifiers gather local evidence about the surface's form and color, which is then propagated using the generalized belief propagation algorithm. The propagation step disambiguates areas of the image where the correct classification is not clear from local evidence. We use real-world images to demonstrate results and show how each component of the system affects the results.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, background): “Through years, researchers have explored recovering 2.5D sketches from texture, shading, or color images [Barron and Malik, 2015, Bell et al., 2014, Horn and Brooks, 1989, Tappen et al., 2003, Weiss, 2001, Zhang et al., 1999].”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, background): “5D sketches from shading, texture, or color images [2,3,20,47,55,63].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, background): “In the past, researchers have explored recovering 2.5D shape from shading, texture, or color images [Horn and Brooks, 1989, Zhang et al., 1999, Tappen et al., 2003, Barron and Malik, 2015, Weiss, 2001, Bell et al., 2014].”
Rendering synthetic objects into legacy photographs (2011, background): “1992] or differently lit images [Weiss 2001] can control these difficulties, as can methods that classify edges into albedo or shading [Tappen et al. 2005; Farenzena and Fusiello 2007].”
Learning Gaussian Conditional Random Fields for Low-Level Vision (2007, background): “Typically, the key to success with Gaussian Markov random ﬁelds is to have the neighboring potential function dependent on the input signal, as in [10, 17].”
Seamless Image Stitching in the Gradient Domain (2004, background): “Similarly to [12,13], we found that good results are obtained when the derivatives are approximated by forward-differencing”
Shape, albedo, and illumination from a single image of an unknown object (2012, background): “This is the insight that underlies the Retinex algorithm [11, 16, 19], and informs more recent intrinsic images work [2, 26, 27, 28].”
Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal (2017, background): “On the contrary, a few works [47, 55, 42] recover the shadow-free image by intrinsic image decomposition and preclude the need of shadow prediction in an end-to-end manner.”
STAR: A Structure and Texture Aware Retinex Model (2019, methodology): “Variational Methods [4]–[11], [20]–[31] have been proposed for Retinex based illumination and reflectance estimation.”
Intrinsic Scene Properties from a Single RGB-D Image (2013, background): “However, it is extremely difficult to produce ground-truth shape, reflectance, shading, and illumination models for real-world natural - - 0.0230 0.0364 0.0354 - - (2) Tappen et al. 2005 [29] - - 0.0281 0.0337 0.0387 - - (3) Gehler et al. 2011 [11] 114.”
User Assisted Separation of Reflections from a Single Image Using a Sparsity Prior (2004, background): “On the related problem of separating shading and reflectance, impressive results have been obtained usin g a single image [19], [3].”
Color Constancy, Intrinsic Images, and Shape Estimation (2012, methodology): “This insight is fundamental to most intrinsic image algorithms [2, 4, 5, 14, 23], and is used in our previous works [1, 22].”
SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' (2017, background): “sion and has been studied in various forms such as intrinsic image decomposition [32] and Shape from Shading (SfS) [22, 21].”
Ieee Transactions on Pattern Analysis and Machine Intelligence Shape, Illumination, and Reflectance from Shading (unknown): “This is the insight that underlies the Retinex algorithm [4], [9], [10], and informs more recent intrinsic images work [13], [14], [15].”
Ground truth dataset and baseline evaluations for intrinsic image algorithms (2009, methodology): “The computer-generated images consisted of shaded ellipsoids with piecewise-constant reflectance [15].”
Flash photography enhancement via intrinsic relighting (2004, background): “Although this type of decoupling is hard [21], [23], [24], tone mapping can get away with a coarse approximation because the layers are eventually recombined.”
Intrinsic Decompositions for Image Editing (2017, unknown): “The methods of Tappen et al. [TFA05,TAF06] use a monochromatic illumination constraint, but comparisons shown in two papers exhibit colored illumination [BPD09, GMLMG12].”