Putting Objects in Perspective

By Derek Hoiem, Alexei A. Efros, Martial Hebert. 2006

1,001 citations recorded by the source corpus.

Image understanding requires not only individually estimating elements of the visual world but also capturing the interplay among them. In this paper, we provide a framework for placing local object detection in the context of the overall 3D scene by modeling the interdependence of objects, surface orientations, and camera viewpoint. Most object detection methods consider all scales and locations in the image as equally likely. We show that with probabilistic estimates of 3D geometry, both in terms of surfaces and world coordinates, we can put objects into perspective and model the scale and location variance in the image. Our approach reflects the cyclical nature of the problem by allowing probabilistic object hypotheses to refine geometry and vice-versa. Our framework allows painless substitution of almost any object detector and is easily extended to include other aspects of image understanding. Our results confirm the benefits of our integrated approach.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

LabelMe: A Database and Web-Based Tool for Image Annotation (2008, background): “Algorithms that exploit context for object recognition (Torralba 2003; Hoiem et al. 2006) would benefit from datasets with many labeled object classes embedded in complex scenes.”
Monocular Object Instance Segmentation and Depth Ordering with CNNs (2015, background): “Shadows, reflectance and saturation present an additional challenge.”
Visual Relationship Detection with Language Priors (2016, background): “A meaning space of relationships have aided the cognitive task of mapping images to captions [35,36,37,38].”
Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs (2013, background): “For example, [8] has shown that global, 3D geometric information can significantly improve the performance of bottom-up object detectors.”
An empirical study of context in object detection (2009, methodology): “Geometric context aims to capture the coarse 3D geometric structure of a scene, or the “surface layout” [16], which can be used to reason about supporting surfaces [17], occlusions [15], contact points, etc.”
A Tree-Based Context Model for Object Recognition (2012, methodology): “We take the approach in [15] and apply the following coordinate transformations to represent object locations in the 3D-world coordinates: Lx = `x `h Hi, Ly = `y `h Hi, Lz = f `h Hi (2) where f is the distance from observer to the image plane, which we set to 1, and Lz is the distance between the ob”
Spatial Memory for Context Reasoning in Object Detection (2017, neutral): “The ﬁrst type of model incorporates image or scene level context [5, 36, Figure 1: Evidence of image-level context reasoning inside ConvNets.”
Pedestrian Recognition with a Learned Metric (2010, background): “, cars), which can be integrated into the decision process to significantly improve the detection performance [4].”
The Visual Extent of an Object (2011, background): “…and Triggs 2005; Fulkerson et al. 2009; Gould et al. 2009; Shotton et al. 2009), 2D scene gist context (Oliva and Torralba 2001), 3D geometric context (Hoiem et al. 2008; Nedovi´c and Smeulders 2010), and semantic context (Malisiewicz and Efros 2009; Rabinovich et al. 2007; Singhal et al. 2003).”
Finding People in Images and Videos (2006, methodology): “(c) Results obtained using Hoiem et al. [2006] approach which takes into consideration the interplay between rough 3-D scene geometry, approximate camera position/orientation and low-level person and car detectors.”
Adopting Abstract Images for Semantic Scene Understanding (2016, methodology): “To summarize our results, we computed the sum of the MI or CMI scores for different feature types to estimate the total information provided by them.”
Relating Things and Stuff via ObjectProperty Interactions (2014, background): “In specific, horizon lines for two boxes are estimated assuming objects' average heights are known, similar to [18].”
Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions (2007, background): “Many techniques have been explored to recover planar surfaces from point and line correspondences and vanishing points [18, 3, 14, 8].”
Recovering Occlusion Boundaries from a Single Image (2007, background): “Good progress has been made in operationalizing several of Biederman’s rules, including likelihood [27], support [13], size [12], and position [25].”
Toward Holistic Scene Understanding: Feedback Enabled Cascaded Classification Models (2012, background): “We use the data set and the algorithm by Hoiem et al. [33] as the first-layer geometric labeling module.”
Discriminatively Trained Dense Surface Normal Estimation (2014, background): “While resulting 3D reconstructions of such methods are of debatable quality, coarse information about the 3D layout of a scene has shown to help boost the performance of applications such as object detection [1], semantic reasoning [2] or general scene understanding [3].”
Visual Object Tracking for Unmanned Aerial Vehicles: A Benchmark and New Motion Models (2017, background): “(Hoiem, Efros, and Hebert 2008) proposed a simplified camera model which assumes that all the objects of interest rest on the ground plane.”