Using the forest to see the trees: exploiting context for visual object detection and localization

By Antonio Torralba, Kevin P. Murphy, William T. Freeman. 2010

147 citations recorded by the source corpus.

Recognizing objects in images is an active area of research in computer vision. In the last two decades, there has been much progress and there are already object recognition systems operating in commercial products. However, most of the algorithms for detecting objects perform an exhaustive search across all locations and scales in the image comparing local image regions with an object model. That approach ignores the semantic structure of scenes and tries to solve the recognition problem by brute force. In the real world, objects tend to covary with other objects, providing a rich collection of contextual associations. These contextual associations can be used to reduce the search space by looking only in places in which the object is expected to be; this also increases performance, by rejecting patterns that look like the target but appear in unlikely places. Most modeling attempts so far have defined the context of an object in terms of other previously recognized objects. The drawback of this approach is that inferring the context becomes as difficult as detecting each object. An alternative view of context relies on using the entire scene information holistically. This approach is algorithmically attractive since it dispenses with the need for a prior step of individual object recognition. In this paper, we use a probabilistic framework for encoding the relationships between context and object properties and we show how an integrated system provides improved performance. We view this as a significant step toward general purpose machine vision systems.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Early work in this area includes generating single words or ﬁxed phrases from images (Duygulu et al., 2002; Barnard et al., 2003) or using contextual information to improve recognition (Gupta and Davis, 2008; Torralba et al., 2010).”
Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation (2018, background): “In computer vision, incorporating context information has been shown to improve performance in various tasks including object recognition [12, 22, 18] and action recognition [9], object detection [3] and segmentation [25].”
Recent Advances in Open Set Recognition: A Survey (2018, background): “To meet this challenge, several related research topics have been explored including lifelong learning [1], [2], transfer learning [3], [4], [5], domain adaptation [6], [7], zero-shot [8], [9], [10], one-shot (few-shot) [11], [12], [13], [14], [15], [16], [17], [18], [19], [20] recognition/learning,”
Baby talk: Understanding and generating simple image descriptions (2011, methodology): “…work, and other work on image parsing and object detection, the spatial relationships be-tween labeled parts – either detections or regions – of images was used to improve labeling accuracy, but the spatial relationships themselves were not considered outputs in their own right [24, 7, 16, 21, 15].”
Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering (2016, methodology): “Contextual information outside the tight box around an object has shown to be useful in tasks such as object detection & classiﬁcation [25,26,27] and segmentation [28].”
Understanding web images by object relation network (2012, unknown): “The probabilistic graphical model can be a hierarchical structure [27, 14], a directed graphical model [26], or a conditional random field [10, 20, 11, 12].”
Spatial Semantic Regularisation for Large Scale Object Detection (2015, background): “Spatial and cooccurrence priors are introduced in [2, 28] to prune detection results.”
Learning to place new objects in a scene (2012, background): “Contextual cues (e.g., Torralba et al., 2010) have proven helpful in many vision applications.”
Baby Talk: Understanding and Generating Image Descriptions (2011, methodology): “…and other work on image parsing and object detection, the spatial relationships between labeled parts – either detections or regions – of images was used to improve labeling accuracy, but the spatial relationships themselves were not considered outputs in their own right [24] [7] [16] [21] [15].”
Semi-supervised Vocabulary-Informed Learning (2016, background): “To compensate for the lack of training instances and enable oneshot learning, knowledge much be transferred from other sources, for example, by sharing features [3], semantic attributes [17, 25, 34, 35], or contextual information [41].”
Recursive deep learning for natural language processing and computer vision (2014, background): “, 2003) or using contextual information to improve recognition (Gupta and Davis, 2008; Torralba et al., 2010).”
BabyTalk: Understanding and Generating Simple Image Descriptions (2013, methodology): “These relationships were used as contextual models to improve labeling accuracy, but the spatial relationships themselves were not considered outputs in their own right [10], [20], [38], [41].”