Context-based vision system for place and object recognition

By Antonio Torralba, Kevin P. Murphy, William T. Freeman, Mark A. Rubin. 2003

1,013 citations recorded by the source corpus.

While navigating in an environment, a vision system has to be able to recognize where it is and what the main objects in the scene are. We present a context-based vision system for place and object recognition. The goal is to identify familiar locations (e.g., office 610, conference room 941, main street), to categorize new environments (office, corridor, street) and to use that information to provide contextual priors for object recognition (e.g., tables are more likely in an office than a street). We present a low-dimensional global image representation that provides relevant information for place recognition and categorization, and show how such contextual information introduces strong priors that simplify object recognition. We have trained the system to recognize over 60 locations (indoors and outdoors) and to suggest the presence and locations of more than 20 different object types. The algorithm has been integrated into a mobile system that provides realtime feedback to the user.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Relation Networks for Object Detection (2017, background): “Object relation in post-processing Most early works use object relations as a post-processing step [12, 17, 46, 47, 36, 17].”
SUN database: Large-scale scene recognition from abbey to zoo (2010, background): “How many kinds of scenes are there? How can the knowledge about environmental scenes be organized? How do the current state-of-art scene models perform on more realistic and ill-controlled environments, and how does this compare to human performance? To date, computational work on scene and place re”
Recognizing indoor scenes (2009, background): “, [15, 16]), but to the best of our knowledge none of them have dealt with the general problem of recognizing a wide range of indoor scenes categories.”
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks (2018, background): “Several pioneering automated vision systems incorporated context as a component of sophisticated rule-based approaches to image understanding [36, 7]; for tasks such as object recognition and detection, low-dimensional, global descriptors have often proven effective as contextual clues [39, 30, 40].”
SUN Database: Exploring a Large Collection of Scene Categories (2014, background): “Whereas most computational work on scene and place recognition has used a limited number of semantic categories, representing typical indoor and outdoor settings (Lazebnik et al. 2006; Fei-Fei and Perona 2005; Renninger and Malik 2004; Vogel and Schiele 2007; Oliva and Torralba 2001; Barnard et al. ”
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories (2006, methodology): “, partition the image into subblocks and compute histograms (or histogram statistics, such as means) of local features in these subblocks — has been practiced numerous times in computer vision, both for global image description [4, 19, 20] and for local description of interest regions [12].”
Next-active-object prediction from egocentric videos (2017, background): “Towards this direction, researchers have investigated methods to understand the user’s environment [3, 4, 5, 6, 7, 8], model his attention [9, 10], categorize his motion [11, 12], summarize the acquired video [13, 14], recognize per-”
Recovering human body configurations: combining segmentation and recognition (2004, background): “The line of research by Torralba et al. [15] develops relationships between local object detectors and the global scene context.”
Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships (2018, background): “The presence of objects in irrelevant scenes is penalized in [41].”
Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition (2015, background): “As per standard arguments around context [32, 12, 10, 25], using the actions of other people in the scene can help to disambiguate the action of this individual.”
Joint People, Event, and Location Recognition in Personal Photo Collections Using Cross-Domain Context (2010, background): “For example, Torralba et al. [11, 12] used scene context as a prior for object detection and recognition.”
1 On what it means to see , and what we can do about it (2008, positive): “Such integration is at the core of the most promising current approaches to object and scene vision (Fei-Fei et al., 2003; Freeman, 1993; Torralba et al., 2003), including the explicitly generative “analysis by synthesis” methods (Yuille and Kersten, 2006).”
FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance (2008, background): “More recently Torralba et al. (2003) represent places by a set of texture features, and describe a system where localization is cast as estimating the state of a hidden Markov model.”
A Bayesian approach for place recognition (2012, background): “Place recognition was also shown to improve the performance of object recognition when both are performed simultaneously [20].”
Blessing of Dimensionality: High-Dimensional Feature and Its Efficient Compression for Face Verification (2013, methodology): “Examples include multi-scale LBP [9] and multi-scale SIFT [18, 19] for face recognition, Gist descriptor for image retrieval [14], and scene classification [32, 35].”
What makes an image popular? (2014, background): “While there has been a signiﬁcant push in the computer vision community towards detecting objects [15, 5], identifying contextual relationships [45, 52], or classifying scenes [40, 34], little work has been expended towards associating key image components with ‘global spread’ or popularity in an…”
Multiscale conditional random fields for image labeling (2004, methodology): “One method of capturing label relationships is through a more conceptual graphical model, such as an abstraction hierarchy consisting of scenes, objects, and features [14].”