Discovering objects and their location in images

By Josef Sivic, Bryan C. Russell, Alexei A. Efros, Andrew Zisserman, William T. Freeman. 2005

1,145 citations recorded by the source corpus.

We seek to discover the object categories depicted in a set of unlabelled images. We achieve this using a model developed in the statistical text literature: probabilistic latent semantic analysis (pLSA). In text analysis, this is used to discover topics in a corpus using the bag-of-words document representation. Here we treat object categories as topics, so that an image containing instances of several categories is modeled as a mixture of topics. The model is applied to images by using a visual analogue of a word, formed by vector quantizing SIFT-like region descriptors. The topic discovery approach successfully translates to the visual domain: for a small set of objects, we show that both the object categories and their approximate spatial layout are found without supervision. Performance of this unsupervised method is compared to the supervised approach of Fergus et al. (2003) on a set of unseen images containing only one object per image. We also extend the bag-of-words vocabulary to include 'doublets' which encode spatially local co-occurring regions. It is demonstrated that this extended vocabulary gives a cleaner image segmentation. Finally, the classification and segmentation methods are applied to a set of images containing multiple objects per image. These results demonstrate that we can successfully build object class models from an unsupervised analysis of images.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “[33] adopt BoW representation of images for discovering objects in images using pLSA [16] for topic modelling.”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “One form of discriminative learning combines clustering with hand-crafted features to learn visual representations such as image-patches [13, 62], object discovery [57, 63].”
Learning to See by Moving (2015, methodology): “SIFT features are extracted using a square window of size 72 pixels and a stride of 8 pixels using the open source code from [39].”
LabelMe: A Database and Web-Based Tool for Image Annotation (2008, neutral): “Also, we show how to extend the dataset to automatically enhance object labels with WordNet, discover bject parts, recover a depth ordering of objects in a scene, and increase the number of labels using minimal user supervision and images from the web.”
Unsupervised Learning of Visual Representations Using Videos (2015, methodology): “The second class of algorithms use manually defined features such as SIFT or HOG and perform clustering over training data to discover semantic classes [42, 38].”
Learning visual groups from co-occurrences in space and time (2015, background): “Grouping based on co-occurrence has received some prior attention in computer vision (Sivic et al. (2005); Faktor & Irani (2012; 2013); Isola et al. (2014)).”
Unsupervised Representation Learning by Sorting Sequences (2017, methodology): “Before the resurgence of CNNs, hand-craft features such as SIFT and HOG have been used to discover semantic classes using clustering [33, 36], or mining discriminative mid-level features [35, 6, 39].”
Deep roto-translation scattering for object classification (2014, methodology): “In the same spirit, unsupervised deep learning [11] as well as unsupervised bag of words [26] or dictionary learning with spatial pyramid[10] have improved classiﬁcation results previously obtained with engineered feature vectors such as SIFT or HOG, on complex object recognition databases.”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, background): “Our work is also closely related to mid-level patch clustering [46, 7, 8] and unsupervised discovery of semantic classes [43, 47] as we attempt to find reliable clusters in our affinity graph.”
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation (2014, methodology): “There are many pooling techniques [41, 12].”
Using Multiple Segmentations to Discover Objects and their Extent in Image Collections (2006, methodology): “To assess the contributions of the different steps of the algorithm, we evaluate: (a) the proposed algorithm (of figure 2), (b) swapping the LDA model for the simpler pLSA model to evaluate the contribution of the Dirichlet prior over the multinomial weights, (c) using only a single segmentation for”
Context Encoding for Semantic Segmentation (2018, methodology): “Then a visual vocabulary (dictionary) is often learned and the global feature statistics are described by classic encoders such as Bag-of-Words (BoW) [8, 13, 26, 46], VLAD [25] or Fisher Vector [44].”
Visual Relationship Detection with Language Priors (2016, background): “Structured learning approaches have improved scene classiﬁcation along with object detection using hierarchial contextual data from co-occurring objects [15,16,17,18].”
Context as Supervisory Signal: Discovering Objects with Predictable Context (2014, background): “Proponents of unsupervised representation learning [1,2,3,4] and unsupervised object discovery [5,6,7,8,9,10,11,12] have long argued that these approaches have the potential to solve two fundamental problems with supervised methods.”
R-CNN minus R (2015, methodology): “The two-stage computation using region proposal generation is a form of cascade detector [20] or jumping window [17, 19].”
Automatic fruit and vegetable classification from images (2010, background): “Therefore, the system must be able to achieve a high level of precision with only a few training examples (e.g., up to 30 images).”
Automatic Attribute Discovery and Characterization from Noisy Web Data (2010, background): “At an extreme, some work looks at automatically identifying object categories from large numbers of images showing those categories with no per image labels [16, 17], but even here, the set of images is chosen for the task.”