Unsupervised Discovery of Mid-Level Discriminative Patches

By Saurabh Singh, Abhinav Kumar Gupta, Alexei A. Efros. 2012

596 citations recorded by the source corpus.

The goal of this paper is to discover a set of discriminative patches which can serve as a fully unsupervised mid-level visual representation. The desired patches need to satisfy two requirements: 1) to be representative, they need to occur frequently enough in the visual world; 2) to be discriminative, they need to be different enough from the rest of the visual world. The patches could correspond to parts, objects, "visual phrases", etc. but are not restricted to be any one of them. We pose this as an unsupervised discriminative clustering problem on a huge dataset of image patches. We use an iterative procedure which alternates between clustering and training discriminative classifiers, while applying careful cross-validation at each step to prevent overfitting. The paper experimentally demonstrates the effectiveness of discriminative patches as an unsupervised mid-level visual representation, suggesting that it could be used in place of visual words for many tasks. Furthermore, discriminative patches can also be used in a supervised regime, such as scene classification, where they demonstrate state-of-the-art performance on the MIT Indoor-67 dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Visual Representation Learning by Context Prediction (2015, methodology): “As part of the qualitative evaluation, we applied our algorithm to the subset of Pascal VOC 2007 selected in [47]: specifically, those containing at least one instance of bus,”
Training Region-Based Object Detectors with Online Hard Example Mining (2016, background): “The idea of dataset boot-strapping [25, 30], typically called hard negative mining in recent work [12], appears in the training of most successful object detectors [7, 9, 12, 13, 15, 16, 23, 25, 29].”
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition (2013, background): “pects of a given domain through discovery of salient clusters, parts, mid-level features, and/or hidden units (Hinton & Salakhutdinov, 2006; Fidler & Leonardis, 2007; Zhu et al., 2007; Singh et al., 2012; Krizhevsky et al., 2012).”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “One form of discriminative learning combines clustering with hand-crafted features to learn visual representations such as image-patches [13, 62], object discovery [57, 63].”
Joint Discovery of Object States and Manipulation Actions (2017, methodology): “Our model builds on unsupervised discriminative clustering methods [4, 41, 46] that group data samples according to a simultaneously learned classifier.”
Learning to See by Moving (2015, background): “Recent advances in computer vision have shown that visual features learnt by training neural networks for the task of object recognition using more than a million labelled images are useful for many computer vision tasks like semantic segmentation, object detection and action classification [22, 13,”
Analyzing the Performance of Multilayer Neural Networks for Object Recognition (2014, background): “, [12]), or by first finding a set of mid-level patches [10,20] and then encoding images in terms of them.”
Unsupervised Learning of Visual Representations Using Videos (2015, neutral): “org/patients-and-public/good-vision-throughoutlife/childrens-vision/infant-vision-birth-to-24-months-of-age selves [41, 6, 7].”
Recent advances in convolutional neural networks (2015, methodology): “6) Multi-scale Orderless Pooling: Inspired by [34], Gong et al.”
Learning to Explain With Complemental Examples (2018, background): “Prototype selection [25, 7, 13, 16, 25, 10] or machine teaching [18] can be considered as example-based explanations.”
Interpretable Convolutional Neural Networks (2017, methodology): “Like the extraction of mid-level features [26] from images, pattern retrieval mainly learns midlevel representations from conv-layers.”
Unsupervised Representation Learning by Sorting Sequences (2017, methodology): “Before the resurgence of CNNs, hand-craft features such as SIFT and HOG have been used to discover semantic classes using clustering [33, 36], or mining discriminative mid-level features [35, 6, 39].”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, background): “Our work is also closely related to mid-level patch clustering [46, 7, 8] and unsupervised discovery of semantic classes [43, 47] as we attempt to find reliable clusters in our affinity graph.”
Joint Unsupervised Learning of Deep Representations and Image Clusters (2016, background): “Deep Representation Learning Many works use raw image intensity or hand-crafted features [9, 18, 19, 23, 42, 50] combined with conventional clustering methods.”
Constrained Convolutional Neural Networks for Weakly Supervised Segmentation (2015, background): “Several heuristics have been proposed to address these issues [30, 31], however they are usually specific to detection.”
Context as Supervisory Signal: Discovering Objects with Predictable Context (2014, methodology): “We slightly outperform [15], especially for the high purity regime (we get nearly 10% extra coverage before we make our first mistake).”
Multi-scale Orderless Pooling of Deep Convolutional Activation Features (2014, background): “In fact, several recent methods achieving state-of-the-art results on this dataset are based on the idea of finding such patches [38,37,36].”
Interpreting CNN knowledge via an Explanatory Graph (2017, methodology): “The other two baselines were typical methods to extract middlelevel features from images [20] and extract patterns from CNNs [17], respectively.”
Weakly Supervised Object Localization with Multi-Fold Multiple Instance Learning (2015, neutral): “[42] on unsupervised vocabulary learning for image classifi-”