Multi-scale Orderless Pooling of Deep Convolutional Activation Features

By Yunchao Gong, Liwei Wang, Ruiqi Guo, Svetlana Lazebnik. 2014

1,117 citations recorded by the source corpus.

Deep convolutional neural networks (CNN) have shown their promise as a universal representation for recognition. However, global CNN activations lack geometric invariance, which limits their robustness for classification and matching of highly variable scenes. To improve the invariance of CNN activations without degrading their discriminative power, this paper presents a simple but effective scheme called multi-scale orderless pooling (MOP-CNN). This scheme extracts CNN activations for local patches at multiple scale levels, performs orderless VLAD pooling of these activations at each level separately, and concatenates the result. The resulting MOP-CNN representation can be used as a generic feature for either supervised or unsupervised recognition tasks, from image classification to instance-level retrieval; it consistently outperforms global CNN activations without requiring any joint training of prediction layers for a particular target dataset. In absolute terms, it achieves state-of-the-art results on the challenging SUN397 and MIT Indoor Scenes classification datasets, and competitive results on ILSVRC2012/2013 classification and INRIA Holidays retrieval datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition (2014, methodology): “In processing test images, our method is 24-102× faster than the R-CNN method, while achieving better or comparable accuracy on Pascal VOC 2007.”
Billion-Scale Similarity Search with GPUs (2017, background): “Popular examples include the text representation known as word2vec [32], representations of images by convolutional neural networks [39, 19], and image descriptors for instance search [20].”
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition (2014, methodology): “Adopting the proposed pipeline of [18] and followed by others [16, 42] we process the extracted 4096 dim features in the following way: L2 normalize → PCA dimensionality reduction → whitening → L2 renormalization.”
Compressing Deep Convolutional Networks using Vector Quantization (2014, background): “…network has achieved great successes in image classiﬁcation (Krizhevsky et al., 2012; Jia, 2013; Donahue et al., 2013; Simonyan et al., 2013; Sermanet et al., 2013; Zeiler & Fergus, 2013), object detection (Girshick et al., 2014), and image retrieval (Razavian et al., 2014; Gong et al., 2014).”
Recent advances in convolutional neural networks (2015, background): “[35] use multi-scale orderless pooling (MOP) to improve the invariance of CNNs without degrading their discriminative power.”
Bilinear CNN Models for Fine-Grained Visual Recognition (2015, methodology): “A number of recent techniques have proposed to use CNN features in an orderless pooling setting such as Fisher vector [7], or VLAD [14].”
Deep Image: Scaling up Image Recognition (2015, methodology): “ each class for training and 20 images for testing. The best result in published work [8] uses a D-CNN descriptor, which is a texture descriptor based on the Fisher Vector (FV), related to the method [19]. The features are extracted using VGG 19 layer model and then a SVM classifier is used. This is”
MultiGrain: a unified image embedding for classes and instances (2019, background): “After the emergence of convolutional neural networks (CNNs) for large-scale classiﬁcation on ImageNet [28, 37], it has become apparent that CNNs trained on classiﬁcation datasets are very competitive image feature extractors for various vision tasks, including instance retrieval [1, 9, 35].”
HCP: A Flexible CNN Framework for Multi-Label Image Classification (2014, background): “Some recent works [6], [11], [15], [17], [30], [33], [34], [36] have demonstrated that CNN models pre-trained on large datasets with data diversity, e.”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, methodology): “…hand-tuned feature extraction stage in BoF models, often using intermediate or higher layer features of pretrained DNNs (Feng et al., 2017; Gong et al., 2014; Ng et al., 2015; Mohedano et al., 2016; Cao et al., 2017; Khan et al., 2016) for tasks such as image retrieval or geographical…”
Particular object retrieval with integral max-pooling of CNN activations (2015, methodology): “They are also employed as region descriptors to be compared to database descriptors (Razavian et al., 2014a) or aggregated in a VLAD manner (Gong et al., 2014).”
R-CNN minus R (2015, background): “In this case proposal generation becomes less important as other speedups such as SPP-CNN [10] become available and may be forego.”
DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection (2014, background): “First, features should capture the most discriminative information of object classes.”
A Deeper Look at Dataset Bias (2015, background): “Indeed, several works demonstrated that DeCAF (as well as Caffe [7], Overfeat [37], VGG-CNN [4] and other speci ﬁ c implementations) can be used as powerful image descriptors [4,17].”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, methodology): “Other approaches proposed hybrid models also involving an encoding technique such as Perronnin and Larlus (2015) that used the FV or Gong et al (2014) and Paulin et al (2015) that considered VLAD.”
SIFT Meets CNN: A Decade Survey of Instance Retrieval (2016, methodology): “Among them, the ImageNet dataset [23] is mostly commonly used.”
DeepID-Net: Object Detection with Deformable Part Based Convolutional Neural Networks (2017, positive): “Because of the power on learning feature representation, deep models have been widely used for object recognition, detection and other vision tasks [1], [6], [7], [19], [20], [21], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47],”