CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

By Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, Stefan Carlsson. 2014

5,100 citations recorded by the source corpus.

Recent results indicate that the generic descriptors extracted from the convolutional neural networks are very powerful. This paper adds to the mounting evidence that this is indeed the case. We report on a series of experiments conducted for different recognition tasks using the publicly available code and model of the OverFeat network which was trained to perform object classification on ILSVRC13. We use features extracted from the OverFeat network as a generic image representation to tackle the diverse range of recognition tasks of object image classification, scene recognition, fine grained recognition, attribute detection and image retrieval applied to a diverse set of datasets. We selected these tasks and datasets as they gradually move further away from the original task and data the OverFeat network was trained to solve. Astonishingly, we report consistent superior results compared to the highly tuned state-of-the-art systems in all the visual classification tasks on various datasets. For instance retrieval it consistently outperforms low memory footprint methods except for sculptures dataset. The results are achieved using a linear SVM classifier (or L2 distance in case of retrieval) applied to a feature representation of size 4096 extracted from a layer in the net. The representations are further modified using simple augmentation techniques e.g. jittering. The results strongly suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, methodology): “…CV models (including object detection, classiﬁcation, and segmentation) are rarely trained from scratch, but instead are ﬁne-tuned from models that have been pretrained on ImageNet, MS-COCO, and other datasets (Sharif Razavian et al., 2014; Long et al., 2015a; He et al., 2016; Huang et al., 2017).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, neutral): “Transfer learning using supervised features has been successful in several computer vision applications (Razavian et al., 2014).”
Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition (2014, background): “Deep-networks-based approaches have recently been substantially improving upon the state of the art in image classification [16, 31, 24], object detection [12, 33, 24], many other recognition tasks [22, 27, 32, 13], and even non-recognition tasks.”
Do Better ImageNet Models Transfer Better? (2018, methodology): “Features from ResNet models [14, 15] trained on ImageNet consistently outperform features from networks that achieve higher accuracy on ImageNet.”
What makes ImageNet good for transfer learning? (2016, methodology): “Using ImageNet pre-trained CNN features, impressive results have been obtained on several image classification datasets [13, 35], object detection [16, 39], action recognition [41], human pose estimation [9], image segmentation [10], optical flow [47], image captioning [12, 22] and others [26].”
Fine-tuned Language Models for Text Classification (2018, methodology): “…models (including object detection, classification, and segmentation) are rarely trained from scratch, but instead are fine-tuned from models that have been pretrained on ImageNet, MS-COCO, and other datasets (Sharif Razavian et al., 2014; Long et al., 2015a; He et al., 2016; Huang et al., 2017).”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “While there is a large body of work on transfer learning, much of it assumes a ﬁxed capacity model [32, 3, 6, 58, 15].”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “Recent work attempted to tackle this problem either by better training algorithms [21, 25, 33, 51], structure sharing [36, 41, 49], episodic memory [9, 27, 29], machine-generated pseudo-data [18, 26, 42], or a combination of these approaches [29, 39].”
Billion-Scale Similarity Search with GPUs (2017, background): “Popular examples include the text representation known as word2vec [32], representations of images by convolutional neural networks [39, 19], and image descriptors for instance search [20].”
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (2019, background): “Representation learning is studied in many contexts: supervised pre-training (Sharif Razavian et al., 2014), self-supervised learning (Doersch et al., 2015), semi-supervised learning (Chapelle et al., 2009), generative modeling (Don-ahue et al., 2017), and disentanglement learning (Higgins et al.,…”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “The prevailing paradigm for learning visual representations is first to pretrain a convolutional network [1, 2] to perform image classification on ImageNet [3, 4], then transfer the learned features to downstream tasks [5, 6].”
Fine-Grained Image Classification via Combining Vision and Language (2017, background): “This has been proved by a large amount of recognition tasks, such as object detection, texture recognition and ﬁne-grained image classiﬁcation [29, 30, 31, 32] etc.”
PlaNet - Photo Geolocation with Convolutional Neural Networks (2016, methodology): “A recent study [39] showed that the activations of Overfeat [44], a CNN trained on ImageNet [9] can serve as powerful features for several computer vision tasks, including image retrieval.”
Learning Visual Features from Large Weakly Supervised Data (2015, background): “…studies have shown that using visual features extracted from convolutional networks trained on large object recognition datasets [26, 43, 46] can lead to state-of-the-art results on many vision problems including ﬁne-grained classiﬁcation [21, 39], object detection [15], and segmentation [36].”
Webly Supervised Learning of Convolutional Networks (2015, methodology): “We did not use any data augmentation or spatial pooling technique, with the only pre-processing step being normalizing the feature vector to unit l2 length [39].”
Memory-based control with recurrent neural networks (2015, background): “These networks, with convolutional layers have proven effective at many sensory processing tasks [11, 18], and been demonstrated to be effective for scaling reinforcement learning to large state spaces [14, 16].”
Adversarial Feature Learning (2016, methodology): “After training these models on a massive database of image-label pairs like ImageNet [25], the network easily adapts to a variety of similar visual tasks, achieving impressive results on image classification [5, 24, 31] or localization [8, 20] tasks.”