Exploiting local features from deep networks for image retrieval

By Joe Yue-Hei Ng, Fan Yang, Larry S. Davis. 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015

424 citations recorded by the source corpus.

Deep convolutional neural networks have been successfully applied to image classification tasks. When these same networks have been applied to image retrieval, the assumption has been made that the last layers would give the best performance, as they do in classification. We show that for instance-level image retrieval, lower layers often perform better than the last layers in convolutional neural networks. We present an approach for extracting convolutional features from different layers of the networks, and adopt VLAD encoding to encode features into a single vector for each image. We investigate the effect of different layers and scales of input images on the performance of convolutional features using the recent deep networks OxfordNet and GoogLeNet. Experiments demonstrate that intermediate layers or higher layers with finer scales produce better results for image retrieval, compared to the last layer. When using compressed 128-D VLAD descriptors, our method obtains state-of-the-art results and outperforms other VLAD and CNN based approaches on two out of three test datasets. Our work provides guidance for transferring deep networks trained on image classification to image retrieval tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/66cd6ea5958c7a6d804d0e9afc296aa0e581e44a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, background): “(2015) [197] on the features extracted from different layers.”
Automatic Classification of Defective Photovoltaic Module Cells in Electroluminescence Images (2018, methodology): “VLAD encoding is sometimes also used for deep learning based features in classification, identification and retrieval tasks [30, 31, 32, 33].”
Deep learning for visual understanding (2017, neutral): “In comparison with prior research [70, 170, 230, 233, 235, 236] which also attempted to incorporate BoW and CNN, BoSP has several di erences: First, BoSP does not need to generate the visual codebook, since the surrogate parts have already been inherently determined by the structure, i.”
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking (2018, methodology): “Methods based on local invariant features [25, 26] and the Bag-of-Words (BoW) model [41, 33, 7, 34, 6, 27, 47, 4, 52, 54, 42] were dominating the field of image retrieval until the advent of CNN-based approaches [38, 3, 48, 20, 1, 10, 37, 28, 51].”
Explicit Shape Encoding for Real-Time Instance Segmentation (2019, background): “Recently, a number of works encode the shape implicitly [9, 29, 38], which is to project the shape content to a latent vector, typically through a black-box design such as deep CNN.”
Dynamic Match Kernel With Deep Convolutional Features for Image Retrieval (2018, methodology): “In recent years, several methods have exploited deep CNNs for image retrieval [37]–[40], thanks to their excellent property”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, methodology): “…feature extraction stage in BoF models, often using intermediate or higher layer features of pretrained DNNs (Feng et al., 2017; Gong et al., 2014; Ng et al., 2015; Mohedano et al., 2016; Cao et al., 2017; Khan et al., 2016) for tasks such as image retrieval or geographical scene classiﬁcation.”
Good Practice in CNN Feature Transfer (2016, methodology): “So its usage as local feature [24] or the inclusion of rotated patches [41] are good choices against these limitations.”
Visual Instance Retrieval with Deep Convolutional Networks (2014, methodology): “Ng et al. 23) replaced max pooling with traditional VLAD over the response map.”
Learning Representations by Predicting Bags of Visual Words (2020, methodology): “Due to its versatility, BoW has been applied to pre-trained convnets as well to compute image representations from intermediate feature maps [22, 44, 68], however few works have dealt with the integration of BoW in the training pipeline of a convnet.”
Unsupervised Joint Mining of Deep Features and Image Labels for Large-Scale Radiology Image Categorization and Scene Recognition (2017, background): “AlexNet is a popular 7-layer CNN architecture and the extracted features from its convolutional or fully-connected layers have been broadly investigated [20, 47, 28, 41].”
Learning and aggregating deep local descriptors for instance-level recognition (2020, background): “A variety of approaches exist [14,35,3,28,50,16] that differ in the training data, in the loss function, or in the global pooling operation.”
Exploit All the Layers: Fast and Accurate CNN Object Detector with Scale Dependent Pooling and Cascaded Rejection Classifiers (2016, background): “The idea of using intermediate convolutional layers to complement high level convolutional features has also been exploited for image classification and segmentation [22, 15, 23], video event detection [40], image retrieval [2, 25] and edge detection [39].”