Aggregating Local Deep Features for Image Retrieval

By Artem Babenko, Victor S. Lempitsky. 2015

724 citations recorded by the source corpus.

Several recent works have shown that image descriptors produced by deep convolutional neural networks provide state-of-the-art performance for image classification and retrieval problems. It also has been shown that the activations from the convolutional layers can be interpreted as local features describing particular image regions. These local features can be aggregated using aggregating methods developed for local features (e.g. Fisher vectors), thus providing new powerful global descriptor. In this paper we investigate possible ways to aggregate local deep features to produce compact descriptors for image retrieval. First, we show that deep features and traditional hand-engineered features have quite different distributions of pairwise similarities, hence existing aggregation methods have to be carefully re-evaluated. Such re-evaluation reveals that in contrast to shallow features, the simple aggregation method based on sum pooling provides the best performance for deep convolutional features. This method is efficient, has few parameters, and bears little risk of overfitting when e.g. learning the PCA matrix. In addition, we suggest a simple yet efficient query expansion scheme suitable for the proposed aggregation method. Overall, the new compact global descriptor improves the state-of-the-art on four common benchmarks considerably.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Information Retrieval & Search

Citations in the corpus

Fast ConvNets Using Group-Wise Brain Damage (2015, positive): “ sets such as ILSVRC can be used as universal spatially localized features in a variety of ways [34, 1], which is particularly valuable for problems with considerably smaller training sets. Recently, [3] showed that descriptors obtained by sum-pooling of the features that emerge in the last convolut”
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition (2015, methodology): “The newly collected TokyoTM database (provided on request) was generated from Time Machine panoramas, downloaded using [101], such that each panorama is represented by a set of 12 perspective images sampled evenly in different orientations [2], [3], [6], [8], [10].”
Do We Train on Test Data? Purging CIFAR of Near-Duplicates (2019, methodology): “ation applied to the 2 feature vectors, the Euclidean distance is equivalent to the cosine distance in this case, which is the distance metric used by most state-of-the-art image retrieval techniques [1, 10, 16]. Since we do not simply use features extracted from pre-trained models, but train the ne”
Visual Instance Retrieval with Deep Convolutional Networks (2014, background): “Babenko and Lempitsky 29) suggested that sum-pooling can perform better than max-pooling and Arandjelovic et al. 12) included learning in the pipeline as opposed to the off-the-shelf usage.”
Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval (2020, methodology): “These were learned with different backbones such as ResNet [25], ResNeXt [61], Squeeze-and-Excitation [27], FishNet [51] and Inception-V4 [52]; pooling methods such as SPoC [6], RMAC [54] or GeM [44]; loss functions such as ArcFace [18], CosFace [57], N-pairs [49] and triplet [48].”
On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location (2020, methodology): “Examples of region cropping in CNNs include: The bounding box in object detection [27, 34, 79]; processing a huge resolution image in patches [42, 86]; local image region matching [32, 66, 108, 107]; local CNN patch pooling encoders [3, 6, 8].”
Deep Feature Factorization for Content-Based Image Retrieval and Localization (2019, neutral): “, global max-pooling [4] (MAC, maximum activation of convolutions), sum-pooling [5], and weighted-sum-pooling [6].”
Deep exemplar-based colorization (2018, methodology): “To measure semantic similarity, we adopt the intermediate features of a pre-trained image classification network as descriptors, which have been widely used in recent image retrieval works [Krizhevsky et al. 2012; Babenko et al. 2014; Gong et al. 2014; Babenko and Lempitsky 2015; Razavian et al. 201”
PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval (2017, neutral): “Currently, these pre-trained CNNs and corresponding variants have beenwidely used for various retrieval tasks ranging from computer vision (Chandrasekhar et al., 2016; Yandex and Lempitsky, 2016; Gordo et al., 2016) to remote sensing (Napoletano, 2016; Zhou et al.”
Medical image retrieval using deep convolutional neural network (2017, methodology): “Deep learning methods have been applied to CBIR task in recent studies [8, 17, 18], but there is less attention on exploring deep learning methods for CBMIR task.”
Large-Scale Image Retrieval with Attentive Deep Local Features (2016, background): “Other works have re-evaluated and proposed different feature aggregation methods using such deep local features [3, 21].”
Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval (2016, methodology): “e following, we report the results for ﬁne-grained image retrieval. We compare the proposed method with several baseline approaches and three state-of-the-art general image retrieval approaches, SPoC [23], CroW [25] and R-MAC [27]. The top-1 and top-5 mAP results are reported in TableIII. Firstly, w”
Deep learning for visual understanding (2017, positive): “[232] aggregated the convolutional features in a simple sum-pooling way, and achieved a substantial boost in the performance.”
Sipakmed: A New Dataset for Feature and Image Based Classification of Normal and Pathological Cervical Cells in Pap Smear Images (2018, methodology): “In the current context, we use the trained models to produce deep features by using the pre-activations of the last convolutional layer (layer conv5), as suggested in [10].”
Hierarchical Bilinear Pooling for Fine-Grained Visual Recognition (2018, background): “Since the diﬀerences between subcategories are subtle and local, capturing global semantic information with merely fully connected layers limits the representation capacity of a framework, and hence restricts further promotion of ﬁnal recognition [1].”
SIFT Meets CNN: A Decade Survey of Instance Retrieval (2016, methodology): “We also note in this survey that several other works [140], [133], [134] also employ similar ideas with [10] in employing max or average pooling on the intermediate feature maps and that Razavian et al.”
FedProto: Federated Prototype Learning across Heterogeneous Clients (2021, background): “Aggregated local features can serve as descriptors for image retrieval (Babenko and Lempitsky 2015).”
Deep learning features at scale for visual place recognition (2017, neutral): “B with two other state-of-the-art convolution feature encoding methods – the cross-layer pooling [33] and the holistic pooling [34].”