Deep Image Retrieval: Learning Global Representations for Image Search

By Albert Gordo, Jon Almazán, Jérôme Revaud, Diane Larlus. 2016

855 citations recorded by the source corpus.

We propose a novel approach for instance-level image retrieval. It produces a global and compact fixed-length representation for each image by aggregating many region-wise descriptors. In contrast to previous works employing pre-trained deep networks as a black box to produce features, our method leverages a deep architecture trained for the specific task of image retrieval. Our contribution is twofold: (i) we leverage a ranking framework to learn convolution and projection weights that are used to build the region features; and (ii) we employ a region proposal network to learn which regions should be pooled to form the final global descriptor. We show that using clean training data is key to the success of our approach. To that aim, we use a large scale but noisy landmark dataset and develop an automatic cleaning approach. The proposed architecture produces a global image representation in a single forward pass. Our approach significantly outperforms previous approaches based on global descriptors on standard datasets. It even surpasses most prior works based on costly local descriptor indexing and spatial verification. Additional material is available at www.xrce.xerox.com/Deep-Image-Retrieval.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Exploring the Limits of Weakly Supervised Pretraining (2018, methodology): “To address this issue, we performed the following deduplication procedure: we compute R-MAC features [22,23] for all candidate images using a ResNet-50 model, and use these features to find the k= 21 nearest neighbors for each of the images in our test sets (additional details are in the supplementa”
Billion-Scale Similarity Search with GPUs (2017, methodology): “The L2 distance is used most often, as it is optimized by design when learning several embeddings ( e.g. , [20]), due to its attractive linear algebra properties.”
Revisiting IM2GPS in the Deep Learning Era (2017, methodology): “We are studying image retrieval geolocalization which has overlap with instance-level scene retrieval [33, 2, 23, 7]; Since this line of work mostly focuses on instance-level matching, benchmarks designed for this task consist of popular scenes or landmarks [21, 22, 12] and similarity between…”
MultiGrain: a unified image embedding for classes and instances (2019, methodology): “56 ResNet-50 RMAC [10] 724 90.”
Learning Metrics From Teachers: Compact Networks for Image Embedding (2019, background): “Furthermore, embedding networks are essential for computer vision, as evidenced by the large variety of tasks in which they are used, including feature-based object retrieval [9], face recognition [25], feature matching [6], domain adaptation [27], weakly supervised learning [36], ranking [35], or z”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, methodology): “This article extends our previous work (Gordo et al. 2016) in the followingmanner: we consider residual network architectures as an alternative when constructing our global”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, background): “Recently, neural networks trained with a ranking loss considering image pairs [27], triplets [35], quadruplets [5] or beyond [32], have been considered for metric learning [17, 35] and for a broad range of search tasks such as face/person identiﬁcation [29, 5, 16, 2] or instance retrieval [10, 27].”
VERI-Wild: A Large Dataset and a New Method for Vehicle Re-Identification in the Wild (2019, background): “Previous attempts [6, 28] focus on seeing more hard negatives by mining them from training set.”
Differentiable Top-k with Optimal Transport (2020, background): “For example, in image retrieval (Babenko et al., 2014; Radenović et al., 2016; Gordo et al., 2016), one needs to query the k nearest neighbors of an input image under certain metrics; in the beam search (Reddy et al.”
Asymmetric metric learning for knowledge transfer (2020, background): “Convolutional networks quickly outperformed shallow representations, using different pooling mechanisms [56, 68] and fine-tuning on relevant datasets, initially with cross-entropy on noisy labels from the web [2] and then with contrastive [53] and triplet [16] loss on labels generated from the visua”
Real-World Anomaly Detection in Surveillance Videos (2018, methodology): “have been used for feature learning [38], highlight detection [41], Graphics Interchange Format (GIF) generation [17], face detection and verification [33], person re-identification [13], place recognition [6], metric learning and image retrieval [16].”
Few-Shot Open-Set Recognition Using Meta-Learning (2020, background): “In result, large-scale classifier embeddings underperform on tasks that require generalization beyond the training set, such as image retrieval [7], face identification [17], pose invariant recognition [10], person re-identification [38], or few-shot learning [33].”
Task-Oriented Multi-User Semantic Communications (2021, background): “The deep network architectures include single feedforward pass models [26], multiple feedforward pass models [27], attention based models [28], and deep hashing embedding based models [29].”
Large-Scale Image Retrieval with Attentive Deep Local Features (2016, methodology): “In the dataset, there are the “full” version, referred to as LF (after removal of overlapping classes with Oxf5k/Par6k, by [11]), containing 140,372 images from 586 landmarks, and the “clean” version (LC) obtained by a SIFT-based matching procedure [11], with 35,382 images of 586 landmarks.”
Composing Text and Image for Image Retrieval - an Empirical Odyssey (2018, background): “Image retrieval and product search: Image retrieval is an important vision problem and significant progress has been made thanks to deep learning [5, 51, 10, 38]; it has numerous applications such as product search [28], face recognition [44, 36] or image geolocalization [13].”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “The CNN activations are heavily used to generate the hash codes for efficient image retrieval by employing the different losses [71], [74], [77].”
Learning Compatible Embeddings (2021, background): “Visual search or retrieval systems [37, 38] are widely used in many real-world applications such as face recognition [43, 40, 34, 27, 28], person re-identiﬁcation [41, 17, 35], car re-identiﬁcation [20] and image retrieval [2, 12].”
SIFT Meets CNN: A Decade Survey of Instance Retrieval (2016, methodology): “Second, among the CNN-based categories, the fine-tuned category [8], [17], [24] is advantageous in specific tasks (such as landmark/scene retrieval) which have similar data distribution with the training set.”