End-to-End Learning of Deep Visual Representations for Image Retrieval

By Albert Gordo, Jon Almazán, Jérôme Revaud, Diane Larlus. 2016

568 citations recorded by the source corpus.

While deep learning has become a key ingredient in the top performing methods for many computer vision tasks, it has failed so far to bring similar improvements to instance-level image retrieval. In this article, we argue that reasons for the underwhelming results of deep methods on image retrieval are threefold: (1) noisy training data, (2) inappropriate deep architecture, and (3) suboptimal training procedure. We address all three issues. First, we leverage a large-scale but noisy landmark dataset and develop an automatic cleaning method that produces a suitable training set for deep retrieval. Second, we build on the recent R-MAC descriptor, show that it can be interpreted as a deep and differentiable architecture, and present improvements to enhance it. Last, we train this network with a siamese architecture that combines three streams with a triplet loss. At the end of the training process, the proposed architecture produces a global image representation in a single forward pass that is well suited for image retrieval. Extensive experiments show that our approach significantly outperforms previous retrieval approaches, including state-of-the-art methods based on costly local descriptor indexing and spatial verification. On Oxford 5k, Paris 6k and Holidays, we respectively report 94.7, 96.6, and 94.8 mean average precision. Our representations can also be heavily compressed using product quantization with little loss in accuracy.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Fixing the train-test resolution discrepancy (2019, background): “Furthermore, advances in image classification translate to improved results on many other tasks [12, 18].”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, methodology): “These methods can be semi-supervised [24,30,49], self-supervised [5,17,39] or perform few-shot learning [12,13,48].”
MultiGrain: a unified image embedding for classes and instances (2019, methodology): “Gordo et al. [10, 11] show that ﬁne-tuning these regionally-aggregated representations end-to-end on an external image retrieval dataset using a ranking loss yields signiﬁcant improvements for instance retrieval.”
Fine-Tuning CNN Image Retrieval with No Human Annotation (2017, methodology): “This makes us directly comparable to [27].”
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking (2018, methodology): “Numerous methods of image retrieval [7, 31, 5, 27, 47, 3, 48, 20, 37, 10] and visual localization [9, 1] have used these datasets for evaluation.”
Unifying Deep Local and Global Features for Image Search (2020, methodology): “As a matter of fact, several of the recently-proposed global features [47,17] use image pyramids with 3 scales; our results indicate that their performance can be improved substantially by adding a local feature head, with small increase in extraction latency, and without degrading the global featur”
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation (2021, background): “trieval [42, 10], speech recognition enhanced with video [9, 28], visual question answering [1, 49], visual captioning [48, 26], speech to image generation [44, 21], and textto-image generation [33, 46], etc.”
Learned Contextual Feature Reweighting for Image Geo-Localization (2017, methodology): “[15] that uses a region proposal network [44] to learn which regions should be pooled to form a global image descriptor.”
One-Shot Instance Segmentation (2018, background): “Both of these setups are closer to particular object retrieval [60, 53, 19], as they localize instances of a particular object rather than instances of the same object category, as is the focus of our work.”
Learning With Average Precision: Training Image Retrieval With a Listwise Loss (2019, background): “Specifically, in order to use hard negative mining (HNM) as in [1], one would need to find correct values for at least 6 hyperparameters: the number N of images in mining pool; the n for when selecting the top-n hardest triplets for mining; the number k of times before updating good candidates; the ”
Label Propagation for Deep Semi-Supervised Learning (2019, background): “Alternatively, in self-supervised learning, one can take advantage of additional information like spatial layout in images [5, 12] or temporal relation in videos [40, 28]; or mine for such information in unstructured data by algorithmic supervision using conventional methods [13, 30].”
Universal Multimodal Representation for Language Understanding (2023, methodology): “Triplet loss [54], [55], [56] is used to enable the images to converge correctly to improve the performance of the proposedmethod: Fig.”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, methodology): “(2017) [78] and a triplet quantization loss is used by Zhou et al.”
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features (2019, background): “The task of image retrieval [3, 18, 37, 40,60,61] also deals with finding correspondences between images in challenging situations with strong illumination or viewpoint changes.”