Learning With Average Precision: Training Image Retrieval With a Listwise Loss

By Jérôme Revaud, Jon Almazán, Rafael Sampaio de Rezende, C. D. Souza, César Roberto De Souza. 2019

436 citations recorded by the source corpus.

Image retrieval can be formulated as a ranking problem where the goal is to order database images by decreasing similarity to the query. Recent deep models for image retrieval have outperformed traditional methods by leveraging ranking-tailored loss functions, but important theoretical and practical problems remain. First, rather than directly optimizing the global ranking, they minimize an upper-bound on the essential loss, which does not necessarily result in an optimal mean average precision (mAP). Second, these methods require significant engineering efforts to work well, e.g., special pre-training and hard-negative mining. In this paper we propose instead to directly optimize the global mAP by leveraging recent advances in listwise loss formulations. Using a histogram binning approximation, the AP can be differentiated and thus employed to end-to-end learning. Compared to existing losses, the proposed method considers thousands of images simultaneously at each iteration and eliminates the need for ad hoc tricks. It also establishes a new state of the art on many standard retrieval benchmarks. Models and evaluation scripts have been made available at: https://europe.naverlabs.com/Deep-Image-Retrieval/.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Unifying Deep Local and Global Features for Image Search (2020, methodology): “We attempt three different initial learning rates for each configuration, and select the best one based on the performance on ROxf/RPar (it is a convention in recent work [48,56,37] to perform ablations/tuning on these datasets).”
Learning and aggregating deep local descriptors for instance-level recognition (2020, neutral): “0 GeMAP[37] R101 7.”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition (2021, methodology): “search in both the computer vision [3, 19, 32, 40, 76, 78, 79] and robotics [10, 11, 12, 25, 36, 42] communities.”
Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval (2020, methodology): “3, where we compare against models trained on other datasets, as well as recent state-of-the-art results – including methods based on global descriptors [44, 45], local feature aggregation [43, 53] and uniﬁed global+local features [10].”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, background): “It was shown to produce state-of-the-art results in patch and image matching [5, 20, 38].”
A Decade Survey of Content Based Image Retrieval Using Deep Learning (2020, background): “in 2019 [228] to directly optimize the global mean average precision in end-toend deep learning.”
Training Vision Transformers for Image Retrieval (2021, positive): “Subsequent works have adopted end-to-end training (Gordo et al., 2016; Radenovi ´ c et al., 2018b; Revaud et al., 2019) with various forms of supervision.”
Do We Train on Test Data? Purging CIFAR of Near-Duplicates (2019, neutral): “feature vectors, the Euclidean distance is equivalent to the cosine distance in this case, which is the distance metric used by most state-of-the-art image retrieval techniques [1, 10, 16].”
DUSt3R: Geometric 3D Vision Made Easy (2023, positive): “To compute camera poses in world coordinates, we use DUSt3R as a 2D-2D pixel matcher (see Section 3.3) between a query and the most relevant database images obtained using off-the-shelf image retrieval AP-GeM [95].”