Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

By Filip Radenovic, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondřej Chum. 2018

459 citations recorded by the source corpus.

In this paper we address issues with image retrieval benchmarking on standard and popular Oxford 5k and Paris 6k datasets. In particular, annotation errors, the size of the dataset, and the level of challenge are addressed: new annotation for both datasets is created with an extra attention to the reliability of the ground truth. Three new protocols of varying difficulty are introduced. The protocols allow fair comparison between different methods, including those using a dataset pre-processing stage. For each dataset, 15 new challenging queries are introduced. Finally, a new set of 1M hard, semi-automatically cleaned distractors is selected. An extensive1 comparison of the state-of-the-art methods is performed on the new benchmark. Different types of methods are evaluated, ranging from local-feature-based to modern CNN based methods. The best results are achieved by taking the best of the two worlds. Most importantly, image retrieval appears far from being solved.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Are we done with ImageNet? (2020, background): “Several studies have revisited common computer vision benchmarks [16, 17, 18, 19] and, regarding ImageNet [1] speciﬁcally, identiﬁed various sources of bias and noise [20, 21, 22].”
Emerging Properties in Self-Supervised Vision Transformers (2021, methodology): “We consider the revisited [53] Oxford and Paris image retrieval datasets [50].”
SuperGlue: Learning Feature Matching With Graph Neural Networks (2019, unknown): “The underlying images come from the set of 1M distractor images in the Oxford and Paris dataset [39], split into training, validation, and test sets.”
University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization (2020, result): “Further, we also report results on the revised Oxford and Paris datasets (denoted as ROxf and RPar) [23].”
Large-scale Landmark Retrieval/Recognition under a Noisy and Diverse Dataset (2019, methodology): “Table 2: Performance (mAP [%]) of our models on the leader-board (LB) of Retrieval track (Public/Private), ROxford and ROxford using Medium and Hard evaluation protocols [18].”
Training Vision Transformers for Image Retrieval (2021, unknown): “We use the default hyper-parameters of Radenovi´c et al. (2018b): the models are optimized using Adam (Kingma & Ba, 2015) with small learning rate of 5 .”
Learning and aggregating deep local descriptors for instance-level recognition (2020, methodology): “9 DELF [29][34] −R50 9.”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
LightGlue: Local Feature Matching at Light Speed (2023, methodology): “To balance the difficulty of image pairs, we bin pairs into three categories based on their visual overlap score [19, 56], with intervals [10, 30]%, [30, 50]%, and [50, 70]%.”
Unifying Deep Local and Global Features for Image Search (2020, methodology): “For reporting the memory footprint of DELG/DELG, we follow a similar convention to previous work [46,56] and report total storage required for local and global descriptors.”
Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval (2020, methodology): “3, where we compare against models trained on other datasets, as well as recent state-of-the-art results – including methods based on global descriptors [44, 45], local feature aggregation [43, 53] and unified global+local features [10].”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, methodology): “We use the cleaned-up version with corrected labels [48].”
Asymmetric metric learning for knowledge transfer (2020, methodology): “We use the revisited R Oxford5k and R Paris6k datasets [52] for testing, each having 70 query images.”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, methodology): “In practice, we use pairs of randomly transformed images from the distractors added recently to the Oxford and Paris retrieval dataset [37], that are basically images from the web.”
Channel Expansion Convolutional Network for Image Classification (2020, background): “CNN has made significant achievements in image tasks such as image classification [1], [2], object detection [3], [4], image captioning [5], [6] and image retrieval [7], [8].”
Visual Product Search Benchmark (2026, unknown): “Early benchmarks such as the Oxford and Paris landmark datasets, later revisited as ROxford and RParis, established standardized evaluation procedures and enabled systematic comparison of retrieval methods across varying levels of visual difficulty [28].”