Lost in quantization: Improving particular object retrieval in large scale image databases

By James Philbin, Ondřej Chum, Michael Isard, Josef Sivic, Andrew Zisserman. 2008

1,690 citations recorded by the source corpus.

The state of the art in visual object retrieval from large databases is achieved by systems that are inspired by text retrieval. A key component of these approaches is that local regions of images are characterized using high-dimensional descriptors which are then mapped to ldquovisual wordsrdquo selected from a discrete vocabulary.This paper explores techniques to map each visual region to a weighted set of words, allowing the inclusion of features which were lost in the quantization stage of previous systems. The set of visual words is obtained by selecting words based on proximity in descriptor space. We describe how this representation may be incorporated into a standard tf-idf architecture, and how spatial verification is modified in the case of this soft-assignment. We evaluate our method on the standard Oxford Buildings dataset, and introduce a new dataset for evaluation. Our results exceed the current state of the art retrieval performance on these datasets, particularly on queries with poor initial recall where techniques like query expansion suffer. Overall we show that soft-assignment is always beneficial for retrieval with large vocabularies, at a cost of increased storage requirements for the index.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Aggregating Local Image Descriptors into Compact Codes (2012, background): “One among the most popular [27], [28] consists in using soft quantization techniques instead of a k-means.”
Product Quantization for Nearest Neighbor Search (2011, methodology): “First, the number of possible distances is signiﬁcantly higher than for competing Hamming embedding methods [20], [17], [19], as the Hamming space used in these techniques allows for a few distinct distances only.”
Revisiting IM2GPS in the Deep Learning Era (2017, background): “…scene retrieval [33, 2, 23, 7]; Since this line of work mostly focuses on instance-level matching, benchmarks designed for this task consist of popular scenes or landmarks [21, 22, 12] and similarity between matched images are visually recognizable (by humans or geometric veriﬁcation).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “We consider the revisited [43] Oxford and Paris image retrieval datasets [40].”
SUN database: Large-scale scene recognition from abbey to zoo (2010, background): “An image is represented by two 1,000 dimension histograms where each SIFT is soft-assigned, as in [22], to its nearest cluster centers.”
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition (2014, background): “Paris6k buildings[35] Similar to the Oxford5k, this collection has 55 queries images of buildings and monuments from Paris and 6412 reference photos.”
Deep Clustering for Unsupervised Learning of Visual Features (2018, neutral): “5 points the mAP of a supervised VGG-16 on the Oxford dataset, but not on Paris.”
SUN Database: Exploring a Large Collection of Scene Categories (2014, background): “histograms where each SIFT is soft-assigned, as in Philbin et al. (2008), to its nearest cluster centers.”
Particular object retrieval with integral max-pooling of CNN activations (2015, methodology): “We evaluate the proposed methods on Oxford Buildings dataset (Philbin et al., 2007) and Paris dataset (Philbin et al., 2008), which are composed of 5063 and 6412 images, respectively.”
From generic to specific deep representations for visual recognition (2014, background): “PASCAL VOC Object [9] H3D human attributes [6] Cat&Dog breeds [25] VOC Human Action [9] Holiday scenes [16] MIT 67 Indoor Scenes [29] Object attributes [10] Bird subordinate [39] Stanford 40 Actions [42] Paris buildings [27] SUN 397 Scene [41] SUN scene attributes [26] 102 Flowers [23] Visual Phrase”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, methodology): “2007), the Paris 6k dataset (Philbin et al. 2008), the INRIAHolidays dataset (Jégou et al.”
TI-POOLING: Transformation-Invariant Pooling for Feature Learning in Convolutional Neural Networks (2016, methodology): “We describe this relation in more details in section 3.3.”
Learnable pooling with Context Gating for video classification (2017, methodology): “For bag-of-visual-words (BOW) encoding, we use softassignment of descriptors to visual word clusters [23], [43] to obtain a differentiable representation.”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “Oxford [45] and Paris [46] are geographic landmark datasets where query and gallery images contain the same building.”
MetaSearch: Incremental Product Search via Deep Meta-Learning (2020, methodology): “In addition to the RPC dataset [52], we also evaluate these methods on the Oxford5k [33], Paris6k [34], and Holidays [26] datasets.”
Deep Image Retrieval: Learning Global Representations for Image Search (2016, methodology): “We experiment mostly with the Oxford 5k building dataset [20] and the Paris 6k dataset [54],”
Improving Bag-of-Features for Large Scale Image Search (2010, positive): “615, which is significantly better than the state-of-the-art (Philbin et al. 2008) of 0.”
Sketch-based manga retrieval using manga109 dataset (2015, methodology): “Various extensions of BoF have been proposed so far, including query expansion [10], soft voting [11], and Hamming embedding [12].”