Aggregating Local Image Descriptors into Compact Codes

By Herv'e J'egou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Pérez, Cordelia Schmid. 2012

1,660 citations recorded by the source corpus.

This paper addresses the problem of large-scale image search. Three constraints have to be taken into account: search accuracy, efficiency, and memory usage. We first present and evaluate different ways of aggregating local image descriptors into a vector and show that the Fisher kernel achieves better performance than the reference bag-of-visual words approach for any given vector dimension. We then jointly optimize dimensionality reduction and indexing in order to obtain a precise vector comparison as well as a compact representation. The evaluation shows that the image representation can be reduced to a few dozen bytes while preserving high accuracy. Searching a 100 million image data set takes about 250 ms on one processor core.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Residual Learning for Image Recognition (2015, methodology): “In image recognition, VLAD [18] is a representation that encodes by the residual vectors with respect to a dictionary, and Fisher Vector [30] can be formulated as a probabilistic version [18] of VLAD.”
Return of the Devil in the Details: Delving Deep into Convolutional Nets (2014, methodology): “We also specify here the benchmark datasets used in the evaluation.”
YouTube-8M: A Large-Scale Video Classification Benchmark (2016, methodology): “Note that the up-projection layer into sparse codes is similar to what Fisher Vectors [27] and VLAD [15] approaches do but the projection (i.e., clustering) is done discriminatively here.”
Recent advances in convolutional neural networks (2015, methodology): “The activations of local patches are aggregated by VLAD encoding [36], which aim to capture more local, fine-grained details of the image as well as enhancing invariance.”
What do we need to build explainable AI systems for the medical domain? (2017, methodology): “Here, we are motivated by the fact that ResNet models have provided excellent classiﬁcation results through the use of convolutional neural networks (Jegou et al., 2012; Xie et al., 2016; Huang and Peng, 2017; Taki, 2017).”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “Alternatively we compute the Dense Trajectories [50] for each video interval and encode them with the Fisher vector [17].”
VGGFace2: A Dataset for Recognising Faces across Pose and Age (2017, methodology): “To alleviate this, duplicate images are removed by clustering VLAD descriptors for all images remaining at stage 4 and only retaining one image per cluster [3], [11].”
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition (2015, methodology): “Each database image is represented using local invariant features [19] such as SIFT [20] that are aggregated into a fixed length vector representation for the entire image such as bag-ofvisual-words [21], [22], VLAD [23], [24] or Fisher vector [25], [26].”
MultiGrain: a unified image embedding for classes and instances (2019, neutral): “ report some older related results on the UKB and C10k datasets, that are not competitive with MultiGrain. Neural codes [1] is one of the ﬁrst works on retrieval with deep features. The Fisher vector [26] is a pooling method that uses local SIFT descriptors. At resolutions 500 we see that the result”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “Then encoding methods are employed to aggregate these local descriptors into a global representation, and typical encoding methods include Bag of Visual Words (BoVW) [41], Fisher vector (FV) [42], Vector of Locally Aggregated Descriptors (VLAD) [43], and Multi-View Super Vector (MVSV) [44].”
The ImageNet Shuffle: Reorganized Pre-training for Video Event Detection (2016, background): “Recently, several works have shown that clustering deep frame representations into a codebook, followed by a VLAD [5] in [25] or Fisher Vector encoding [19] in [15], leads to strong video representations.”
Privacy Preserving Image-Based Localization (2019, methodology): “Traditional localization methods based on image retrieval [34,67] and based on learning [12, 35, 81, 82] have the advantage of not requiring the explicit storage of 3D maps.”
Author manuscript, published in "International Journal of Computer Vision (2013)" International Journal of Computer Vision manuscript No. (will be inserted by the editor) Image Classification with the Fisher Vector: Theory and Practice (2013, neutral): “To train the one-vs-rest linear SVMs, we also follow Perronnin et al (2012) and subsample the negatives.”
DeepHash: Getting Regularization, Depth and Fine-Tuning Right (2015, methodology): “• PQ [24].”