NetVLAD: CNN Architecture for Weakly Supervised Place Recognition

By Relja Arandjelovi'c, Petr Gronát, Akihiko Torii, Tomás Pajdla, Josef Sivic. 2015

3,219 citations recorded by the source corpus.

We tackle the problem of large scale visual place recognition, where the task is to quickly and accurately recognize the location of a given query photograph. We present the following three principal contributions. First, we develop a convolutional neural network (CNN) architecture that is trainable in an end-to-end manner directly for the place recognition task. The main component of this architecture, NetVLAD, is a new generalized VLAD layer, inspired by the "Vector of Locally Aggregated Descriptors" image representation commonly used in image retrieval. The layer is readily pluggable into any CNN architecture and amenable to training via backpropagation. Second, we develop a training procedure, based on a new weakly supervised ranking loss, to learn parameters of the architecture in an end-to-end manner from images depicting the same places over time downloaded from Google Street View Time Machine. Finally, we show that the proposed architecture significantly outperforms non-learnt image representations and off-the-shelf CNN descriptors on two challenging place recognition benchmarks, and improves over current state of-the-art compact image representations on standard image retrieval benchmarks.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, methodology): “Variants of soft k-means clustering with learnable, cluster-specific parameters have been introduced in the computer vision [74] and speech recognition communities [75], but they differ from our approach in that they do not use a recurrent, multi-step update, and do not respect permutation symmetry ”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Which language model? Finally, we also experiment with different language models (1 layer LSTM [35] or GRU [18], 1 layer and 8 attention heads Transformer [75] and NetVLAD with 32 clusters [6]) and compare them to our simple model (see Table 1) in Table 2f.”
CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps (2018, neutral): “NetVLAD [6] is a global image descriptor trained end-to-end for place recognition on street view data using a ranking loss.”
Revisiting IM2GPS in the Deep Learning Era (2017, methodology): “Similar to [2], we can also match images that are not only close in the GPS label space but also close in the current feature space.”
SplitNN-driven Vertical Partitioning (2020, background): “, 2020), NetVLAD (Arandjelović et al., 2016) instead of the pooling mechanisms for a more robust representation learning.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “ce objects (buildings, squares, etc.), conventional object proposals and recognition were stable enough over signiﬁcant viewpoints and condition changes, which make it suitable for place recognition. Arandjelovic et al. (2016) combined the features extracted from CNN models with Vector of Locally Ag”
Bilinear CNN Models for Fine-Grained Visual Recognition (2015, background): “Index Terms—Fine-grained recognition, Texture recognition, Bilinear models, Convolutional Neural Networks, Visualization F”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, methodology): “For sentence representation, we use NetVLAD [46] with 32 clusters.”
Context Encoding for Semantic Segmentation (2018, background): “Recent work has made great progress in generalizing traditional encoders in a CNN framework [1, 58].”
Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet (2019, background): “A set of superﬁcially similar but unrelated approaches are region proposal models (Wei et al., 2016; Tang et al., 2017; 2016; Arandjelovic et al., 2015).”
Particular object retrieval with integral max-pooling of CNN activations (2015, background): “A very recent work (Arandjelovic et al., 2015) shows how MAC performance is improved by end-to-end fine tunning where the objective is based on MAC similarity.”
Use What You Have: Video retrieval using representations from collaborative experts (2019, methodology): “Finally, the word embeddings in each sentence are aggregated using NetVLAD.”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, methodology): “The recent NetVLAD by Arandjelovic et al (2016) also highlights the importance of learning to rank.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “[21] found that us-ing NetVLAD [3] results in an increase in accuracy over GRUs or LSTMs for aggregation of both visual and text features.”
Learnable pooling with Context Gating for video classification (2017, methodology): “However, the codebook can also be learned in a discriminative manner [34], [37], [38] or the entire encoding module can be included within the convolutional neural network architecture and trained in the end-to-end manner [23].”
Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization (2017, background): “Recently, researchers have shown increasing interests in exploring structured layers to enhance representation capability of networks [12, 25, 1, 22].”
Mining on Manifolds: Metric Learning Without Labels (2018, background): “[1] learn visual descriptors for location recognition by assigning labels based on the GPS location of each image.”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, methodology): “It also fails to outperform the simple DenseVLAD baseline on the Street scene, which is the largest and most complex scene in the Cambridge Landmarks dataset [10, 50].”