Fine-Tuning CNN Image Retrieval with No Human Annotation

By Filip Radenovic, Giorgos Tolias, Ondřej Chum. 2017

1,567 citations recorded by the source corpus.

Image descriptors based on activations of Convolutional Neural Networks (CNNs) have become dominant in image retrieval due to their discriminative power, compactness of representation, and search efficiency. Training of CNNs, either from scratch or fine-tuning, requires a large amount of annotated data, where a high quality of annotation is often crucial. In this work, we propose to fine-tune CNNs for image retrieval on a large collection of unordered images in a fully automated manner. Reconstructed 3D models obtained by the state-of-the-art retrieval and structure-from-motion methods guide the selection of the training data. We show that both hard-positive and hard-negative examples, selected by exploiting the geometry and the camera positions available from the 3D models, enhance the performance of particular-object retrieval. CNN descriptor whitening discriminatively learned from the same training data outperforms commonly used PCA whitening. We propose a novel trainable Generalized-Mean (GeM) pooling layer that generalizes max and average pooling and show that it boosts retrieval performance. Applying the proposed method to the VGG network achieves state-of-the-art performance on the standard benchmarks: Oxford Buildings, Paris, and Holidays datasets.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

Citations in the corpus

Fixing the train-test resolution discrepancy (2019, methodology): “Similar pooling techniques have been employed in image retrieval for a few years [29, 38], where high-resolution images are required to achieve a competitive performance.”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “The features are obtained as the concatenation of the output [CLS] token and of the GeM pooled [54] output patch tokens.”
Memory-Efficient Incremental Learning Through Feature Adaptation (2020, methodology): “These methods can be semi-supervised [24,30,49], self-supervised [5,17,39] or perform few-shot learning [12,13,48].”
MultiGrain: a unified image embedding for classes and instances (2019, methodology): “s regionalized (RMAC) or magniﬁes activations (GeM). The triplet loss requires a batching strategy with pairs of matching images. classiﬁcation retrieval spatial pooling avg. pooling RMAC [42] or GeM [34] loss cross-entropy triplet [10] batch sampling diverse similar images in batch whitening no yes”
An Adaptive Random Path Selection Approach for Incremental Learning. (2019, methodology): “Training: For the CIFAR100 dataset, we use resnet-18 along with max pooling after 5th, 7th blocks and global generalized mean pooling (GeM) [27] with pooling parameter 3 is used after 9th block.”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “The baselines include: • Generalized Mean Pooling (GeM) [36], an adaptive pooling function with a single trainable parameter, and is popular in image search literature; • Feature-sorting Pooling (FSPool) [51], a learnable pooling that handles variable-sized inputs by interpolating a fixed-size learn”
Deep Learning for Person Re-Identification: A Survey and Outlook (2020, methodology): “We adopt a learnable pooling layer, named generalized-mean (GeM) pooling [234], formulated by where p k is a pooling hyperparameter, which can be learned in the back-propagation process [234].”
DINOv2: Learning Robust Visual Features without Supervision (2023, background): “A first family of self-supervised methods focuses on pretext tasks built from the image, i.e., extracting a signal from the image to be predicted from the rest of the image.”
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (2021, background): “Recently, many of these components have been learned with great success [2, 23, 25, 58, 60, 65, 67, 71, 95], but often independently and not end-to-end due to the complexity of such systems.”
Rethinking Spatial Dimensions of Vision Transformers (2021, background): “Some studies [6, 27, 5] show that the pooling layer contributes to the expressiveness and generalization performance of the network.”
MetaSearch: Incremental Product Search via Deep Meta-Learning (2020, methodology): “Raden-ovi´c [37] proposed ﬁne-tuning CNNs for image retrieval on a large collection of unordered images.”
Understanding the Limitations of CNN-Based Absolute Camera Pose Regression (2019, background): “State-of-the-art approaches use compact image-level descriptors to enable efficient and scalable retrieval [3, 52, 71].”
Google Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval (2020, methodology): “7M StreetView City Y Landmarks-PointCloud [36] 2012 1k 10k 205k Flickr label + SfM Worldwide Y 24/7 Tokyo [56] 2015 125 315 1k Smartphone + Manual City Y Paris500k [61] 2015 13k 3k 501k Manual City N Landmark URLs [7, 22] 2016 586 140k - Text query + Feature matching Worldwide N Flickr-SfM [45] 2016”
MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detection (2021, background): “nally proposed for image retrieval [41], can be used to overcome the first downside of GMP.”