CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples

By Filip Radenovic, Giorgos Tolias, Ondřej Chum. 2016

615 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) achieve state-of-the-art performance in many computer vision tasks. However, this achievement is preceded by extreme manual annotation in order to perform either training from scratch or fine-tuning for the target task. In this work, we propose to fine-tune CNN for image retrieval from a large collection of unordered images in a fully automated manner. We employ state-of-the-art retrieval and Structure-from-Motion (SfM) methods to obtain 3D models, which are used to guide the selection of the training data for CNN fine-tuning. We show that both hard positive and hard negative examples enhance the final performance in particular object retrieval with compact codes.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Revisiting IM2GPS in the Deep Learning Era (2017, methodology): “We are studying image retrieval geolocalization which has overlap with instance-level scene retrieval [33, 2, 23, 8]; Since this line of work mostly focuses on instance-level matching, benchmarks designed for this task consist of popular scenes or landmarks [21, 22, 13] and similarity between matche”
A New Representation of Skeleton Sequences for 3D Action Recognition (2017, methodology): “The feature maps in the convolutional layer have been successfully applied for action recognition and image retrieval [19, 20].”
NetVLAD: CNN Architecture for Weakly Supervised Place Recognition (2015, background): “Very recent works [77], [78], published after the first version of this paper [79], train CNNs end-to-end for image retrieval by making use of image correspondences obtained from structurefrom-motion models, i.”
End-to-End Learning of Deep Visual Representations for Image Retrieval (2016, methodology): “A preliminary version of our work (Gordo et al, 2016), together with a concurrent work (Radenovic et al, 2016), confirmed that finetuning the pretrained models for retrieval can bring a significant improvement, but demonstrated that even more crucial are the combination of i) a good image…”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, background): “Recently, neural networks trained with a ranking loss considering image pairs [27], triplets [35], quadruplets [5] or beyond [32], have been considered for metric learning [17, 35] and for a broad range of search tasks such as face/person identiﬁcation [29, 5, 16, 2] or instance retrieval [10, 27].”
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search (2019, methodology): “A core novelty of our approach lies within the method by which visual suggestions are generated, exploiting the reversibility of deep neural networks (DNNs) that are commonly used to encode image features to create the search index in visual search systems [26, 13, 7, 6].”
Composing Text and Image for Image Retrieval - an Empirical Odyssey (2018, background): “Image retrieval and product search: Image retrieval is an important vision problem and significant progress has been made thanks to deep learning [5, 51, 10, 38]; it has numerous applications such as product search [28], face recognition [44, 36] or image geolocalization [13].”
R2D2: Repeatable and Reliable Detector and Descriptor (2019, methodology): “As for the local descriptor, it is trained with a listwise ranking loss, leveraging recent advances in metric learning based on an approximated Average Precision (AP) metric, instead of using a standard triplet or contrastive loss [14, 50, 38].”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “High-performance descriptors can be fine-tuned specifically for retrieval after unsupervised training [49, 50] using structure-from-motion (SfM) or contrastive objectives [14, 27].”
Image Matching Across Wide Baselines: From Paper to Practice (2020, background): “Matching two or more views of a given scene is at the core of fundamental computer vision problems, including image retrieval [48, 7, 69, 91, 63], 3D reconstruction [3, 43, 79, 106], re-localization [74, 75, 51], and SLAM [61, 30, 31].”
Asymmetric metric learning for knowledge transfer (2020, background): “In its most well-known form [16, 53], the task is supervised, but the supervision is originating from automated data analysis rather than humans.”
Sketchformer: Transformer-Based Representation for Sketched Structure (2020, methodology): “Triplet loss models are commonly used for visual search in the photographic domain [21, 22, 23], and have been extended to SBIR.”
Differentiable Top-k with Optimal Transport (2020, background): “For example, in image retrieval (Babenko et al., 2014; Radenović et al., 2016; Gordo et al., 2016), one needs to query the k nearest neighbors of an input image under certain metrics; in the beam search (Reddy et al., 1977; Wiseman and Rush, 2016) algorithm for neural machine translation, one…”
Visual Grounding with Transformers (2021, background): “It can benefit many other multimodal tasks such as visual question answering [13], image caption [17] and crossmodal retrieval [29], to name a few.”
Learning With Average Precision: Training Image Retrieval With a Listwise Loss (2019, methodology): “Ranking losses used to date consider either image pairs [44], triplets [18], quadruplets [11], or n-tuples [52].”
A Discriminatively Learned CNN Embedding for Person Reidentification (2016, background): “on another scene dataset proposed in [41], which comprises of a number of buildings without overlapping with the Oxford5k.”
ASLFeat: Learning Local Features of Accurate Shape and Localization (2020, methodology): “In contrast to D2-Net [8] which starts from an ImageNet pretrained model with only the last convolution ﬁne-tuned, we train our model from scratch with ground-truth cameras and depths obtained from GL3D [35] and [28] (the same data used in [21, 20]).”
Deep Image Retrieval: Learning Global Representations for Image Search (2016, background): “[18], concurrent to ours and published in these same proceedings, that also proposes to learn representations for retrieval using a Siamese network on a geometrically-verified landmark dataset.”
Geolocation Estimation of Photos Using a Hierarchical Model and Scene Classification (2018, methodology): “These mainly apply retrieval techniques to match a query image against a reference dataset [3,12,18,20,29,34,46].”