Training Vision Transformers for Image Retrieval

By Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, Herv'e J'egou. 2021

180 citations recorded by the source corpus.

Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt vision transformers for generating image descriptors and train the resulting model with a metric learning objective, which combines a contrastive loss with a differential entropy regularizer. Our results show consistent and significant improvements of transformers over convolution-based approaches. In particular, our method outperforms the state of the art on several public benchmarks for category-level retrieval, namely Stanford Online Product, In-Shop and CUB-200. Furthermore, our experiments on ROxford and RParis also show that, in comparable settings, transformers are competitive for particular object retrieval, especially in the regime of short vector representations and low-resolution images.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, background): “We have also seen emerged two properties that can be leveraged in future applications: the quality of the features in k-NN classification has a potential for image retrieval where ViT are already showing promising results [22].”
Going deeper with Image Transformers (2021, background): “This choice, which does not employ any class embedding, is typical in convolutional networks, but possibly weaker with transformers when transferring to other tasks [20].”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68].”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [115, 99], 3D reconstruction [72], pose estimation [107], generative modeling [14], image retrieval [27], medical image segmentation [13, 97, 111], point clouds [40], video in”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, methodology): “The vision transformer has been thereafter successfully adapted for a wider range of computer vision tasks including object detection [30], semantic segmentation [31] and image retrieval [32].”
XCiT: Cross-Covariance Image Transformers (2021, background): “Promising results have been obtained for other vision tasks, including image retrieval [22], object detection and semantic segmentation [44, 70, 81, 83], as well as video understanding [2, 7, 23].”
DeiT III: Revenge of the ViT (2022, unknown): “A recent work suggests that such auto-encoders are mostly interesting in a data starving context [15], but this questions their advantage in the case where more labelled data is actually available.”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “Recent approaches adopt strong vision transformers as architectural backbones for retrieval [6, 19, 27, 35, 61].”
CMT: Convolutional Neural Networks Meet Vision Transformers (2021, methodology): “Besides, transformers are also used to solve other vision tasks such as object detection [2, 70], semantic segmentation [69], image retrieval [11], and low-level vision task [3].”
Residual Mixture of Experts (2022, background): “Triggered by ViT, a series of follow-up works [8, 9, 14, 17, 21, 25, 30, 38, 49, 50, 52, 54, 57, 59, 60] continuously refresh the image classification records and push the potential of vision transformers in computer vision to a new height.”
SiT: Self-supervised vIsion Transformer (2021, background): “transformers achieve outstanding results in downstream tasks with minimal changes [3], [4], [5].”
Task-Oriented Multi-User Semantic Communications (2021, background): “While the training algorithms focus on classiﬁcation based learning [30], metric based learning [31], and unsupervised-based learning [32].”