Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers

By Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, Andrew Zisserman. 2021

166 citations recorded by the source corpus.

Our objective is language-based search of large-scale image and video datasets. For this task, the approach that consists of independently mapping text and vision to a joint embedding space, a.k.a. dual encoders, is attractive as retrieval scales and is efficient for billions of images using approximate nearest neighbour search. An alternative approach of using vision-text transformers with cross-attention gives considerable improvements in accuracy over the joint embeddings, but is often inapplicable in practice for large-scale retrieval given the cost of the cross-attention mechanisms required for each sample at test time. This work combines the best of both worlds. We make the following three contributions. First, we equip transformer-based models with a new fine-grained cross-attention architecture, providing significant improvements in retrieval accuracy whilst preserving scalability. Second, we introduce a generic approach for combining a Fast dual encoder model with our Slow but accurate transformer-based model via distillation and reranking. Finally, we validate our approach on the Flickr30K image dataset where we show an increase in inference speed by several orders of magnitude while having results competitive to the state of the art. We also extend our method to the video domain, improving the state of the art on the VATEX dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “Cross-attention itself is in widespread use as a mechanism for allowing information from a source to influence processing of a target domain of a different size or structure [18, 20, 53, 88, 64, 68, 34, 51].”
X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks (2022, background): “This task is also popular in recent V+L learning efforts [34, 49].”
Egocentric Video-Language Pretraining (2022, methodology): “We then transfer the pretrained model to downstream tasks relevant to the egocentric domain. modality [27, 28].”
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, unknown): “, fast- [78] and slow- [58] retrieval [26, 50].”