Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval

By Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulic, Iryna Gurevych. 2021

64 citations recorded by the source corpus.

Current state-of-the-art approaches to cross- modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that attend over all words and objects in an image. While offering unmatched retrieval performance, such models: 1) are typically pretrained from scratch and thus less scalable, 2) suffer from huge retrieval latency and inefficiency issues, which makes them impractical in realistic applications. To address these crucial gaps towards both improved and efficient cross- modal retrieval, we propose a novel fine-tuning framework that turns any pretrained text-image multi-modal model into an efficient retrieval model. The framework is based on a cooperative retrieve-and-rerank approach that combines: 1) twin networks (i.e., a bi-encoder) to separately encode all items of a corpus, enabling efficient initial retrieval, and 2) a cross-encoder component for a more nuanced (i.e., smarter) ranking of the retrieved small set of items. We also propose to jointly fine- tune the two components with shared weights, yielding a more parameter-efficient model. Our experiments on a series of standard cross-modal retrieval benchmarks in monolingual, multilingual, and zero-shot setups, demonstrate improved accuracy and huge efficiency benefits over the state-of-the-art cross- encoders.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Mohit Bansal
4. Lijuan Wang
5. Cordelia Schmid
6. Zicheng Liu
7. Ivan Laptev
8. Linjie Li

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Cordelia Schmid
3. Andrew Zisserman
4. Jianfeng Gao
5. Dacheng Tao
6. Trevor Darrell
7. Ming-Hsuan Yang
8. Xiaogang Wang

In Information Retrieval & Search

1. Andrew Zisserman
2. Josef Sivic
3. Ivan Laptev
4. Zhe Gan
5. Yinfei Yang
6. Jason Baldridge
7. Zi-Yi Dou
8. Ting-Li Chen

In Multimodal Learning

1. Zhe Gan
2. Linjie Li
3. Lijuan Wang
4. Jianfeng Gao
5. Zicheng Liu
6. Jianfeng Wang
7. Josef Sivic
8. Ivan Laptev

Citations in the corpus

• MURAL: Multimodal, Multitask Retrieval Across Languages (2021, methodology): “Unfortunately, such models do not support efficient retrieval (Geigle et al., 2021), and they use object detection, machine translation, bilingual dictionaries and many losses.”
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers (2021, result): “OSCAR results were reproduced from recent work [17].”
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “(11)Notably, the main bottleneck for current models is the retrieval task as this requires pairing each image with every caption (Geigle et al., 2022).”
• Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, methodology): “Further, we explore combining the strengths of both strategies by performing re-ranking as in [16, 31, 32].”
• Partially Does It: Towards Scene-Level FG-SBIR with Partial Input (2022, unknown): “, fast- [79] and slow- [59] retrieval [27, 51].”