LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval

By Siqi Sun, Yen-Chun Chen, Linjie Li, Shuohang Wang, Yuwei Fang, Jingjing Liu. 2021

94 citations recorded by the source corpus.

Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computational cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by extracting pre-cached feature indexes offline, and employing instant dot-product matching online, which significantly speeds up retrieval process. In fact, our LightningDOT achieves superior performance across mainstream ITR benchmarks such as Flickr30k and COCO datasets, outperforming existing pre-trained models that consume 1000 times magnitude of computational hours using the same features.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

  1. Lijuan Wang
  2. Zicheng Liu
  3. Zhe Gan
  4. Jianfeng Wang
  5. Zhengyuan Yang
  6. Xiaowei Hu
  7. Jianfeng Gao
  8. Cordelia Schmid

In Information Retrieval & Search

  1. Zhe Gan

In Multimodal Learning

  1. Lijuan Wang
  2. Jianfeng Wang
  3. Zicheng Liu
  4. Zhe Gan
  5. Zhengyuan Yang
  6. Xiaowei Hu
  7. Yumao Lu
  8. Xiyang Dai

In Representation & Self-Supervised Learning

  1. Lijuan Wang
  2. Zicheng Liu
  3. Zhe Gan
  4. Jianfeng Wang
  5. Xiaowei Hu
  6. Zhengyuan Yang
  7. Ding Jiang
  8. Mang Ye

Citations in the corpus