A Comprehensive Survey on Cross-modal Retrieval

By Kaiye Wang, Qiyue Yin, Shu Wu, Liang Wang. 2016

332 citations recorded by the source corpus.

In recent years, cross-modal retrieval has drawn much attention due to the rapid growth of multimodal data. It takes one type of data as the query to retrieve relevant data of another type. For example, a user can use a text to retrieve relevant pictures or videos. Since the query and its retrieved results can be of different modalities, how to measure the content similarity between different modalities of data remains a challenge. Various methods have been proposed to deal with such a problem. In this paper, we first review a number of representative methods for cross-modal retrieval and classify them into two main groups: 1) real-valued representation learning, and 2) binary representation learning. Real-valued representation learning methods aim to learn real-valued common representations for different modalities of data. To speed up the cross-modal retrieval, a number of binary representation learning methods are proposed to map different modalities of data into a common Hamming space. Then, we introduce several multimodal datasets in the community, and show the experimental results on two commonly used multimodal datasets. The comparison reveals the characteristic of different kinds of cross-modal retrieval methods, which is expected to benefit both practical applications and future research. Finally, we discuss open problems and future research directions.

Subfields: Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “In Table 5 we compare our results with supervised and unsupervised multi-modal retrieval methods discussed in [40] and [19].”
Achieving Human Parity on Visual Question Answering (2021, background): “The main challenge is the semantic gap [79] between different modalities of data, and the key lies in learning a common representation space, where content similarity between different modalities can be directly measured.”
Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-Modal Pretraining (2021, background): “Furthermore, with the ever-accelerating accumulation of heterogeneous data generated by multimedia, it remains a problem how an algorithm can handle large-scale and weakly annotated data [45] to perform multi-modal retrieval.”
Scalable Deep Multimodal Learning for Cross-Modal Retrieval (2019, background): “With the rapid growth of multimedia data including image, text, video, and audio on the Internet, cross-modal retrieval is becoming increasingly important for the search engine as well as multimedia data management [39].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “Besides, another important topic is cross-modal retrieval [327], where we consider a retrieval setting across different modalities (e.”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, unknown): “including vision-language pre-training (VLP) [7] tasks like image-text retrieval [50] and visual question answering [1], as well as vision specific tasks like zero-shot image classification [23, 41, 58] and object detection [24, 61].”
Learning modality-invariant representations for speech and images (2017, background): “[3] gave a comprehensive overview of existing approaches to another practical application of shared semantic spaces: cross-modality information retrieval.”
Fine-Grained Image Retrieval: the Text/Sketch Input Dilemma (2017, background): “[25] offers an excellent survey of the ﬁeld, including shallow methods and deep learning methods.”
DeepStore: In-Storage Acceleration for Intelligent Queries (2019, background): “, the keys in transactional database and key-value store), and can comprise of images/image-patches [92, 96, 106], sketch [70], color map [91], context map [98, 99], music [72, 97], and many more, intelligent query systems also leverage DNNs to automatically extract the features from query input to ”
Adversarial Cross-Modal Retrieval (2017, methodology): “In order to provide an answer to the above challenge, research towards reliable solutions for cross-modal applications [37, 38], e.g. cross-modal retrieval [32], which operate across modality boundaries, has gained significant momentum recently.”
VLP: A Survey on Vision-language Pre-training (2022, unknown): “Vision-Language Retrieval (VLR) [81, 82, 83, 84].”
Deep Cross-Modal Audio-Visual Generation (2017, background): “Our work differs from other various work in cross-modal retrieval [6, 19, 21, 32] as stated in Sec.”
RGB-Infrared Cross-Modality Person Re-identification (2017, background): “The models for crossmodality retrieval can be classified into real-value representation and binary representation learning [43].”
When Embedding Models Meet: Procrustes Bounds and Applications (2025, unknown): “Models such as CLIP and SigLIP train text and image encoders into a shared embedding space, enabling cross-modal retrieval (Wang et al., 2016).”