CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

By Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, Jing Shao. 2019

356 citations recorded by the source corpus.

Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In this paper, we propose Cross-modal Adaptive Message Passing (CAMP), which adaptively controls the information flow for message passing across modalities. Our approach not only takes comprehensive and fine-grained cross-modal interactions into account, but also properly handles negative pairs and irrelevant information with an adaptive gating scheme. Moreover, instead of conventional joint embedding approaches for text-image matching, we infer the matching score based on the fused features, and propose a hardest negative binary cross-entropy loss for training. Results on COCO and Flickr30k significantly surpass state-of-the-art methods, demonstrating the effectiveness of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Multimodal Learning

Citations in the corpus

Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, background): “There are two classic cross-modal retrieval tasks including searching for images through a text [34], [35], and searching for texts through an image [36], [37].”
Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval (2020, background): “Great strides have been made towards cross-modal image retrieval [12, 28], predominantly using text as a query [20, 45].”
Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “…visual and textual observations, a large proportion of methods (Liu et al. 2017; Nam, Ha, and Kim 2017; Lee et al. 2018; Song and Soleymani 2019; Wang et al. 2019c; Li et al. 2019; Wang et al. 2020) (Faghri et al. 2017) map the whole image and the full sentence into a common vector space, and…”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, background): “, text-to-image retrieval [44], image caption [48], visual question answering [1], referring segmentation [19,49,50] and so on.”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, methodology): “Unlike methods that employ cross-modal reasoning modules [24, 28, 30, 31, 34, 53, 54, 60], computing match probabilities at test time for PCME reduces to computing a function over pairwise Euclidean distances.”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “Later meth-ods with cross-attention (Lee et al., 2018, 2019; Wang et al., 2019; Zhang et al., 2020) become more popular, with signiﬁcant performance gain.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “For example, the stacked cross-attention network [15] and an adaptive message passing method [33] are designed to capture cross-modal pairwise interactions.”
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, methodology): “To overcome the constraint that the parallel corpus size places on the model performance, we propose a novel pre-training objective inspired by NAT (Gu et al., 2017; Wang et al., 2019a) and BT methods called BTMLM to align cross-lingual semantics with the monolingual corpus.”