Multi-Modality Cross Attention Network for Image and Sentence Matching

By Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, Feng Wu. Computer Vision and Pattern Recognition, 2020

451 citations recorded by the source corpus.

The key of image and sentence matching is to accurately measure the visual-semantic similarity between an image and a sentence. However, most existing methods make use of only the intra-modality relationship within each modality or the inter-modality relationship between image regions and sentence words for the cross-modal matching task. Different from them, in this work, we propose a novel MultiModality Cross Attention (MMCA) Network for image and sentence matching by jointly modeling the intra-modality and inter-modality relationships of image regions and sentence words in a unified deep model. In the proposed MMCA, we design a novel cross-attention mechanism, which is able to exploit not only the intra-modality relationship within each modality, but also the inter-modality relationship between image regions and sentence words to complement and enhance each other for image and sentence matching. Extensive experimental results on two standard benchmarks including Flickr30K and MS-COCO demonstrate that the proposed model performs favorably against state-of-the-art image and sentence matching methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/caabcf61499e00c78d8ee692b8939caf98544a9c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays (2021, background): “, masked region feature regression [7, 40], text generation [33, 11, 5, 8], and object classification [34, 24].”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “To further tackle the two aforementioned challenges, recent methods combining intra- and intermodal interactions are developed, such as the serial pattern [22] and the parallel pattern [35, 40] displayed in Figure 2(c).”
ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, background): “Transformer has been applied to multimodal tasks as a result of its significant performance improvement in NLP and CV. Multi-Modality Cross Attention (MMCA) [28] is proposed for image and text matching, which jointly models the intra-modal and inter-modal relationships between image and sentence in ”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, methodology): “Prior work [16, 54] observed a split between one-to-one and many-to-many matching in visual-semantic embedding spaces characterized by the use of one or several embedding representations per image or caption.”
Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “(Song and Soleymani 2019; Wei et al. 2020) employed multi-head self-attention to combine global context with locally-guided features.”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, methodology): “For this reason, some works in image-text matching use BERT to extract contextualized word embeddings for representing sentences [46, 50, 58, 59].”
Changer: Feature Interaction is What You Need for Change Detection (2022, unknown): “VQA [28], RGB-D segmentation [8]), Registration [41], Matching [40] and network architecture design [20].”