Similarity Reasoning and Filtration for Image-Text Matching

By Haiwen Diao, Ying Zhang, Lingyun Ma, Huchuan Lu. 2021

434 citations recorded by the source corpus.

Image-text matching plays a critical role in bridging the vision and language, and great progress has been made by exploiting the global alignment between image and sentence, or local alignments between regions and words. However, how to make the most of these alignments to infer more accurate matching scores is still underexplored. In this paper, we propose a novel Similarity Graph Reasoning and Attention Filtration (SGRAF) network for image-text matching. Specifically, the vector-based similarity representations are firstly learned to characterize the local and global alignments in a more comprehensive manner, and then the Similarity Graph Reasoning (SGR) module relying on one graph convolutional neural network is introduced to infer relation-aware similarities with both the local and global alignments. The Similarity Attention Filtration (SAF) module is further developed to integrate these alignments effectively by selectively attending on the significant and representative alignments and meanwhile casting aside the interferences of non-meaningful alignments. We demonstrate the superiority of the proposed method with achieving state-of-the-art performances on the Flickr30K and MSCOCO datasets, and the good interpretability of SGR and SAF with extensive qualitative experiments and analyses.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mingqian Tang
2. Xiangteng He
3. Yiliang Lv
4. Liyue Shen
5. Jing Liu
6. Peng Wu
7. Serena Yeung
8. Matthew P. Lungren

In Computer Vision and Pattern Recognition

1. Mingqian Tang
2. Xiangteng He
3. Yiliang Lv
4. Liyue Shen
5. Jing Liu
6. Peng Wu
7. Serena Yeung
8. Matthew P. Lungren

In Multimodal Learning

1. Xiangteng He
2. Liyue Shen
3. Serena Yeung
4. Matthew P. Lungren

In Representation & Self-Supervised Learning

1. Xiangteng He
2. Liyue Shen
3. Serena Yeung
4. Matthew P. Lungren

Citations in the corpus

• HANet: Hierarchical Alignment Networks for Video-Text Retrieval (2021, positive): “Since there are multiple components in the video and text at the individual level, following [4, 6, 20], we use the stack attention mechanism to align multiple components and compute the overall similarity score.”
• GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition (2021, background): “While the idea of learning local representations has been explored in several other contexts for natural images [7, 27, 25, 4], including image-text retrieval and text-toimage generation, these works typically require pre-trained object detection models to extract localized image features, which are”