Visual Semantic Reasoning for Image-Text Matching

By Kunpeng Li, Yulun Zhang, K. Li, Yuanyuan Li, Yun Raymond Fu. 2019

614 citations recorded by the source corpus.

Image-text matching has been a hot research topic bridging the vision and language areas. It remains challenging because the current representation of image usually lacks global semantic concepts as in its corresponding text caption. To address this issue, we propose a simple and interpretable reasoning model to generate visual representation that captures key objects and semantic concepts of a scene. Specifically, we first build up connections between image regions and perform reasoning with Graph Convolutional Networks to generate features with semantic relationships. Then, we propose to use the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually generate the representation for the whole scene. Experiments validate that our method achieves a new state-of-the-art for the image-text matching on MS-COCO and Flickr30K datasets. It outperforms the current best method by 6.8% relatively for image retrieval and 4.8% relatively for caption retrieval on MS-COCO (Recall@1 using 1K test set). On Flickr30K, our model improves image retrieval by 12.6% relatively and caption retrieval by 5.8% relatively (Recall@1).

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Combined Scaling for Zero-shot Transfer Learning (2021, neutral): “…Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al., 2020d; Huang et al., 2020; Chen et al., 2021).”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, methodology): “R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Zero-shot (Faghri et al., 2018), VSRN (Li et al., 2019), DE I2T (Parekh et al., 2021), and DE T2T+I2T (Parekh et al., 2021). image R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 VSE++ ERNIE-ViL (Yu et al., 2020), VILLA (Gan et al., 2020) and O”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, methodology): “Learning representations for both vision and language jointly should be even more effective—indeed, much progress has been made on such cross-modal learning using image captioning data (Karpathy and Li, 2015; Harwath and Glass, 2017; Faghri et al., 2018; Li et al., 2019).”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, methodology): “As aforementioned in § 1, complex aggregators f have been investigated in the VSE literature [17, 27, 43, 45, 46], such as sequence-to-sequence encoder (Seq2Seq), graph convolution network (GCN), self-attention encoder (SelfAttn), etc.”
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval (2020, result): “And compared with VSRN* [15], our single model can achieve competitive results in both Flick30K and MS COCO.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “By analyzing this table, we gained the following observations: • Among intra-modal interaction pattern basedmethods, CAM- ERA* [26] surpasses VSRN* [16] by a large margin on two 8We reproduced CAMERA (https://acmmmcamera.wixsite.com/camera) for the following significance test. datasets.”
Combined Scaling for Open-Vocabulary Image Classiﬁcation (2022, neutral): “…Hodosh et al., 2013; Frome et al., 2013; Norouzi et al., 2013; Kiros et al., 2014; Socher et al., 2014; Akata et al., 2015b,a; Nam et al., 2017; Faghri et al., 2017; Li et al., 2019; Liu et al., 2019; Lu et al., 2019; Messina et al., 2020; Chen et al., 2020d; Huang et al., 2020; Chen et al., 2021).”
Exploring Visual Engagement Signals for Representation Learning (2021, background): “Learning image representations using natural language There is growing interest in learning joint visual-language representations [16, 72, 39, 49, 54, 5, 9].”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, methodology): “To determine whether low performance on Winoground is unique to transformer-based models, we include results for two sequence-based models, which are VSRN [45] and VSE++ [21].”
Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “For global alignments, some works (Faghri et al. 2017; Wang, Li, and Lazebnik 2016; Liu et al. 2017; Song and Soleymani 2019; Nam, Ha, and Kim 2017; Li et al. 2019) explored a joint space and calculated the inner product (e.g. cosine distance) for similarity computation.”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, methodology): “[25] employ a graphical convolutional network [22] for semantic reasoning of region proposals obtained from a Faster-RCNN [40] detector.”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “The graph-based reasoning has great applications in computer vision tasks such as action recognition [32, 36], scene graph generation [42], referring expression grounding [23, 35], and visual question answering [16, 24] etc.”
GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition (2021, background): “While the idea of learning local representations has been explored in several other contexts for natural images [7, 27, 25, 4], including image-text retrieval and text-toimage generation, these works typically require pre-trained object detection models to extract localized image features, which are”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, positive): “Concerning sentence processing, many works [10, 16, 19, 22, 24] employ GRU or LSTM recurrent networks to process natural language.”