Learning Semantic Concepts and Order for Image and Sentence Matching

By Yan Huang, Qi Wu, Liang Wang. 2017

325 citations recorded by the source corpus.

Image and sentence matching has made great progress recently, but it remains challenging due to the large visual-semantic discrepancy. This mainly arises from that the representation of pixel-level image usually lacks of high-level semantic information as in its matched sentence. In this work, we propose a semantic-enhanced image and sentence matching model, which can improve the image representation by learning semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its semantic concepts, including objects, properties, actions, etc. Then, considering that different orders of semantic concepts lead to diverse semantic meanings, we use a context-gated sentence generation scheme for semantic order learning. It simultaneously uses the image global context containing concept relations as reference and the groundtruth semantic order in the matched sentence as supervision. After obtaining the improved image representation, we learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate the effectiveness of our learned semantic concepts and order, by achieving the state-of-the-art results on two public benchmark datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Stacked Cross Attention for Image-Text Matching (2018, background): “9 SCO [17] (current state-of-the-art) 55.”
Learning the Best Pooling Strategy for Visual Semantic Embedding (2020, background): “, max pooling), can surpass prior state-of-the-art VSE methods with complex aggregators [17, 27, 43, 45, 46].”
Visual Semantic Reasoning for Image-Text Matching (2019, methodology): “Following the common protocol [5, 14, 23], the results are obtained by averaging over 5 folds of 1K test”
CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval (2019, background): “[10] proposed a model to learn semantic concepts and order for better image and sentence matching.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, background): “There has been a plethora of successful work connecting images to natural language descriptions across tasks such as image captioning [25, 24, 55, 60], text-to-image retrieval [56, 22], and visual grounding [20, 35, 67].”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, result): “Compared with recent works, VSE++ [10], GXN [17] and CNP [29], our result is also competitive.”
Heterogeneous Attention Network for Effective and Efficient Cross-modal Retrieval (2021, methodology): “As shown in Table 1, the methods using global features are not as competitive as the methods based on local features such as SCO [15], SCAN [20] and PVSE [36] and our HAN.”
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval (2021, methodology): “When compared with methods without cross-attention (VSE++ (Faghri et al., 2017) and SCO (Huang et al., 2018)), LightningDOT achieves nearly Model COCO Full (123K Images) Flickr30K Full (31K Images) Text Retrieval Image Retrieval Text Retrieval Image Retrieval R@5 R@10 R@20 R@5 R@10 R@20 AR R@5 R@10…”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, unknown): “ of the bottom-up perception module, which should carefully process the resulting image regions to obtain an expressive representation of the overall scene. Concerning sentence processing, many works [10, 16, 19, 22, 24] employ GRU or LSTM recurrent networks to process natural language. Recently, th”
Semantic Communications: Principles and Challenges (2021, unknown): “Huang [52] proposed the image and sentence matching approach by utilizing the image global context to learn the semantic concept, such as objects, properties, and actions.”
Graph Optimal Transport for Cross-Domain Alignment (2020, unknown): “5 SCO (ResNet) (Huang et al., 2018) 55.”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, methodology): “Appendix A contains examples of words that most strongly Model DE → EN BLEU EN → DE BLEU MIXER (Ranzato et al., 2016) 21.8 Beam Search Optimization (Wiseman and Rush, 2016) 25.5 Actor-Critic + Log Likelihood (Bahdanau et al., 2017) 28.5 Neural Phrase-based Machine Translation (Huang et al., 2018a) 2”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, background): “For fair comparison, we do not include [10] and [16] that finetunes the image encoder or adds extra training data.”
Similarity Reasoning and Filtration for Image-Text Matching (2021, methodology): “To improve the discriminative ability of the unified embeddings, many strategies such as semantic concept learning (Huang et al. 2018; Shi et al. 2019) and region relationship reasoning (Li et al.”
GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition (2021, background): “Recent studies have achieved progress by utilizing localized representations through stacked attention [25], semantic ordering [15] and graph convolutional neural networks [27, 7].”
Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning (2020, background): “unseen dataset. A new binary selection task to demonstrate the ability to distinguish ﬁne-grained semantic differences is proposed as well. 2. Related Works Image-TextMatching. Most of previous works [6,8,9,17, 21] for image-text matching encode images and sentences as ﬁx-dimensional vectors in a co”