Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM

By Yan Huang, Liang Wang. 2016

233 citations recorded by the source corpus.

Effective image and sentence matching depends on how to well measure their global visual-semantic similarity. Based on the observation that such a global similarity arises from a complex aggregation of multiple local similarities between pairwise instances of image (objects) and sentence (words), we propose a selective multimodal Long Short-Term Memory network (sm-LSTM) for instance-aware image and sentence matching. The sm-LSTM includes a multimodal context-modulated attention scheme at each timestep that can selectively attend to a pair of instances of image and sentence, by predicting pairwise instance-aware saliency maps for image and sentence. For selected pairwise instances, their representations are obtained based on the predicted saliency maps, and then compared to measure their local similarity. By similarly measuring multiple local similarities within a few timesteps, the sm-LSTM sequentially aggregates them with hidden states to obtain a final matching score as the desired global similarity. Extensive experiments show that our model can well match image and sentence with complex content, and achieve the state-of-the-art results on two public benchmark datasets.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Stacked Cross Attention for Image-Text Matching (2018, background): “[16] developed a context-modulated attention scheme to selectively attend to a pair of instances appearing in both the image and sentence.”
VSE++: Improved Visual-Semantic Embeddings (2017, background): “In Huang et al. (2016), the authors use a multi-modal context-modulated attention mechanism to compute the matching score between an image and a caption.”
Visual Semantic Reasoning for Image-Text Matching (2019, neutral): “…Methods Caption Retrieval Image Retrieval R@ 1 R@ 5 R@ 10 R@ 1 R@ 5 R@ 10 (R-CNN, AlexNet) DVSA CVPR (cid:48) 15 [15] 38 [20] 39.4 67.9 80.9 25.1 59.8 76.6 OEM ICLR (cid:48) 16 [35] 46.7 - 88.9 37.9 - 85.9 VQA ECCV (cid:48) 16 [29] 50.5 80.1 89.7 37.0 70.9 82.9 SMlstm CVPR (cid:48) 17 [13] Table 1.”
Learning Visually-Grounded Semantics from Contrastive Adversarial Samples (2018, methodology): “(Huang et al., 2017) utilizes a multi-modal context-modulated attention mechanism to compute the similarity between an image and a caption.”
VSE++: Improving Visual-Semantic Embeddings with Hard Negatives (2017, background): “In Huang et al. (2016), the authors use a multi-modal context-modulated attention mechanism to compute the similarity between an image and a caption.”
Multi-Modality Cross Attention Network for Image and Sentence Matching (2020, methodology): “Following [12], we roughly divide the related methods into two categories including one-to-one matching and many-to-many matching.”
DeepStore: In-Storage Acceleration for Intelligent Queries (2019, background): “We believe these identified intelligent query applications show the generality of emerging intelligent query systems [23, 30, 43, 46, 51, 57, 69, 83, 84, 102].”
IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval (2020, methodology): “Recently, several works have been devoted to exploring the latent fine-grained vision-language correspondence for cross-modal imagetext [1, 11, 20, 8, 18, 13].”
Graph Structured Network for Image-Text Matching (2020, background): “[8] propose to recurrently select corresponding region-word pairs.”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, background): “Prior work [16, 54] observed a split between one-to-one and many-to-many matching in visual-semantic embedding spaces characterized by the use of one or several embedding representations per image or caption.”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, methodology): “Shown in Table 1, Unified VSE outperforms other baselines with various model architecture and training techniques [11, 49, 28, 40, 15].”
Dual-Path Convolutional Image-Text Embedding (2017, methodology): “In the field of image-text matching, most recent methods directly use fixed CNN features [23, 14, 20, 16, 36, 26, 10, 27, 37] as input which are extracted from models pre-trained on ImageNet for object recognition.”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, methodology): “In this paper, in place of RNNs which are more commonly seen in image-text matching, we explore the usage of CNNs for text representation learning.”
Learning Semantic Concepts and Order for Image and Sentence Matching (2017, methodology): “Most existing methods [12, 16, 23] jointly represent all the concepts by extracting a global CNN [30] feature vector, in which the concepts are tangled with each other.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, background): “…the fact that vision and language research has been gaining popularity in image domains (e.g., image cap-tioning [24, 55, 60, 34], image-text matching [13, 28, 31, 21, 14], and text-to-image generation [49, 14, 53]), only very few researchers focus on the joint domain of vision and language in 3D.”
Graph Optimal Transport for Cross-Domain Alignment (2020, methodology): “In our experiment, K = { 1 , 5 , 10 , and Rsum (Huang et al., 2017) (summation over all R@ K ) is used to evaluate the overall performance.”
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders (2020, unknown): “some point in the pipeline. Concerning image processing, the standard approach consists in using Convolutional Neural Networks (CNNs), usually pretrained on image classification tasks. In particular, [9, 15, 20, 29, 44] used VGGs, [10, 11, 16, 31] used ResNets. The problem with these kinds of CNNs i”