Multimodal Convolutional Neural Networks for Matching Image and Sentence

By Lin Ma, Zhengdong Lu, Lifeng Shang, Hang Li. 2015

352 citations recorded by the source corpus.

In this paper, we propose multimodal convolutional neural networks (m-CNNs) for matching image and sentence. Our m-CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content and one matching CNN modeling the joint representation of image and sentence. The matching CNN composes different semantic fragments from words and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results demonstrate that the proposed m-CNNs can effectively capture the information necessary for image and sentence matching. More specifically, our proposed m-CNNs significantly outperform the state-of-the-art approaches for bidirectional image and sentence retrieval on the Flickr8K and Flickr30K datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “These results are comparable to previous approach of (Ma et al., 2015) that did not do transfer but directly learned the sentence encoding on the image-caption retrieval task.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, neutral): “5 m-CNN (Ma et al. 2015) 42.”
Order-Embeddings of Images and Language (2015, methodology): “An alternative is to learn an unconstrained binary relation, either with a neural language model conditioned on the image (Vinyals et al., 2015; Mao et al., 2015) or using a multimodal CNN (Ma et al., 2015).”
MUTAN: Multimodal Tucker Fusion for Visual Question Answering (2017, background): “The main task in multimodal visual and textual analysis aims at learning an alignment between feature spaces [29, 23, 18].”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, background): “Recently, the multimodal learning between image and language (Ma et al. 2015; Nakamura et al. 2013; Xu et al. 2015b) has become an increasingly popular research area of artificial intelligence (AI).”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “We report the performance of mDAN in bidirectional image and sentence retrieval using the same metrics as previous work [34, 19, 20, 30].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, neutral): “9 mCNN(ensemble) [63] 33.”
Position Focused Attention Network for Image-Text Matching (2019, background): “Therefore, many researchers have dedicated their efforts to study the relationship between the visual and the textual contents [Lee et al., 2018; Gu et al., 2018; Eisenschtat et al., 2017; Ma et al., 2015; Wang et al., 2018; Klein et al., 2015].”
Enhancing Video Summarization via Vision-Language Embedding (2017, methodology): “Such models have been used for cross-modal retrieval [16, 19, 26, 29, 27, 37, 39], image caption generation [12, 13, 27, 38, 44], and visual storytelling [11, 50].”
Emergent Translation in Multi-Agent Communication (2017, background): “) or its variants (Hodosh et al., 2013; Andrew et al., 2013; Chandar et al., 2016). Other works minimize pairwise ranking loss to learn multimodal embeddings (Socher et al., 2014; Kiros et al., 2014; Ma et al., 2015; Vendrov et al., 2015; Kiela et al., 2017). Most recently, others extended this work”
Multilevel Language and Vision Integration for Text-to-Clip Retrieval (2018, background): “In contrast, the early fusion approach combines the features from each modality at an earlier stage (Ma et al. 2015; Wang et al. 2018; Yu et al. 2017) and predicts similarity scores directly based on the fused feature representation.”
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM (2016, methodology): “To systematically validate the effectiveness, we experiment with five variants of sm-LSTMs: (1) sm-LSTM-mean does not use the attention scheme to obtain weighted sum representations for selected instances but instead directly uses mean vectors, (2) sm-LSTM-att only uses the attention scheme but does”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “, images and sentences) typically requires strong representations of the individual modalities [213].”
Deep Cross-Modal Projection Learning for Image-Text Matching (2018, background): “Exploring the relationship between image and natural language has recently attracted great interest among researchers, due to its great importance in various applications, such as bi-directional image and text retrieval [44, 22], natural language object retrieval [10], image captioning [43, 35], and”
Adversarial Cross-Modal Retrieval (2017, methodology): “For the evaluation on the MSCOCO dataset, we quote the mAP results obtained by the baselines [12, 14, 18, 19] as referred to [33].”
Visually Grounded Neural Syntax Acquisition (2019, neutral): “…widely studied approach, and has achieved remarkable results on image-caption retrieval (Kiros et al., 2014; Faghri et al., 2018; Shi et al., 2018a), image caption generation (Kiros et al., 2014; Karpathy and Fei-Fei, 2015; Ma et al., 2015), and visual question answering (Malinowski et al., 2015).”