Explain Images with Multimodal Recurrent Neural Networks

By Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan L. Yuille. 2014

405 citations recorded by the source corpus.

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel sentence descriptions to explain the content of images. It directly models the probability distribution of generating a word given previous words and the image. Image descriptions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on three benchmark datasets (IAPR TC-12, Flickr 8K, and Flickr 30K). Our model outperforms the state-of-the-art generative method. In addition, the m-RNN model can be applied to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, background): “The field has tackled a diverse set of tasks such as captioning [15, 13, 40, 39, 24], alignment [13, 17, 38], Q&A [22, 21], visual model learning from textual descriptions [9, 29], and semantic visual search with natural multisentence queries [19].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “These ideas were recently extended to multimodal recurrent networks with significant improvements [7].”
VQA: Visual Question Answering (2015, methodology): “In particular, research in image and video captioning that combines Computer Vision (CV), Natural Language Processing (NLP), and Knowledge Representation & Reasoning (KR) has dramatically increased in the past year [13, 7, 9, 32, 21, 19, 45].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “Based on the B-1 scores in Table 3, generation using LRCN performs comparably with m-RNN [25] in terms of the information conveyed in the description.”
Show and tell: A neural image caption generator (2014, background): “All LSTMs share the same parameters. image and each sentence word such that all LSTMs share the same parameters and the output mt−1 of the LSTM at time t − 1 is fed to the LSTM at time t (see Figure 3).”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We ﬁlter words to those that occur at least 5 times in the training set, which results in 2538, 7414, and 8791 words for Flickr8k, Flickr30K, and MSCOCO datasets respectively.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering (2016, background): “[12,8,11,26,51,29,37] and visual question answering (VQA) [6,35,36,14,41] have gained popularity in recent years as the computer vision research community is progressing beyond “bucketed” recognition and towards solving multi-modal”
CIDEr: Consensus-based image description evaluation (2014, methodology): “This dataset is gaining traction with recent image description approaches [19, 6, 10, 43, 27, 21].”
From captions to visual concepts and back (2014, methodology): “Finally, in contrast to [18, 25], we formulate the problem of generation as an optimization problem and search for the most likely sentence [37].”
Exploring Models and Data for Image Question Answering (2015, background): “Recently, researchers studying image caption generation (Vinyals et al., 2015; Kiros et al., 2015; Karpathy et al., 2013; Mao et al., 2014; Donahue et al., 2014; Chen & Zitnick, 2014; Fang et al., 2015; Xu et al., 2015; Lebret et al., 2015; Klein et al., 2015) have developed powerful methods of…”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, background): “In particular, many studies have made rapid progress on the task of image captioning [26, 15, 14, 40, 6, 8, 4, 19, 16, 42].”
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, methodology): “Following previous work [32, 21, 49, 8, 4], we use the region codes to condition an RNN language model [15, 33, 44].”
Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “Propelled by successes of deep learning, several groups released record breaking results in just the past year (Don-ahue et al., 2014; Mao et al., 2014; Karpathy et al., 2014; Fang et al., 2014; Kiros et al., 2014; Vinyals et al., 2014; Kuznetsova et al., 2014).”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, methodology): “We report three scores for Flickr 8K corresponding to the methodologies proposed by [32], [13] and [23] respectively, and for Flickr 30K [8] and [23].”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, background): “Numerous papers have explored the task of caption ranking [26], [33], [10], [17], [2], [25].”