Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

By Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Alan L. Yuille. 2014

1,283 citations recorded by the source corpus.

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, neutral): “, 2015) and image caption generation (Mao et al., 2015).”
An efficient framework for learning sentence representations (2018, methodology): “Image Search Model R@1 R@5 R@10 Med r R@1 R@5 R@10 Med The purely supervised models are respectively from (Karpathy & Fei-Fei, 2015), (Klein et al., 2015), (Mao et al., 2014) and (Vendrov et al., 2015).”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “Examples include machine translation (Sutskever et al., 2014), image captioning (Mao et al., 2015) and dialogue systems (Serban et al., 2015).”
Skip-Thought Vectors (2015, neutral): “6 4 m-RNN [34] 41.”
A Diversity-Promoting Objective Function for Neural Conversation Models (2015, background): “2 On a somewhat different task, Mao et al. (2015, Section 6) utilize a mutual information objective in image caption retrieval.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Visual-Semantic Embedding: Joint visual-semantic models have shown excellent performance on several multimedia tasks, e.g., cross-modal retrieval [18, 31, 38, 55], image captioning [23, 36], image classification [10, 12, 20] video summarization [4, 41].”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “[Mao et al., 2014; Kiros et al., 2014; Vinyals et al., 2014; Chen and Zitnick, 2014].”
Order Matters: Sequence to sequence for sets (2015, neutral): “, 2015a), to image captioning (Vinyals et al., 2015c; Mao et al., 2015; Donahue et al., 2015), speech recognition (Chan et al.”
Generative Adversarial Text to Image Synthesis (2016, background): “However, in the past year, there has been a breakthrough in using recurrent neural network decoders to generate text descriptions conditioned on images (Vinyals et al., 2015; Mao et al., 2015; Karpathy & Li, 2015; Donahue et al., 2015).”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “tion and Recall@K, the number of images or captions for which a correct caption or image is retrieved within the top Kresults. Our model consistently outperforms the strong baselines from recent work [27], [28], [29], [30], [31] as can be seen in Table 4. Here, we note that the VGGNet model in [31] ”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, background): “Unlike Kiros et al. (2014a) and Mao et al. (2014) whose models see the image at each time step of the output word sequence, Vinyals et al. (2014) only show the image to the RNN at the beginning.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “…USA C. M. Cervantes University of Illinois at Urbana Champaign, Urbana, IL, USA J. C. Caicedo Fundación Universidad Konrad Lorenz, Bogotá, Colombia J. Hockenmaier University of Illinois at Urbana Champaign, Urbana, IL, USA S. Lazebnik University of Illinois at Urbana Champaign, Urbana, IL, USA”
Stacked Attention Networks for Image Question Answering (2015, background): “The essential structure of a LSTM unit is a memory cell ct which reserves the state of a sequence.”
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks (2015, background): “The task can be formulated as a mapping of an image onto a sequence of words describing its content in some natural language, and most proposed approaches employ some form of recurrent network structure with simple decoding schemes [5, 6, 14, 15, 16].”
Neural Approaches to Conversational AI (2018, neutral): “g issues. 5.2.1 Response blandness Utterances generated by neural response generation systems are often bland and deﬂective. While this problem has been noted in other tasks such as image captioning (Mao et al., 2015), the problem is particularly acute in E2E response generation, as commonly used mo”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, background): “There is a surge of interest in image-text tasks such as image captioning [10, 22, 23, 25, 31, 43, 46, 50] and visual question answering [2, 12, 52].”