Learning a Recurrent Visual Representation for Image Caption Generation
By Xinlei Chen, C. Lawrence Zitnick. 2014
203 citations recorded by the source corpus.
In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. We propose learning this mapping using a recurrent neural network. Unlike previous approaches that map both sentences and images to a common embedding, we enable the generation of novel sentences given an image. Using the same model, we can also reconstruct the visual features associated with an image given its visual description. We use a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are preferred by humans over $19.8\%$ of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.
Subfields: Computer Vision and Pattern Recognition, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Yi Yang
- Jitendra Malik
- Chunhua Shen
- Ming-Hsuan Yang
In Natural Language Generation
- Marcus Rohrbach
- Trevor Darrell
- Kate Saenko
- Devi Parikh
- Anna Rohrbach
- Li Fei-Fei
- Qi Wu
- Subhashini Venugopalan
Citations in the corpus
- ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Early works utilize the features extracted from a pre-trained classification network [6, 9, 13, 42], while later works [4, 19, 47] exploit the more expressive features of an object detection network [31].”
- Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
- Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “More closely related to our approach is the work of Srivas-tava et al. [43] who use a Deep Boltzmann Machine to learn a joint distribution over a images and tags.”
- CIDEr: Consensus-based image description evaluation (2014, methodology): “This dataset is gaining traction with recent image description approaches [19, 6, 10, 43, 27, 21].”
- Stacked Attention Networks for Image Question Answering (2015, background): “The essential structure of a LSTM unit is a memory cell ct which reserves the state of a sequence.”
- Ask Me Anything: Dynamic Memory Networks for Natural Language Processing (2015, background): “…have recently been proposed for algorithmic problems (Joulin & Mikolov, 2015; Kaiser & Sutskever, 2015), caption generation for images (Malinowski & Fritz, 2014; Chen & Zitnick, 2014), visual question answering (Yang et al., 2015) or other NLP problems and datasets (Hermann et al., 2015).”
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “We also validate our method on the test set of MS COCO by their evaluation server (Chen et al. (2015)).”
- MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “Having access to such data enabled the community to shift from hand-crafted language templates typically used for image description [19] or retrieval-based approaches [11, 26, 45] to deep neural models [6, 13, 15, 42] that achieve impressive captioning results.”
- Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, background): “In particular, many studies have made rapid progress on the task of image captioning [26, 15, 14, 40, 6, 8, 4, 19, 16, 42].”
- A dataset for Movie Description (2015, background): “…on image description show impressive results by learning the relations between images and sentences and generating novel sentences [41, 19, 48, 56, 35, 31, 65, 13], the video description works typically rely on retrieval or templates [16, 63, 26, 27, 37, 39, 62] and frequently use a separate…”
- Dynamic Memory Networks for Visual and Textual Question Answering (2016, methodology): “Shortly thereafter, recurrent neural networks were used to generate often novel sentences based on images (Karpathy & Fei-Fei, 2015; Chen & Zitnick, 2014; Fang et al., 2015; Xu et al., 2015).”
- Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, methodology): “In computer vision, along with improvements in deep learning based visual recognition, there has been an explosion of recent interest in methods to automatically generate natural language descriptions for images [5, 9, 15, 32, 16, 20] or videos [31, 8].”
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, methodology): “Several recent approaches to Image Caption-ing [32, 22, 48, 9, 5, 25, 12] rely on a combination of RNN language model conditioned on image information.”
- Sequence to Sequence -- Video to Text (2015, background): “Describing visual content with natural language text has recently received increased interest, especially describing images with a single sentence [8, 5, 16, 18, 20, 23, 29, 40].”
- Learning to Answer Questions from Image Using Convolutional Neural Network (2015, methodology): “By chop-ping out the top softmax layer and the last ReLU layer, the obtained image representation are non-linearly mapped to a new representation, according to Eq.”
- Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources (2015, background): “Currently the most successful approach to image captioning [6, 9, 14, 21, 28] is to attach a CNN to an RNN to learn the mapping from images to sentences directly.”
- Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “to-text translation using RNNs was seen in the work of image captioning [30, 22, 19, 49, 8], which can be treated as a special case of video captioning when each video has a single frame and no temporal structure.”