Phrase-based Image Captioning

By Rémi Lebret, Pedro H. O. Pinheiro, Ronan Collobert. 2015

124 citations recorded by the source corpus.

Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representation (generated from a previously trained Convolutional Neural Network) and phrases that are used to described them. The system is then able to infer phrases from a given image sample. Based on caption syntax statistics, we propose a simple language model that can produce relevant descriptions for a given test image using the phrases inferred. Our approach, which is considerably simpler than state-of-the-art models, achieves comparable results in two popular datasets for the task: Flickr30k and the recently proposed Microsoft COCO.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
From captions to visual concepts and back (2014, methodology): “Also related are several contemporaneous papers [29, 45, 6, 18, 9, 46, 25].”
Exploring Models and Data for Image Question Answering (2015, methodology): “…2015; Kiros et al., 2015; Karpathy et al., 2013; Mao et al., 2014; Donahue et al., 2014; Chen & Zitnick, 2014; Fang et al., 2015; Xu et al., 2015; Lebret et al., 2015; Klein et al., 2015) have developed powerful methods of jointly learning from image and text inputs to form higher level…”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, methodology): “In computer vision, along with improvements in deep learning based visual recognition, there has been an explosion of recent interest in methods to automatically generate natural language descriptions for images [5, 9, 15, 32, 16, 20] or videos [31, 8].”
Neural Summarization by Extracting Sentences and Words (2016, background): “It would also be interesting to apply the neural models presented here in a phrase-based setting similar to Lebret et al. (2015).”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, background): “We hope to provide context for the recent advances in this area [18], [26], [35], [16], [19], [6], [8], [2], [23], [22], [21], [36].”
Weakly-Supervised Alignment of Video with Text (2015, methodology): “We discuss several rounding procedures to obtain a solution to the initial problem.”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “The general RNN-based ranking and generation approach is also followed by Lebret et al. (2015).”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “[124] use a CNN-based image representation to infer phrases that describe it.”
Minimal gated unit for recurrent neural networks (2016, background): “…captioning, i.e., Submitted to ICML 2016. summarizing the semantic meaning of an image into a sentence (Xu et al., 2015; Karpathy & Fei-Fei, 2015; Lebret et al., 2015), recognizing actions in videos (Donahue et al., 2015; Srivastava et al., 2015), or short-term precipitation prediction (Shi et…”
Image Question Answering: A Visual Semantic Embedding Model and a New Dataset (2015, background): “Recently, image caption generation (Vinyals et al., 2014; Kiros et al., 2014; Karpathy et al., 2014; Mao et al., 2014; Donahue et al., 2014; Chen & Zitnick, 2014; Fang et al., 2014; Xu et al., 2015; Lebret et al., 2015) has shown us feasible ways of jointly learning image and text to form higher lev”
Evaluating the Energy Efficiency of Deep Convolutional Neural Networks on CPUs and GPUs (2016, background): “Examples of emerging applications based on NNs include: auto tagging [33], the estimation of a person’s pose [34], and the generation of a descriptive sentence from an image [35].”
Syntax-Aware Action Targeting for Video Captioning (2020, background): “For example, Lebret et al. [27] analyzed phrases in captions and suggested learning a common space for image and phrase representations.”
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language (2016, background): “Recently, researchers have explored to leverage sequence learning to generate sentences [4, 8, 11, 13, 19, 22, 36, 37].”
Bootstrapping Generators from Noisy Data (2018, background): “It would also make sense for the model to decode hierarchically, taking sequences of words and sentences into account (Zhang and Lapata, 2014; Lebret et al., 2015).”
Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition (2017, methodology): “In [27], Lebret et al . predict phrases from an image, and combine them with a simple language model to generate the description.”
Recurrent Highway Networks with Language CNN for Image Captioning (2016, methodology): “[21] use phrases instead of words as basic sentence building blocks and concatenated candidate phrases by a constrained language model.”