Simple Image Description Generator via a Linear Phrase-Based Approach

By Rémi Lebret, Pedro H. O. Pinheiro, Ronan Collobert. 2014

36 citations recorded by the source corpus.

Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representation (generated from a previously trained Convolutional Neural Network) and phrases that are used to described them. The system is then able to infer phrases from a given image sample. Based on caption syntax statistics, we propose a simple language model that can produce relevant descriptions for a given test image using the phrases inferred. Our approach, which is considerably simpler than state-of-the-art models, achieves comparable results on the recently release Microsoft COCO dataset.

Subfields: Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “Recently there has been a surprising resurgence of interest in this area [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], due to the renewed interest in neural network learning techniques [31], [32] and increasingly large datasets [33], [34], [35], [7], [36], [37], [38].”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, background): “In particular, many studies have made rapid progress on the task of image captioning [26, 15, 14, 40, 6, 8, 4, 19, 16, 42].”
Exploring Nearest Neighbor Approaches for Image Captioning (2015, background): “Once a set of k NN images are found, the captions describing these images are combined into a set of candidate captions from which the ﬁnal caption is selected, Figure 1.”
Image Captioning with Semantic Attention (2016, background): “[11] and [22] are the latest attempts along this direction and they achieve close to the state-of-the-art performance on various image captioning benchmarks.”
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images (2015, background): “Very recent works of image captioning includes [39, 25, 24, 55, 10, 15, 7, 32, 37, 26, 57, 35].”
Towards Diverse and Natural Image Descriptions via a Conditional GAN (2017, methodology): “object categories, relationships, and attributes) using CRFs [2, 6, 12], SVMs [16], or CNNs [5,17], then generate descriptions thereon using simple methods, such as sentence templates [12, 16], or by retrieving relevant sentences from existing data [5, 6, 13, 15].”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, neutral): “In bottom-up approaches, [32, 34, 74, 76, 78, 80], visual concepts (e.”
Deep Reinforcement Learning-Based Image Captioning with Embedding Reward (2017, methodology): “Early approaches tackled this problem using bottom-up paradigm [10, 23, 27, 47, 24, 8, 26, 9], which first generated descriptive words of an image by object recognition and attribute prediction, and then combined them by language models.”