See No Evil, Say No Evil: Description Generation from Densely Labeled Images

By Mark Yatskar, Michel Galley, Lucy Vanderwende, Luke S. Zettlemoyer. 2014

59 citations recorded by the source corpus.

This paper studies generation of descriptive sentences from densely annotated images. Previous work studied generation from automatically detected visual information but produced a limited class of sentences, hindered by currently unreliable recognition of activities and attributes. Instead, we collect human annotations of objects, parts, attributes and activities in images. These annotations allow us to build a significantly more comprehensive model of language generation and allow us to study what visual information is required to generate human-like descriptions. Experiments demonstrate high quality output and that activity annotations and relative spatial location of objects contribute most to producing high quality sentences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Parsing Algebraic Word Problems into Equations (2015, background): “More closely related is work on language grounding, whose goal is the interpretation of a sentence in the context of a world representation (Branavan et al., 2009; Liang et al., 2009; Chen et al., 2010; Bordes et al., 2010; Feng and Lapata, 2010; Hajishirzi et al., 2011; Matuszek et al., 2012; Hajis”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
CIDEr: Consensus-based image description evaluation (2014, methodology): “on separate one to five [29, 37, 44, 11] or pairwise scales [45].”
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels (2015, background): “r attributes. A recurring challenge when annotating images is their complexity, i.e., a single image can contain an enormous amount of information. For most tasks, exhaustively annotating all content [54,56] in an image is infeasible. If exhaustive labeling is performed, it is typically done for a s”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…followed by a step mapping these outputs to linguistic structures (‘sentence plans’ of the sort discussed in Section 2 and 3), such as trees or templates (e.g. Kulkarni et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Elliott & De Vries, 2015; Yatskar et al., 2014; Kuznetsova et al., 2014).”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, background): “In terms of the representations used, existing models have conceptualized images in a number of different ways, relying on spatial relationships (Farhadi et al., 2010), corpus-based relationships (Yang et al., 2011), or spatial and visual attributes (Kulkarni et al., 2011).”
Commonly Uncommon: Semantic Sparsity in Situation Recognition (2016, background): “For the structured case, image captioning models [45, 22, 7, 11, 33, 20, 35, 31] have been observed to suffer from a lack of diversity and generalization [42].”
A Survey of Current Datasets for Vision and Language Research (2015, methodology): “One dataset useful to this end provides image annotation for content selection: • Microsoft Research Dense Visual Annotation Corpus (Yatskar et al., 2014) provides a set of 500 images from the Flickr 8K dataset (Rashtchian et al., 2010) that are densely labeled with 100,000 textual labels, with…”
Situation Recognition: Visual Semantic Role Labeling for Image Understanding (2016, background): “In particular, sentence generation is closely related and has received significant attention [50, 26, 2, 11, 46, 34, 24, 36, 32].”
Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes (2016, background): “In Section 4, we describe each of the modules in detail, making concrete their respective features, and other details.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2017, unknown): “Instead of using a fixed template, some approaches that use a generative grammar have also been developed [35] [48].”