Collective Generation of Natural Image Descriptions

By Polina Kuznetsova, Vicente Ordonez, Alexander C. Berg, Tamara L. Berg, Yejin Choi. 2012

393 citations recorded by the source corpus.

Subfields: Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, background): “These approaches aim to harness existing image-caption databases by extracting components of related captions and composing them together to generate novel descriptions [26, 27].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Some approaches focus on the direction of describing the contents of images, formulated either as a task of mapping images to a fixed set of sentences written by people [5, 6], or as a task of automatically generating novel captions [7, 8, 9, 10, 11, 12].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Conversely, when given a query image, we would like to find a description that goes beyond a single label by providing a correct sentence describing it, a task that has recently garnered a lot of attention (Farhadi et al., 2010; Ordonez et al., 2011; Kuznetsova et al., 2012).”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “ically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “Crowdsourcing platforms such as Amazon Mechanical Turk offer new possibilities for evaluation because they enable us to collect a large number of human judgments rapidly and inexpensively, and a number of researchers have evaluated caption generation systems on Mechanical Turk (Ordonez et al., 2011;”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Long-term recurrent convolutional networks for visual recognition and description (2014, neutral): “and scene recognition with template or tree based approaches to generate captions. Such sentences are typically simple and are easily distinguished from more ﬂuent human generated descriptions. [46], [57] address this by composing new sentences from existing caption fragments which, though more huma”
Show and tell: A neural image caption generator (2014, background): “…at time t via the three gates; the cell value is fed back via the forget gate; the predicted word at time t− 1 is fed back in addition to the memory output m at time t into the Softmax for word prediction. read its input (input gate i) and whether to output the new cell value (output gate o).”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “The second approach was based on ﬁrst retrieving similar captioned images from a large database then modifying these retrieved captions to ﬁt the query (Kuznetsova et al., 2012; 2014).”
ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, background): “…of recent work trying to estimate higher level semantics, including exciting efforts to automatically produce natural language descriptions of images and video (Farhadi et al., 2010; Kulkarni et al., 2011; Yang et al., 2011; Ordonez et al., 2011; Kuznetsova et al., 2012; Feng and Lapata, 2013).”
From captions to visual concepts and back (2014, neutral): “Previous work utilizes syntactic and semantic constraints in the generation process [28, 45, 23, 21, 20, 44], and we compare against prior state of the art in this line of work.”
Movie Description (2016, background): “Prior work on image description includes Farhadi et al. (2010); Kulkarni et al. (2011); Kuznetsova et al. (2012); Li et al. (2011); Kuznetsova et al. (2014); Mitchell et al. (2012); Socher et al. (2014).”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, methodology): “The second type of approach explored methods to make use of existing text either directly associated with an image [11, 1] or retrieved from visually similar images [27, 19, 24].”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, background): “Numerous papers have explored the area of generating novel image descriptions [7, 36, 19, 37, 28, 11, 20, 17].”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “This extraction strategy, originally proposed by Kuznetsova et al. (2012), attempts to make the best use of linguistic regularities with respect to objects, actions, and scenes, making it possible to obtain richer textual descriptions than what current state-of-the-art vision techniques can provide…”
Comparing Automatic Evaluation Measures for Image Description (2014, methodology): “An analysis of the distribution of TER scores in Figure 2(a) shows that differences in candidate and reference length are prevalent in the image description task.”
Image Captioning with Semantic Attention (2016, methodology): “The topdown paradigm [4, 35, 26, 16, 8, 37, 25] starts from a “gist” of an image and converts it into words, while the bottom-up one [12, 19, 23, 9, 20, 11, 22] first comes up with words describing various aspects of an image and then combines them.”
Image Description using Visual Dependency Representations (2013, background): “, 2010), generated descriptions by retrieving the descriptions of similar images (Ordonez et al., 2011; Kuznetsova et al., 2012), relied on an external corpus to predict the relationships between objects (Yang et al.”