BabyTalk: Understanding and Generating Simple Image Descriptions

By Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, Tamara L. Berg. 2013

1,067 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “Older work made use of fixed visual representations and translated them into textual descriptions [7, 18].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Template-based methods involve ﬁlling in sentence templates, such as triplets, based on the results of object detections and spatial relationships [21, 22, 23, 24, 25].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “Inspired by prior work [7], as a modeling assumption we observe that the subject of most sentence descriptions are attributes of objects and their context in a scene.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Farhadi et al. (2010) and Kulkarni et al. (2011), on the other hand, use a more automatic method to parse images.”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “Following common practice in natural language generation, Yang et al. (2011) and Kulkarni et al. (2011) evaluated captions on a graded scale for relevance Hodosh, Young & Hockenmaier”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Early methods [19,31,37,65,79] focus on filling templates with extracted objects, attributes, and relationships.”
Attention on Attention for Image Captioning (2019, background): “It requires not only to recognize salient objects in an image, understand their interactions, but also to verbalize them using natural language, which makes itself very challenging [25, 45, 28, 12].”
VQA: Visual Question Answering (2015, methodology): “Related to VQA are the tasks of image tagging [8, 23], image captioning [24, 14, 34, 7, 13, 45, 9, 19, 32, 21] and video captioning [39, 17], where words or sentences are generated to describe visual content.”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “Several early works [53], [54], [55], [56] on image captioning combine object and scene recognition with template or tree based approaches to generate captions.”
Show and tell: A neural image caption generator (2014, background): “In blue we show the recurrent connections – the output m at time t − 1 is fed back to the memory at time t via the three gates; the cell value is fed back via the forget gate; the predicted word at time t− 1 is fed back in addition to the memory output m at time t into the Softmax for word…”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “The ﬁrst involved generating caption templates which were ﬁlled in based on the results of object detections and attribute discovery (Kulkarni et al. (2013), Li et al. (2011), Yang et al. (2011), Mitchell et al. (2012), Elliott & Keller (2013)).”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “…Donahue et al. 2015; Fang et al. 2015; Farhadi et al. 2010; Hodosh et al. 2013; Karpathy and Fei-Fei 2015; Kiros et al. 2014; Klein et al. 2014; Kulkarni et al. 2011; Lebret et al. 2015; Mao et al. 2015b; Ordonez et al. 2011; Vinyals et al. 2015; Yao et al. 2010) and visual question answering…”
SPICE: Semantic Propositional Image Caption Evaluation (2016, background): “On the PASCAL 1K dataset, Bleu-1 was found to exhibit weak or no correlation (Pearson’s r of -0.17 and 0.05) [7].”
nocaps: novel object captioning at scale (2019, methodology): “Two concurrent works, Neural Baby Talk [25] and the Decoupled Novel Object Captioner [43], take inspiration from Baby Talk [19] and propose neural approaches to generate slotted caption templates, which are then filled using visual concepts identified by modern state-of-the-art object detectors.”
ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, methodology): “∗ Available online at http://referitgame.com”
From captions to visual concepts and back (2014, methodology): “Captions using our approach (black), Midge [30] (blue) and Baby Talk [22] (red) are shown. correlate with human judgment [9].”
MovieQA: Understanding Stories in Movies through Question-Answering (2015, background): “Having access to such data enabled the community to shift from hand-crafted language templates typically used for image description [19] or retrieval-based approaches [11, 26, 45] to deep neural models [6, 13, 15, 42] that achieve impressive captioning results.”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”