Baby Talk: Understanding and Generating Image Descriptions

By Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, Tamara L. Berg. 2011

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [11, 8, 15]), which is a model of the form p(y|x), where y is the manually provided caption and x is the image.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Farhadi et al. (2010) and Kulkarni et al. (2011), on the other hand, use a more automatic method to parse images.”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Other generative approaches have explored creating sentences by inference over image detections and text-based priors [22] or exploiting word cooccurrences using syntactic trees [29].”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “As mentioned in the Introduction, the most common imagelanguage understanding task in the literature is automatic image captioning (Chen and Zitnick 2015; Donahue et al. 2015; Fang et al. 2015; Farhadi et al. 2010; Hodosh et al. 2013; Karpathy and Fei-Fei 2015; Kiros et al. 2014; Klein et al. 2014; ”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “Each part is associated with an object or an attribute in the image (e.g. Kulkarni et al. (2011) uses a Conditional Random Field model and Farhadi et al. (2010) uses a Markov Random Field model).”
Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering (2015, methodology): “The neural attention mechanism has been widely used in different areas of computer vision and natural language processing, see for example the attention models in image captioning [19], video description generation [20], machine translation [21][22] and machine reading systems [23].”
Dynamic Memory Networks for Visual and Textual Question Answering (2016, background): “Related to visual question answering is the task of describing images with sentences (Kulkarni et al., 2011).”
Explain Images with Multimodal Recurrent Neural Networks (2014, methodology): “[20] uses a Conditional Random Field model and [6] uses a Markov Random Field model).”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, background): “The ﬁrst type of approach focused on detecting content elements such as objects, attributes, activities, or spatial relationships and then composing captions for images [18, 33, 26, 10] or videos [17] using linguistically inspired templates.”
Image Captioning with Semantic Attention (2016, background): “The topdown paradigm [4, 35, 26, 16, 8, 37, 25] starts from a “gist” of an image and converts it into words, while the bottom-up one [12, 19, 23, 9, 20, 11, 22] first comes up with words describing various aspects of an image and then combines them.”
Logic Tensor Networks for Semantic Image Interpretation (2017, background): “In [Kulkarni et al., 2011], potential functions are defined as text priors such as co-occurrence of terms found in the image descriptions of Flickr (e.g. object labels with property labels or relationship labels).”
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images (2015, background): “Multimodal learning of language and vision The methods of image-sentence retrieval [17, 50], image description generation [28, 42, 20] and visual question-answering [18, 36, 1] have developed very fast in recent years.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “…the system’s rendition of the content (e.g. Lester & Porter, 1997; Sripada et al., 2005; Hunter et al., 2012), a criterion often used in subjective evaluations of image-captioning systems as well (e.g. Kulkarni et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Elliott & Keller, 2013).”
Visual Relationship Detection with Language Priors (2016, background): “Some papers explicitly collected relationships in images [25,26,27,28,29] and videos [27,30,31] and helped models map these relationships from images to language.”
Diagram Understanding in Geometry Questions (2014, background): “Coupling visual and textual information has recently attracted attention in both vision and NLP (Farhadi et al. 2010; Kulkarni et al. 2011; Gupta and Mooney 2010).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “However, in terms of performance, there exists a large gap between the evaluation of the metrics and human judgment of evaluation [20, 30, 74].”
Detecting Visual Relationships with Deep Relational Networks (2017, background): “Earlier efforts often focus on speciﬁc types of relationships, such as positional relations [8–13] and actions ( i.e . interactions be-tween objects) [14–24].”
Semantic Aware Video Transcription Using Random Forest Classifiers (2014, methodology): “[16] proposed a method to detect candidate objects and their attributes from static images, and applied CRF for sentence generation.”
Deepdiary: Lifelogging image captioning and summarization (2018, methodology): “Besides employing neural network models for caption generation, Kulkarni et al. [54] jointly reason about objects and attributes through a CRF and ﬁ ll corresponding slots in a template.”