Baby talk: Understanding and generating simple image descriptions

By Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, Tamara L. Berg. 2011

190 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Devi Parikh
3. Jianfeng Gao
4. Trevor Darrell
5. Yejin Choi
6. Marcus Rohrbach
7. Yoshua Bengio
8. Li Fei-Fei

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Trevor Darrell
3. Dacheng Tao
4. Li Fei-Fei
5. Xiaogang Wang
6. Andrew Zisserman
7. Bernt Schiele
8. Yi Yang

In Multimodal Learning

1. Devi Parikh
2. Marcus Rohrbach
3. Dhruv Batra
4. Mohit Bansal
5. Li Fei-Fei
6. Trevor Darrell
7. Zhe Gan
8. Qi Wu

Citations in the corpus

• From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “ically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
• Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, positive): “Some pioneering approaches that address the challenge of generating image descriptions have been developed [23, 7].”
• ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, methodology): “Bottom up methods (Kulkarni et al., 2011; Yang et al., 2011; Yao et al., 2010) rely on visual classifiers to first recognize image content and then construct captions from scratch, perhaps with some”
• Movie Description (2016, background): “Prior work on image description includes Farhadi et al. (2010); Kulkarni et al. (2011); Kuznetsova et al. (2012); Li et al. (2011); Kuznetsova et al. (2014); Mitchell et al. (2012); Socher et al. (2014).”
• Phrase-based Image Captioning (2015, background): “Many of them use syntactic and semantic constraints in the generation process (Yao et al., 2010; Mitchell et al., 2012; Kulkarni et al., 2011; Kuznetsova et al., 2012).”
• Neural Baby Talk (2018, methodology): “[24] detect objects, attributes, and prepositions, jointly reason about these through a CRF, and finally fill appropriate slots in a template.”
• Choosing Linguistics over Vision to Describe Images (2012, neutral): “Some of these approaches (Kulkarni et al. 2011; Yang et al. 2011; Li et al. 2011) explore the use of corpus to smooth the errors in the detections.”
• Label-Embedding for Image Classification (2015, neutral): “Attributes have been used for image description [19], [18], [9], caption generation [27], [41], face recognition [29], [51], [10], image retrieval [28], [56], [15], action recognition [32], [69], novelty detection [62] and object classification [30], [18], [64], [65], [34], [54], [38].”
• See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, methodology): “…et al., 2010; Ordonez et al., 2011), using text surrounding an image in a news article (Feng and Lapata, 2010), filling visual sentence templates (Kulkarni et al., 2011; Yang et al., 2011; Elliott and Keller, 2013), and stitching together existing sentence descriptions (Gupta and Mannem, 2012;…”
• Associating neural word embeddings with deep image representations using Fisher Vectors (2015, background): “[18] suggested a system that generates a descriptive text (not from a fixed set) for a given query image.”
• Composing Simple Image Descriptions using Web-scale N-grams (2011, methodology): “Last, we allow for some creativity in the generation process which produces more human-like descriptions than a closely related, very recent approach that drives annotation more directly from computer vision inputs (Kulkarni et al., 2011).”
• Generating Natural-Language Video Descriptions Using Text-Mined Knowledge (2013, unknown): “Most prior work on natural-language description of visual data has focused on static images (Felzenszwalb et al., 2008; Kulkarni et al., 2011; Kuznetsova et al., 2012; Laptev et al., 2008; Li et al., 2011; Yao et al., 2010).”
• Generalizing Image Captions for Image-Text Parallel Corpus (2013, background): “Several recent studies presented approaches to automatic caption generation for images (e.g., Farhadi et al. (2010), Feng and Lapata (2010a), Feng and Lapata (2010b), Yang et al. (2011), Kulkarni et al. (2011), Li et al. (2011), Kuznetsova et al. (2012)).”
• Nonparametric Method for Data-driven Image Captioning (2014, background): “Much previous work in image captioning focuses on generating captions that concretely describe detected objects and entities (Kulkarni et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Yu and Siskind, 2013).”
• Towards Transparent Systems: Semantic Characterization of Failure Modes (2014, methodology): “Attributes have also been used for generating automatic textual description of images [22, 37] that can potentially point out anomalies in objects [23].”