Midge: Generating Image Descriptions From Computer Vision Detections

By Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa C. Mensch, Alexander C. Berg, Tamara L. Berg, Hal Daumé Iii. 2012

495 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “In the last decade of work descendant from image captioning (Farhadi et al., 2010; Mitchell et al., 2012), a myriad of tasks on visual question answering (Antol et al.”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Other approaches to multimodal learning include the use of deep Boltzmann machines [16], log-bilinear neural language models [2], autoencoders [17], recurrent neural networks [7] and topic-models [18].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Some approaches focus on the direction of describing the contents of images, formulated either as a task of mapping images to a fixed set of sentences written by people [5, 6], or as a task of automatically generating novel captions [7, 8, 9, 10, 11, 12].”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “ically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “It has been used by a large number of image description systems (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Gupta et al., 2012), but since almost all of these systems (the only exception being Gupta et al., 2012) rely on detectors trained…”
Attention on Attention for Image Captioning (2019, background): “It requires not only to recognize salient objects in an image, understand their interactions, but also to verbalize them using natural language, which makes itself very challenging [25, 45, 28, 12].”
VQA: Visual Question Answering (2015, methodology): “Related to VQA are the tasks of image tagging [8, 23], image captioning [24, 14, 34, 7, 13, 45, 9, 19, 32, 21] and video captioning [39, 17], where words or sentences are generated to describe visual content.”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “Several early works [53], [54], [55], [56] on image captioning combine object and scene recognition with template or tree based approaches to generate captions.”
Show and tell: A neural image caption generator (2014, background): “The nonlinearities are sigmoid σ and hyperbolic tangent h .”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
From captions to visual concepts and back (2014, positive): “Captions using our approach (black), Midge [28] (blue) and Baby Talk [20] (red) are shown.”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “They parse ... the sentence and divide it into several parts (Mitchell et al. (2012); Gupta & Mannem (2012)).”
Movie Description (2016, background): “Prior work on image description includes Farhadi et al. (2010); Kulkarni et al. (2011); Kuznetsova et al. (2012); Li et al. (2011); Kuznetsova et al. (2014); Mitchell et al. (2012); Socher et al. (2014).”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
Explain Images with Multimodal Recurrent Neural Networks (2014, background): “They parse the sentence and divide it into several parts [28, 11].”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, background): “The ﬁrst type of approach focused on detecting content elements such as objects, attributes, activities, or spatial relationships and then composing captions for images [18, 33, 26, 10] or videos [17] using linguistically inspired templates.”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, methodology): “We experiment on all the image-sentence datasets described previously and compare to the RNN baselines and other previous papers [28, 19].”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “…level of linguistic expressiveness, flexibility, and creativity than those based on rules or templates (Kulkarni et al., 2011; Yang et al., 2011; Mitchell et al., 2012), while also addressing long-distance grammatical relations in a more principled way than those based on hand-coded constraints…”