Corpus-Guided Sentence Generation of Natural Images

By Yezhou Yang, Ching Lik Teo, Hal Daumé Iii, Yiannis Aloimonos. 2011

423 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “Our work is also related to prior studies on multi-modal embedding [44, 56] that explore approaches such as kernel canonical component analysis [16, 18], restricted Boltzmann machines [45], topic models [22], and log-bilinear models [25].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Other approaches to multimodal learning include the use of deep Boltzmann machines [16], log-bilinear neural language models [2], autoencoders [17], recurrent neural networks [7] and topic-models [18].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Some approaches focus on the direction of describing the contents of images, formulated either as a task of mapping images to a fixed set of sentences written by people [5, 6], or as a task of automatically generating novel captions [7, 8, 9, 10, 11, 12].”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “ically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “Following common practice in natural language generation, Yang et al. (2011) and Kulkarni et al. (2011) evaluated captions on a graded scale for relevance Hodosh, Young & Hockenmaier”
Attention on Attention for Image Captioning (2019, background): “It requires not only to recognize salient objects in an image, understand their interactions, but also to verbalize them using natural language, which makes itself very challenging [25, 45, 28, 12].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “Several early works [53], [54], [55], [56] on image captioning combine object and scene recognition with template or tree based approaches to generate captions.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “The ﬁrst involved generating caption templates which were ﬁlled in based on the results of object detections and attribute discovery (Kulkarni et al. (2013), Li et al. (2011), Yang et al. (2011), Mitchell et al. (2012), Elliott & Keller (2013)).”
CIDEr: Consensus-based image description evaluation (2014, background): “on separate one to five [29, 37, 44, 11] or pairwise scales [45].”
ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, methodology): “Bottom up methods (Kulkarni et al., 2011; Yang et al., 2011; Yao et al., 2010) rely on visual classi-ﬁers to ﬁrst recognize image content and then construct captions from scratch, perhaps with some Figure 1: An example game.”
From captions to visual concepts and back (2014, methodology): “Previous work utilizes syntactic and semantic constraints in the generation process [28, 45, 23, 21, 20, 44], and we compare against prior state of the art in this line of work.”
Visual Madlibs: Fill in the blank Image Generation and Question Answering (2015, background): “The ﬁrst type of approach focused on detecting content elements such as objects, attributes, activities, or spatial relationships and then composing captions for images [18, 33, 26, 10] or videos [17] using linguistically inspired templates.”
Generation and Comprehension of Unambiguous Object Descriptions (2015, methodology): “Our methods are inspired by a long line of inquiry in joint models of images and text, primarily in the vision and learning communities [13, 23, 49, 43, 34, 56, 36].”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, methodology): “Numerous papers have explored the area of generating novel image descriptions [7, 36, 19, 37, 28, 11, 20, 17].”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “There has been a recent spike in efforts to automatically describe visual content in natural language (Yang et al., 2011; Kulkarni et al., 2011; Li et al., 2011; Farhadi et al., 2010; Krishnamoorthy et al., 2013; Elliott and Keller, 2013; Yu and Siskind, 2013; Socher et al., 2014).”
Comparing Automatic Evaluation Measures for Image Description (2014, methodology): “Recent approaches to this task have been based on slot-ﬁlling (Yang et al., 2011; Elliott and Keller, 2013), combining web-scale n-grams (Li et al., 2011), syntactic tree substitution (Mitchell et al., 2012), and description-by-retrieval (Farhadi et al., 2010; Ordonez et al., 2011; Hodosh et al.,…”
Image Description using Visual Dependency Representations (2013, methodology): “Yang et al. (2011) fill in a sentence template by selecting the likely objects, verbs, prepositions, and scene types based on a Hidden Markov Model.”