Composing Simple Image Descriptions using Web-scale N-grams

By Siming Li, Girish Kulkarni, Tamara L. Berg, Alexander C. Berg, Yejin Choi. 2011

416 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

Citations in the corpus

Learning Visual N-Grams from Web Data (2016, neutral): “Our study is most similar to that of [40], which uses n-gram to generate image descriptions; unlike [40], we we do not rely on separately trained image-classification pipelines.”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “scribed here in more detail: Template-based methods. Template-based methods involve ﬁlling in sentence templates, such as triplets, based on the results of object detections and spatial relationships [21, 22, 23, 24, 25]. While 2 Figure 2: Encoder: A deep convolutional network (CNN) and long short-t”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Some approaches focus on the direction of describing the contents of images, formulated either as a task of mapping images to a fixed set of sentences written by people [5, 6], or as a task of automatically generating novel captions [7, 8, 9, 10, 11, 12].”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “ically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012; Hodosh et al., 2013).”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, background): “It has been used by a large number of image description systems (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Gupta et al., 2012), but since almost all of these systems (the only exception being Gupta et al., 2012) rely on detectors trained…”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, neutral): “Early methods [19,31,37,65,79] focus on filling templates with extracted objects, attributes, and relationships.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “The ﬁrst involved generating caption templates which were ﬁlled in based on the results of object detections and attribute discovery (Kulkarni et al. (2013), Li et al. (2011), Yang et al. (2011), Mitchell et al. (2012), Elliott & Keller (2013)).”
From captions to visual concepts and back (2014, neutral): “Previous work utilizes syntactic and semantic constraints in the generation process [28, 45, 23, 21, 20, 44], and we compare against prior state of the art in this line of work.”
Movie Description (2016, background): “We also discuss works which build on our dataset.”
A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, background): “There has been a recent spike in efforts to automatically describe visual content in natural language (Yang et al., 2011; Kulkarni et al., 2011; Li et al., 2011; Farhadi et al., 2010; Krishnamoorthy et al., 2013; Elliott and Keller, 2013; Yu and Siskind, 2013; Socher et al., 2014).”
Comparing Automatic Evaluation Measures for Image Description (2014, methodology): “Image description has been compared to translating an image into text (Li et al., 2011; Kulkarni et al., 2011) or summarising an image 1.”
Image Captioning with Semantic Attention (2016, background): “The topdown paradigm [4, 35, 26, 16, 8, 37, 25] starts from a “gist” of an image and converts it into words, while the bottom-up one [12, 19, 23, 9, 20, 11, 22] first comes up with words describing various aspects of an image and then combines them.”
Image Description using Visual Dependency Representations (2013, methodology): “Previous approaches to automatic description generation have typically tackled the problem using an object recognition system in conjunction with a natural language generation component based on language models or templates (Kulkarni et al., 2011; Li et al., 2011).”
Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich ﬁxed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation (2014, background): “The works of [28, 19, 23] have focused on generating novel descriptive sentences for a query image.”