Every Picture Tells a Story: Generating Sentences from Images

By Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, D.A. Forsyth. 2010

1,284 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “Older work made use of fixed visual representations and translated them into textual descriptions [7, 18].”
Experience Grounds Language (2020, background): “In the last decade of work descendant from image captioning (Farhadi et al., 2010; Mitchell et al., 2012), a myriad of tasks on visual question answering (Antol et al.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “In [9], a method for mapping visual and textual data to a common space based on extracting a triplet of object, action, and scene is presented.”
Learning Visual Features from Large Weakly Supervised Data (2015, background): “Some works co-embed images and words [14], whereas others co-embed images and sentences or n-grams [13, 23, 53].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “Template-based methods involve ﬁlling in sentence templates, such as triplets, based on the results of object detections and spatial relationships [21, 22, 23, 24, 25].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Inspired by previous work [5, 22] we observe that a dependency tree of a sentence provides a rich set of typed relationships that can serve this purpose more effectively than individual words or bigrams.”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, methodology): “Conversely, when given a query image, we would like to find a description that goes beyond a single label by providing a correct sentence describing it, a task that has recently garnered a lot of attention (Farhadi et al., 2010; Ordonez et al., 2011; Kuznetsova et al., 2012).”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, background): “…that they should also be of great interest for people who aim to build systems that automatically associate image with sentences that describe them (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Kuznetsova et al., 2012; Gupta et al., 2012;…”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, methodology): “But with the exception of Farhadi et al. (2010), who incorporate Lin's similarity into their model, but do not evaluate what benefit they obtain from it, WordNet's hypernym-hyponym relations have only been used superficially for associating images and text (Weston et al., 2010; Ordonez et al.,…”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “Early methods [19,31,37,65,79] focus on filling templates with extracted objects, attributes, and relationships.”
VQA: Visual Question Answering (2015, methodology): “Related to VQA are the tasks of image tagging [8, 23], image captioning [24, 14, 34, 7, 13, 45, 9, 19, 32, 21] and video captioning [39, 17], where words or sentences are generated to describe visual content.”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “Several early works [53], [54], [55], [56] on image captioning combine object and scene recognition with template or tree based approaches to generate captions.”
Show and tell: A neural image caption generator (2014, methodology): “Descriptions were preprocessed with basic tokenization, keeping all words that appeared at least 5 times in the training set.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “Some pioneering approaches that address the challenge of generating image descriptions have been developed [23, 7].”
CIDEr: Consensus-based image description evaluation (2014, methodology): “Story [14] is the lowest ranked, by both humans and CIDEr.”
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, background): “As mentioned in the Introduction, the most common imagelanguage understanding task in the literature is automatic image captioning (Chen and Zitnick 2015; Donahue et al. 2015; Fang et al. 2015; Farhadi et al. 2010; Hodosh et al. 2013; Karpathy and Fei-Fei 2015; Kiros et al. 2014; Klein et al. 2014; ”