Déjà Image-Captions: A Corpus of Expressive Descriptions in Repetition

By Jianfu Chen, Polina Kuznetsova, David Scott Warren, Yejin Choi. 2015

43 citations recorded by the source corpus.

We present a new approach to harvesting a large-scale, high quality image-caption corpus that makes a better use of already existing web data with no additional human efforts. The key idea is to focus on Deja Image-Captions: naturally existing image descriptions that are repeated almost verbatim – by more than one individual for different images. The resulting corpus provides association structure between 4 million images with 180K unique captions, capturing a rich spectrum of everyday narratives including figurative and pragmatic language. Exploring the use of the new corpus, we also present new conceptual tasks of visually situated paraphrasing, creative image captioning, and creative visual paraphrasing.

Subfields: Computer Vision and Pattern Recognition, Language and Linguistics, Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Language and Linguistics

In Multimodal Learning

In Natural Language Generation

Citations in the corpus

Exploring Nearest Neighbor Approaches for Image Captioning (2015, background): “This surge in research is due in part to the creation of large caption datasets [11], [29], [13], [37], [1], [24], and new learning techniques [20], [12].”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “Source http://mscoco.org/dataset/#captions-eval The Déjà-Image Captions dataset (Chen et al., 2015) contains 4,000,000 images with 180,000 near-identical captions harvested from Flickr.”
Captioning Images Taken by People Who Are Blind (2020, background): “Over the past decade, nearly 20 publicly-shared captioning datasets have been created to support the development of automated captioning algorithms [6, 14, 15, 17, 18, 20, 24, 25, 26, 29, 30, 40, 43, 50, 51, 54].”
Visual Storytelling (2016, background): “Work in vision to language has exploded, with researchers examining image captioning (Lin et al., 2014; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Xu et al., 2015; Chen et al., 2015; Young et al., 2014; Elliott and Keller, 2013), question answering (Antol et al.”
Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation (2017, background): “Signiﬁcant advances in image captioning (Chen et al., 2015; Fang et al., 2014; Donahue et al., 2014; Chen et al., 2015) have enabled much interdisciplinary research in vision and language, from video transcription (Rohrbach et al., 2012; Venugopalan et al., 2015), to answering questions about…”
A Survey of Current Datasets for Vision and Language Research (2015, methodology): “• D´ej`a Images Dataset (Chen et al., 2015) consists of 180K unique user-generated captions associated with 4M Flickr images, where one caption is aligned with multiple images.”
Generating Natural Questions About an Image (2016, background): “We are witnessing a renewed interest in interdisciplinary AI research in vision & language, from descriptions of the visual input such as image cap-tioning (Chen et al., 2015; Fang et al., 2014; Don-ahue et al., 2014; Chen et al., 2015) and video Natural Questions: -”
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning (2018, background): “Scene captioning ([14, 18, 42, 71]) consists in generating a sentence that describes an image, or matching an image with one or several captions.”