Im2Text: Describing Images Using 1 Million Captioned Photographs

By Vicente Ordonez, Girish Kulkarni, Tamara L. Berg. 2011

1,594 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Microsoft COCO: Common Objects in Context (2014, methodology): “We accomplished this using a surprisingly simple yet effective technique that queries for pairs of objects in conjunction with images retrieved via scene-based queries [17], [3].”
Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Several methods have been proposed to create larger datasets automatically with Ordonez et al. (2011) as a notable early example.”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “We also use additional Out-of-domain data from Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011) for model training 7 .”
A Multimodal Framework for the Detection of Hateful Memes (2020, neutral): “The large-scale pretraining is performed over four V+L datasets – COCO, Visual Genome, CC, and SBU Captions [26].”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “In general image-caption datasets [45, 29, 54, 41, 43], many image-text pairs contain either no scene text-related visual regions or no scene text-related language referring, and are thus less helpful to Text-VQA/TextCaption.”
Learning Video Representations from Textual Web Supervision (2020, methodology): “Another approach is to use text captions, and label each image with key words present in the caption [50, 30, 39, 48].”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Conceptual Captions [14] and SBU Captions [15] are the most widely-used datasets for image-text pre-training, with 3.0M and 1.0M image descriptions respectively.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “entation Details For UNITER experiments, we pre-train with the same four large-scale datasets used in the original model: COCO [36], Visual Genome (VG) [28], Conceptual Captions [58] and SBU Captions [49]. VILLA is applied to both MLM and ITM pre-training tasks. The original UNITER-base (12 layers) ”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “superior performance in our experiments. Pre-training Corpus We have built the pre-training corpus based on the existing V+L datasets, including COCO [21], Conceptual Captions (CC) [32], SBU captions [27], icker30k [45], GQA [13] etc. As shown in Table 1, the number of unique images is 4.1 million. ”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “UNITER[21] combines four datasets (Conceptual Captions[2], SBU Captions[3], Visual Genome[22] and MSCOCO[5]) together to form a 9.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “Due to some broken urls, the size of image-caption pairs of Conceptual Captions dataset is about 3M. Similar to Conceptual Captions, SBU Captions (Ordonez, Kulkarni, and Berg 2011) dataset is also automatically collected from Web and contains 1M image-caption pairs.”
Self-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces (2017, neutral): “This includes classical image classification [5, 10], annotation [9, 17], and captioning [24, 23].”
Learning Visual Features from Large Weakly Supervised Data (2015, neutral): “In contrast to our work, many prior studies also attempt to explicitly discard low-quality labels by developing algorithms that identify relevant image-hashtag pairs from a weakly labeled dataset [12, 35, 54].”
Webly Supervised Learning of Convolutional Networks (2015, background): “We believe a more juicier direction is to feed more data into these networks (in fact, deeper networks would often need more data to train).”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “We generated image descriptions for roughly 800 images from the SBU captioned photo dataset [40].”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, background): “Some approaches focus on the direction of describing the contents of images, formulated either as a task of mapping images to a fixed set of sentences written by people [5, 6], or as a task of automatically generating novel captions [7, 8, 9, 10, 11, 12].”
Grounded Compositional Semantics for Finding and Describing Images with Sentences (2014, background): “Conversely, when given a query image, we would like to find a description that goes beyond a single label by providing a correct sentence describing it, a task that has recently garnered a lot of attention (Farhadi et al., 2010; Ordonez et al., 2011; Kuznetsova et al., 2012).”
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions (2014, methodology): “Coreference resolution has also been used for word sense disambiguation by Preiss (2001) and Hu and Liu (2011).”