Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
By Piyush Sharma, Nan Ding, Sebastian Goodman, Radu Soricut. 2018
3,041 citations recorded by the source corpus.
We present a new dataset of image caption annotations, Conceptual Captions, which contains an order of magnitude more images than the MS-COCO dataset (Lin et al., 2014) and represents a wider variety of both images and image caption styles. We achieve this by extracting and filtering image caption annotations from billions of webpages. We also present quantitative evaluations of a number of image captioning models and show that a model architecture based on Inception-ResNetv2 (Szegedy et al., 2016) for image-feature extraction and Transformer (Vaswani et al., 2017) for sequence modeling achieves the best performance when trained on the Conceptual Captions dataset.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Multimodal Learning
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Jianfeng Gao
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Douwe Kiela
- Samuel R. Bowman
- Matt Gardner
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, background): “…demonstrated an additional set of (image, text) pairs collected from the internet could improve retrieval performance and several new automatically constructed datasets such as Conceptual Captions (Sharma et al., 2018), LAIT (Qi et al., 2020), and OCR-CC (Yang et al., 2020) have been created.”
- High-Resolution Image Synthesis with Latent Diffusion Models (2021, methodology): “Text-conditional image synthesis on the Conceptual Captions [74] dataset: Our model outperforms autoregressive methods involving much more parameters by a significant margin.”
- Experience Grounds Language (2020, background): “via conceptual captions (Sharma et al., 2018)) or further broadened to include audio (Tsai et al.”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Our UNITER model is trained on a large-scale V+L dataset composed of four subsets: ( i ) COCO (Lin et al., 2014); ( ii ) Visual Genome (VG) (Krishna et al., 2017); ( iii ) Conceptual Captions (CC) (Sharma et al., 2018); and ( iv ) SBU Captions (Ordonez et al., 2011).”
- Root Mean Square Layer Normalization (2019, methodology): “and requiring no dependencies among training cases, LayerNorm has been widely applied to different neural architectures, which enables remarkable success on various tasks ranging from computer vision [19, 26], speech recognition [37] to natural language processing [31, 35]. In some cases, LayerNorm ”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, background): “See § 3.3 for more details. on Conceptual Captions (Sharma et al., 2018) and conduct evaluation on four datasets, two of which are also considered in our work.”
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “2.2 to the Conceptual Captions dataset [24].”
- A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “The large-scale pretraining is performed over four V+L datasets – COCO, Visual Genome, CC, and SBU Captions [26].”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Conceptual Captions [14] and SBU Captions [15] are the most widely-used datasets for image-text pre-training, with 3.0M and 1.0M image descriptions respectively.”
- VirTex: Learning Visual Representations from Textual Annotations (2020, methodology): “These methods employ complex pretraining pipelines: they typically (1) start from an ImageNet-pretrained CNN; (2) extract region features using an object detector finetuned on Visual Genome [73], following [74]; (3) optionally start from a pretrained language model, such as BERT [64]; (4) combine th”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “izability test. Implementation Details For UNITER experiments, we pre-train with the same four large-scale datasets used in the original model: COCO [36], Visual Genome (VG) [28], Conceptual Captions [57] and SBU Captions [49]. VILLA is applied to both MLM and ITM pre-training tasks. The original UN”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, methodology): “Specifically, we include the official multi-modally pretrained versions in the ViLBERT (trained on Conceptual Captions [66], ViLBERT CC ) and Visual BERT (trained on COCO, Visual BERT COCO ).”
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “methods, it yields superior performance in our experiments. Pre-training Corpus We have built the pre-training corpus based on the existing V+L datasets, including COCO [21], Conceptual Captions (CC) [32], SBU captions [27], icker30k [45], GQA [13] etc. As shown in Table 1, the number of unique imag”
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “UNITER[21] combines four datasets (Conceptual Captions[2], SBU Captions[3], Visual Genome[22] and MSCOCO[5]) together to form a 9.”
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “UNITER add about 1 million image-caption pair besides the Conceptual Captions and in-domain MSCOCO Caption and Visual Genome Dense Captions (Krishna et al. 2017) data.”
- Zero-Shot Text-to-Image Generation (2021, methodology): “As described in Section 2.3, this dataset incorporates Conceptual Captions, the text-image pairs from Wikipedia, and a filtered subset of YFCC100M.”
- Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “The scene-based transformer is trained on a union of CC12m [7], CC [51], and subsets of YFCC100m [55] and Redcaps [10], amounting to 35 m text-image pairs.”
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “To demonstrate the zero-shot task transfer ability of our model, we also consider a variant that is pretrained on the Google Conceptual Captions 3M (CC3M) dataset [41], which consists of 3.”