Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

By Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut. 2021

1,502 citations recorded by the source corpus.

The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pretraining. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pretraining data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [54] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for visionand-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Zhe Gan
4. Percy Liang
5. Caiming Xiong
6. Quoc V. Le
7. Luke S. Zettlemoyer
8. Furu Wei

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Trevor Darrell
6. Jitendra Malik
7. Cordelia Schmid
8. Yu Qiao

Citations in the corpus

• Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “The scene-based transformer is trained on a union of CC12m [7], CC [51], and subsets of YFCC100m [55] and Redcaps [10], amounting to 35 m text-image pairs.”
• SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “We also evaluate on Conceptual Captions 3M (CC3M) [49] and Conceptual Captions 12M (CC12M) [8].”
• Vector Quantized Diffusion Model for Text-to-Image Synthesis (2021, methodology): “To further demonstrate the scalability of our method, we also train our model on large scale datasets, including Conceptual Captions [6, 53] and LAION-400M [51].”
• Flamingo: a Visual Language Model for Few-Shot Learning (2022, neutral): “To alleviate this lack of data, numerous works [14, 50, 98, 110] automatically scrape readily available paired vision-text data.”
• Visual Instruction Tuning (2023, background): “The community has witnessed a surge in the amount of public multimodal data such as image-text pairs, ranging from CC [8] to LAION [45].”
• MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “For all ablations in 4.2 we train on CC12M [7] for a total of around 3M samples ensuring comparability with Frozen .”
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “For language modeling with image clues and image-text retrieval, we use a combination of COCO Caption [13], SBU Captions (SBU) [58], Visual Genome [36], CC3M [68], CC12M [9] and YFCC [31].”
• Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “Validation Set 1 Encoder-Decoder [4] CC3M [35] 81.”
• Florence: A New Foundation Model for Computer Vision (2021, methodology): “, 2018), CC12M (Changpinyo et al., 2021), SBU (Ordonez et al.”
• LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “Specifically, we tune an ImageNet-21k pre-trained ViTL/16 model [54] on the union of the YFCC100m-CLIP and CC12M datasets.”
• FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, background): “, 2016) and CC12M (Changpinyo et al., 2021), as well as larger-scale datasets with more than 100M samples in CLIP (Radford et al.”
• MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “With fine-tuning, MURAL-BASE (CC12M+EOBT) improves on both fine-tuned M3P and UC2 (14 vs 10,11), except for Japanese.”
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “To show that our method is scalable with larger-scale web data, we also include the much noisier Conceptual 12M dataset [43], increasing the total number of images to 14.”
• Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Conceptual-Captions 12M [9]: A dataset comprising of 12 million captioned images, intended for large-scale vision language pre-training.”
• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “PT denotes pre-training on additional datasets [8, 29, 58, 7, 57], FT denotes fine-tuning on COCO Caption.”
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, positive): “We use the same pre-training dataset as BLIP with 129M images in total, including COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), CC3M (Sharma et al., 2018), CC12M (Changpinyo et al., 2021), SBU (Ordonez et al., 2011), and 115M images from the LAION400M dataset (Schuhmann et al., 2021”