RedCaps: web-curated image-text data created by the people, for the people

By Karan Desai, Gaurav Kaul, Z. Aysola, Justin Johnson. 2021

213 citations recorded by the source corpus.

Large datasets of paired images and text have become increasingly popular for learning generic representations for vision and vision-and-language tasks. Such datasets have been built by querying search engines or collecting HTML alt-text -- since web data is noisy, they require complex filtering pipelines to maintain quality. We explore alternate data sources to collect high quality data with minimal filtering. We introduce RedCaps -- a large-scale dataset of 12M image-text pairs collected from Reddit. Images and captions from Reddit depict and describe a wide variety of objects and scenes. We collect data from a manually curated set of subreddits, which give coarse image labels and allow us to steer the dataset composition without labeling individual instances. We show that captioning models trained on RedCaps produce rich and varied captions preferred by humans, and learn visual representations that transfer to many downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Evaluation & Benchmarks

Citations in the corpus

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, methodology): “The scene-based transformer is trained on a union of CC12m [7], CC [51], and subsets of YFCC100m [55] and Redcaps [10], amounting to 35 m text-image pairs.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “In contrast to relying on alt-text, RedCaps used the captions provided by Reddit users to collect higher quality captions [14].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “To show that our approach is generalizable to other training datasets, besides CC [3, 11] and filtered YFCC [12], we also train GroupViT on the recently released RedCaps dataset [6], which contains 12 millions image-text pairs from Reddit, of similar size as filtered YFCC.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, unknown): “…Captions (CC) [63], SBU Captions [52], Flickr30k [88], VQA 2.0 [27], VCR [89], NLVR2 [74], SNLI-VE [87], QNLI [57], MLNI-mm [84], QQP [36], Localized Narratives (LN) [55], Wikipedia Image Text (WIT) [73], Conceptual Captions 12M (CC 12M) [10], Red Caps (RC) [15], YFCC100M [77], and SST-2 [72].”
Power Hungry Processing: Watts Driving the Cost of AI Deployment? (2023, unknown): “By testing a variety of architectures on different tasks and datasets, we aim to gain a better understanding of the degree of variance that can be observed and how seemingly small user choices can result in large differences in model’s environmental impacts.”
Sigmoid Loss for Language Image Pre-Training (2023, background): “Seminal works CLIP [29] and ALIGN [18] established the viability of this approach at a large scale, and following their success, many large image-text datasets became available privately [45, 9, 17, 37] and publicly [31, 3, 11, 4, 32].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, background): “Text-based Generation has seen significant advances due to large-scale image-text datasets [73, 72, 14, 71] and scalable generative model architectures [16, 66, 60, 32], enabling synthesis of novel images from text [20, 3, 54].”
Improving CLIP Training with Language Rewrites (2023, methodology): “RedCaps is a 12M-instance dataset collected exclusively from Reddit, potentially exhibiting distinct distributions compared to other datasets.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “• RedCaps [95] is a large-scale dataset with 12M image-text samples collected from 350 sub-reddits.”
DataComp: In search of the next generation of multimodal datasets (2023, methodology): “Even for public image-text datasets like SBU [97], Flickr30k [142], MS-COCO [22], Conceptual Captions [124], CC12M [20], RedCaps [34], WIT [127], Shutterstock [94], YFCC-100M [131], COYO700M [16], LAION-400M [121], or LAION-2B [122] little is known about what constitutes a good image-text dataset.”
TiC-CLIP: Continual Training of CLIP Models (2023, unknown): “We also create T I C-YFCC and T I C-RedCaps on top of existing YFCC15M (Thomee et al., 2016; Radford et al., 2021) and Redcaps (Desai et al., 2021) datasets to highlight that our findings are broadly applicable to carefully curated datasets from diverse sources such as Reddit and Flickr.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, methodology): “Second, we design a pretraining strategy that effectively leverages cross-modal supervision in the form of transcribed speech from unlabeled narrated videos by reformulating sentence boundaries as pseudo event boundaries, as presented in Section 3.2 and Figure 3.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, background): “To this end, researches usually leverage existing web images with alt-text [12–16] and videos with ASR transcriptions [2, 4, 5, 3, 6, 17, 18] for scalable learning.”
StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis (2023, methodology): “We train using a union of several datasets: CC12m (Changpinyo et al., 2021), CC (Sharma et al., 2018), YFCC100m (ﬁltered) (Thomee et al., 2016; Singh et al., 2022), Redcaps (Desai et al., 2021), LAION-aesthetic-6+ (Schuhmann et al., 2022).”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “• Red Caps [100] is a image-text dataset collected from social media Reddit, which contains 12M image-text pairs covering various objects and scenes.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “Data sources with image and text pairs (Desai et al., 2021), bounding boxes (Krishna et al., 2017) or semantic segmentation (Gupta et al., 2019) can be used to build these tasks.”
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? (2022, methodology): “However, the documentation of Desai et al. [2021] shows that careful manual documentation is difficult, if not even unfeasible, due to the immense size of current datasets: ‘ We manually checked 50K [out of 12M] random images in RedCaps and found one image containing nudity (exposed buttocks; no…”