DataComp: In search of the next generation of multimodal datasets

By S. Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, G. Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang and 24 more. Neural Information Processing Systems, 2023

734 citations recorded by the source corpus.

Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model architectures or training algorithms. To address this shortcoming in the ML ecosystem, we introduce DataComp, a testbed for dataset experiments centered around a new candidate pool of 12.8 billion image-text pairs from Common Crawl. Participants in our benchmark design new filtering techniques or curate new data sources and then evaluate their new dataset by running our standardized CLIP training code and testing the resulting model on 38 downstream test sets. Our benchmark consists of multiple compute scales spanning four orders of magnitude, which enables the study of scaling trends and makes the benchmark accessible to researchers with varying resources. Our baseline experiments show that the DataComp workflow leads to better training sets. In particular, our best baseline, DataComp-1B, enables training a CLIP ViT-L/14 from scratch to 79.2% zero-shot accuracy on ImageNet, outperforming OpenAI's CLIP ViT-L/14 by 3.7 percentage points while using the same training procedure and compute. We release DataComp and all accompanying code at www.datacomp.ai.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f9570989919338079088270a9cf1a7afc8db8093

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, methodology): “We follow [12] to use a threshold of 0.1 to filter out potentially toxic topic classes.”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, neutral): “[53].”
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (2023, background): “∈ R d × d learnable projection matrices, and C l ∈ R ( L V + L T ) × d context features of l -th layer.”
Kimi-VL Technical Report (2025, positive): “We have integrated various open-source Chinese and English caption datasets like (Schuhmann et al. 2022; Gadre et al. 2024) and also collected substantial in-house caption data from multiple sources.”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond (2023, methodology): “DataComp (Gadre et al., 2023) and Coyo (Byeon et al., 2022) are collections of image-text pairs.”
TiC-CLIP: Continual Training of CLIP Models (2023, positive): “We create this dataset on top of the existing DataComp benchmark (Gadre et al., 2023).”
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, methodology): “Research [16, 38] explores methods to filter and blend raw and synthetic captions to enhance the CLIP model.”
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (2023, neutral): “Method Data MSR-VTT LSMDC DiDeMo MSVD ANet T2V V2T T2V V2T T2V V2T T2V V2T T2V V2T CLIP DataComp-1B [70] 37.”
OmniGen: Unified Image Generation (2024, neutral): “Her eyes were full of expectation, and the sun shone on her warm face.”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, background): “…including Conceptual Captions (CC3M/CC12M) (Changpinyo et al., 2021), COCO (Lin et al., 2014), Laion (Schuhmann et al., 2022), COYO (Byeon et al., 2022), DataComp (Gadre et al., 2023), Wukong (Gu et al., 2022), ImageNet (Deng et al., 2009), OCR-CC (Yang et al., 2021) and SBU (Ordonez et al., 2011).”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, background): “The best brand is subjective and depends on personal preferences and the quality of the product.”
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, methodology): “In this way, DRO-style data-centric methods such as DoReMi can improve the representation disparity between majority and minority subgroups in a dataset.”
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning (2025, neutral): “The process begins with the aggregation of an initial pool of over 10 billion image-text pairs from diverse sources, including public datasets like LAION[40], DataComp [12], DFN[8], and Wukong [14], supplemented by data from web search engines.”
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (2024, neutral): “With just two steps, Show-o can produce an image that is roughly related to the given prompt.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “Data Ratio Unlike previous work in VLM data curation [36, 129] which curate noisy raw image-text pairs by scraping the internet, Cambrian-10M is designed for visual instruction tuning.”
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models (2023, methodology): “When training contrastive vision-language models, filtering image-text pairs by CLIP cosine similarity has proven particularly helpful for improving data quality [31, 9].”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, background): “…(Schuhmann et al., 2021, 2022), Conceptual Captions (Sharma et al., 2018; Changpinyo et al., 2021), ALIGN (Jia et al., 2021), COYO (Byeon et al., 2022), and DataComp (Gadre et al., 2023), contain billions of image-text pairs and are usually obtained through web-crawling and alt-text extraction.”