Reproducible Scaling Laws for Contrastive Language-Image Learning
By Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev. 2022
1,494 citations recorded by the source corpus.
Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data & models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study is available at https://github.eom/LAION-AI/sealing-laws-openelip.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Yoshua Bengio
- Luke S. Zettlemoyer
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Sergey Levine
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Andrea Vedaldi
- Andrew Zisserman
In Multimodal Learning
- Ludwig Schmidt
- Luke S. Zettlemoyer
- Armand Joulin
- Rohit Girdhar
- Ali Farhadi
- Hannaneh Hajishirzi
- Ishan Misra
- Tim Dockhorn
In Representation & Self-Supervised Learning
- Ishan Misra
- Yann LeCun
- Armand Joulin
- Piotr Bojanowski
- Julien Mairal
- Andrea Vedaldi
- Jean Ponce
- Herv'e J'egou
In Scaling & Learning Theory
- Julien Mairal
- Saining Xie
- Micah Goldblum
- Ali Farhadi
- Aleksander Madry
- Jean Ponce
- Adrien Bardes
- Herv'e J'egou
Citations in the corpus
- Stable and low-precision training for large-scale vision-language models (2023, methodology): “[9] when training ViT-Huge CLIP models and remedied by switching to bfloat16.”
- Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, methodology): “Baselines: We provide results using two strong discriminative zero-shot models: (a) CLIP ResNet-50 [59] and (b) OpenCLIP ViT-H/14 [11].”
- DataComp: In search of the next generation of multimodal datasets (2023, background): “In numerous areas of machine learning, larger datasets lead to better performance [80, 71, 65, 100, 60, 24, 15, 104, 105].”
- Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “Cherti et al. [74] supplement this open-source effort by studying the scaling laws of CLIP.”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “…and group them into four categories: language-supervised models (i.e., OpenAI CLIP [102], SigLIP [136], DFN-CLIP [34], EVA-CLIP [113] and OpenCLIP [28]), self-supervised models (i.e., DINOv2 [96], I-JEPA [8], MAE [45], MoCo v3 [24]), class-supervised models (ImagetNet22K ViT [33]) and other…”
- EVA-02: A Visual Representation for Neon Genesis (2023, background): “[30] also reports a small percentage of images from IN-1K along with its variants, Flickr30K and COCO were detected in the LAION-400M dataset.”
- Vision-Language Models for Vision Tasks: A Survey (2023, background): “As discussed above, although VLMs benefit clearly while data/model size scales up, they still suffer from several limitations: (1) When data/model size keeps increasing, the performance saturates and further scaling up won’t improve performance [113], [202]”
- Scaling Laws for Generative Mixed-Modal Language Models (2023, background): “Cherti et al. (2022) also examined multi-modal training but did not specifically focus on generative models.”
- RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Recent advancements in CLIP models have primarily concentrated on scaling the model size and data size [53], incorporating self-supervision [54], [55], [56], enhancing pretraining efficiency [57], [58], and few-shot adaptation [59], [60], among others.”
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (2024, methodology): “…in Figure 5, the additional normalization prevents the attention logit growth instability, confirming findings by Dehghani et al. (2023) and Wortsman et al. (2023) and enables efficient training at bf16-mixed (Chen et al., 2019) precision when combined with ϵ = 10 − 15 in the AdamW…”
- VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, methodology): “The image embedding is extracted using CLIP [14] and injected into the SD UNet through cross attention [54], similar to the injection of text embeddings.”
- Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (2023, methodology): “We choose Stable Diffusion-v2 (Rombach et al., 2022) as our generative model, and the open-source CLIP model, OpenCLIP-ViT/H (Cherti et al., 2022) for crafting the prompt, as both share the same text encoder.”
- ImageBind One Embedding Space to Bind Them All (2023, methodology): “Specifically, we use the pretrained vision (ViT-H 630M params) and text encoders (302M params) from OpenCLIP [11, 30].”
- DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data (2023, methodology): “LPIPS [94] and DISTS [24] use CNN backbones, whereas DINO [10], CLIP [66], OpenCLIP [17], and MAE [31] use transformer-based backbones [25].”
- LERF: Language Embedded Radiance Fields (2023, methodology): “We use the OpenClip [10] ViT-B/16 model trained on the LAION-2B dataset, with an image pyramid varying from s min = .”
- GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023, methodology): “This may be explained by the change in text encoder: SDv2 uses OpenCLIP ViT-H/14 [8], which is larger and trained on different data compared to the CLIP ViT-L/14 [31] used by SDv1.”
- TRAK: Attributing Model Behavior at Scale (2023, positive): “While the quality of these representations—as measured by aggregate metrics such as downstream zeroshot accuracy—appears to be driven largely by the properties and scale of the training datasets (Fang et al., 2022; Santurkar et al., 2022; Cherti et al., 2022), we lack a finegrained understanding of ”
- DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “Second, we report performance of open-source weakly-supervised models such as CLIP (Radford et al., 2021), OpenCLIP (Ilharco et al., 2021; Cherti et al., 2023), and SWAG (Singh et al., 2022).”