Reproducible Scaling Laws for Contrastive Language-Image Learning

By Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev. 2022

1,494 citations recorded by the source corpus.

Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data & models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study is available at https://github.eom/LAION-AI/sealing-laws-openelip.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Yoshua Bengio
  3. Luke S. Zettlemoyer
  4. Jianfeng Gao
  5. Quoc V. Le
  6. Percy Liang
  7. Sergey Levine
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Trevor Darrell
  5. Jitendra Malik
  6. Xiaogang Wang
  7. Andrea Vedaldi
  8. Andrew Zisserman

In Multimodal Learning

  1. Ludwig Schmidt
  2. Luke S. Zettlemoyer
  3. Armand Joulin
  4. Rohit Girdhar
  5. Ali Farhadi
  6. Hannaneh Hajishirzi
  7. Ishan Misra
  8. Tim Dockhorn

In Representation & Self-Supervised Learning

  1. Ishan Misra
  2. Yann LeCun
  3. Armand Joulin
  4. Piotr Bojanowski
  5. Julien Mairal
  6. Andrea Vedaldi
  7. Jean Ponce
  8. Herv'e J'egou

In Scaling & Learning Theory

  1. Julien Mairal
  2. Saining Xie
  3. Micah Goldblum
  4. Ali Farhadi
  5. Aleksander Madry
  6. Jean Ponce
  7. Adrien Bardes
  8. Herv'e J'egou

Citations in the corpus