Sigmoid Loss for Language Image Pre-Training

By Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. 2023

3,447 citations recorded by the source corpus.

We propose a simple pairwise sigmoid loss for imagetext pre-training. Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. With only four TPUv4 chips, we can train a Base CLIP model at 4k batch size and a Large LiT model at 20k batch size, the latter achieves 84.5% ImageNet zero-shot accuracy in two days. This disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Mohit Bansal
  3. Yoshua Bengio
  4. Quoc V. Le
  5. Percy Liang
  6. Jianfeng Gao
  7. Luke S. Zettlemoyer
  8. Ruslan Salakhutdinov

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Percy Liang
  3. Tim Dettmers
  4. Neil Houlsby
  5. Ping Luo
  6. Armand Joulin
  7. Oriol Vinyals
  8. Zhilin Yang

In Multimodal Learning

  1. Saining Xie
  2. Neil Houlsby
  3. Antoine Miech
  4. Ali Farhadi
  5. Percy Liang
  6. Jean-Baptiste Alayrac
  7. Ping Luo
  8. Bo Li

Citations in the corpus