Sigmoid Loss for Language Image Pre-Training
By Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer. 2023
3,447 citations recorded by the source corpus.
We propose a simple pairwise sigmoid loss for imagetext pre-training. Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. With only four TPUv4 chips, we can train a Base CLIP model at 4k batch size and a Large LiT model at 20k batch size, the latter achieves 84.5% ImageNet zero-shot accuracy in two days. This disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
- Luke S. Zettlemoyer
- Percy Liang
- Tim Dettmers
- Neil Houlsby
- Ping Luo
- Armand Joulin
- Oriol Vinyals
- Zhilin Yang
In Multimodal Learning
Citations in the corpus
- Stable and low-precision training for large-scale vision-language models (2023, background): “Loss spikes can emerge when scaling up models [8, 25, 14, 69, 71, 54, 74].”
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, methodology): “Notice that there has been significant and ongoing progress in scaling up both training data and model size for CLIP [10, 43, 54, 62, 66].”
- OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “Yet beyond robotics, existing foundation models for vision and language such as CLIP [8], SigLIP [9], and Llama 2 [10] are capable of these types of generalization and more, stemming from the priors captured by their Internet-scale pretraining datasets.”
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (2024, positive): “To extract compact representations, we use an image-text-aligned pre-trained vision encoder, SigLIP (Zhai et al., 2023).”
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, positive): “OpenVLA has a fused vision encoder with both SigLIP [52] and DINOv2 [34] vision transformers, and we apply FiLM to both transformers.”
- LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “For instance, employing the SigLIP (Zhai et al., 2023) encoder for a video with T = 100 results in 67,600 tokens, assuming M = 729 tokens per frame, which often exceeds GPU memory limits.”
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “For multimodal understanding, we use the SigLIP [89] encoder to extract high-dimensional semantic features from images.”
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “CLIP [51] and SigLIP [70] are widely-used for image encoding, while LLaMA [56] and Vicuna [12] serve as popular choices for LLMs.”
- Kimi-VL Technical Report (2025, positive): “MoonViT is initialized from and continually pre-trained on SigLIP-SO-400M (Zhai et al. 2023), which originally employs learnable fixed-size absolute positional embeddings to encode spatial information.”
- SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, positive): “2, we first employ SigLIP [68] visual encoder to extract 2D semantic visual features X ∈ R d × h × w to inherit the alignment between vision and language, and calculate their corresponding 3D positions P ∈ R 3 × h × w in the egocentric 3D coordinate system.”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “For instance, many pioneering frameworks such as LLaVA [75] use vision transformer-based CLIP models [102, 136], which are strongly supervised by language 1 , as the vision feature extractor.”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…been made to address these challenges, including enhancing model architectures [220, 232, 5, 172, 157, 210], scaling vision encoders [252, 66, 36, 293, 185] and language models [231, 235, 64, 19, 229, 221, 62], incorporating more diverse and high-quality datasets [124, 234, 25, 155], and…”
- Scaling Open-Vocabulary Object Detection (2023, methodology): “Since the G/14 model uses a different backbone than our other models (SigLIP [37] instead of CLIP [24]), we do not include it in Figure 4, but show in Table 1 that it is currently the best-performing model on zero-shot LVIS, with 47.2% mAP rare .”
- Video Instruction Tuning With Synthetic Data (2024, positive): “For instance, employing the SigLIP (Zhai et al., 2023) encoder for a video with T = 100 results in 67,600 tokens, assuming M = 729 tokens per frame, which often exceeds GPU memory limits.”
- UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, positive): “The architecture integrates a fused visual encoder derived from SigLip [90] and DINOv2 [62], a projection layer to align visual embeddings with the language modality, and the LLaMA-2 large language model (LLM) [75].”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “These include a hierarchical decomposition of robot learning, a multimodal transformer, and discrete latent plans.”