Improving CLIP Training with Language Rewrites

By Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, Yonglong Tian. Neural Information Processing Systems, 2023

296 citations recorded by the source corpus.

Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d4af12327385260116dfd68ed1ec6d0602d26d1f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sheng Jin
2. Shijian Lu
3. Dahua Lin
4. Jingyi Zhang
5. Jiaxing Huang
6. Xiao-wen Dong
7. Jiaqi Wang
8. Jinsong Li

In Computer Vision and Pattern Recognition

1. Shijian Lu
2. Xiao-wen Dong
3. Jiaqi Wang
4. Sheng Jin
5. Pan Zhang
6. Jiaxing Huang
7. Jingyi Zhang
8. Conghui He

Citations in the corpus

• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions (2023, background): “LaCLIP [13] leverages LLMs to rewrite raw cap-tions, but its effectiveness is often hindered by hallucinations due to limited visual information and the low quality of original captions.”
• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For example, LA-CLIP [126] and ALIP [127] employ large language models to augment synthetic captions for given images while RA-CLIP [125] retrieves relevant image-text pairs for image-text pair augmentation.”
• Better Language Models Exhibit Higher Visual Alignment (2024, unknown): “Further analysis into the role of vision backbones and using more sophisticated projection network architectures can be found in Appendix D.”