SLIP: Self-supervision meets Language-Image Pre-training

By Norman Mu, Alexander Kirillov, David A. Wagner, Saining Xie. 2021

639 citations recorded by the source corpus.

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across ImageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Hierarchical Text-Conditional Image Generation with CLIP Latents (2022, background): “Recent progress in computer vision has been driven by scaling models on large datasets of captioned images collected from the internet [10, 44, 60, 39, 31, 16].”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “Recently, image-text pre-training has become increasingly popular in computer vision as a pre-training task (Radford et al., 2021; Jia et al., 2021; Mu et al., 2021; Pham et al., 2021; Yu et al., 2022).”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “The success has quickly inspired many follow-up large-scale pre-training works [68, 72, 69, 36, 43, 20, 34, 74].”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, background): “Relevant concurrent work including: SLIP Mu et al. (2021) introduces self-supervision to CLIP.”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, methodology): “We evaluate our SimTrack when using ViT-B/16 as backbone and initializing the backbone with parameters pre-trained with several recent methods, including DeiT [35], MOCO-V3 [8], SLIP [28], and CLIP [31].”
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (2024, background): “SLIP [39] explores the synergy between CLIP and contrastive SSL, but focusing primarily on standard classiﬁcation tasks.”
Scaling Language-Image Pre-Training via Masking (2022, methodology): “7 We follow the data preparation scripts provided by [25] and [46] and load data using Tensorflow Datasets.”
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Recent advancements in CLIP models have primarily concentrated on scaling the model size and data size [53], incorporating self-supervision [54], [55], [56], enhancing pretraining efficiency [57], [58], and few-shot adaptation [59], [60], among others.”
Improving CLIP Training with Language Rewrites (2023, methodology): “Follow-up methods that build upon language-image pre-training, such as SLIP [41] and FLIP [35], exhibit similar advantageous scaling performance.”
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (2022, background): “This has inspired many follow-up works to further enhance the pre-training [40, 78, 48, 75], among which GLIP [37] shows that OD can also be cast as a VL task (i.”
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond (2022, background): “Other works extend of CLIP by adding self-supervision [95], data scaling [96] or enabling flexibility to the encoders [97].”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, methodology): “For the cache model, we investigate other pre-trained foundation models besides CLIP and DINO [2], including SimCLR [3], MAE [4], and SLIP [5].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “75 SLIP [185] arXiv-2021 image-text ViT CS, InfoNCE Combine the self-supervised learning and CLIP pre-training in a multi-task framework 38M URL”
K-LITE: Learning Transferable Visual Models with External Knowledge (2022, background): “Another line of work focuses on task-level zero-shot transfer [52, 46, 64, 71, 36, 97].”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, methodology): “All the models are pre-trained on YFCC15M-V1 for 32 epochs, except SLIP [32] for 100 epochs.”
ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding (2022, methodology): “During pre-training, we utilize an advanced version of CLIP, namely SLIP [32], that shows superior performance as our image-text encoders.”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “[10], [18], [64] modify ViT by adding a normalization layer before the transformer encoder.”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, background): “These models [30, 53, 57], trained on Internet-scale image-text pairs, have shown strong openvocabulary classification capabilities.”