Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

By Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fisher Yu, Junjie Yan. 2021

583 citations recorded by the source corpus.

Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De-CLIP can learn generic visual features more efficiently. Instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighbor supervision from other similar pairs. Benefiting from intrinsic supervision, our DeCLIP-ResNet50 can achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1 x fewer data. Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks. Moreover, Scaling up the model and computing also works well in our framework.Our code, dataset and models are released at: https://github.com/Sense-GVT/DeCLIP

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “Concurrently to this work, DeCLIP [40] adds several additional training objectives and more data collected in-house to CLIP in order to improve data efﬁciency.”
Learning to Prompt for Vision-Language Models (2021, background): “By pre-training at a large scale, models can learn diverse visual concepts and can readily be transferred to any downstream task through prompting (Radford et al., 2021; Jia et al., 2021; Fürst et al., 2021; Li et al., 2021; Singh et al., 2021; Yuan et al., 2021).”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “The success has quickly inspired many follow-up large-scale pre-training works [68, 72, 69, 36, 43, 20, 34, 74].”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, methodology): “Recently, CLIP [48], DeCLIP [36] and ALIGN [28] demonstrate that learning from simple contrastive vision-language pairs obtains promising transferable features for zero-shot recognition over diverse datasets.”
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text (2022, unknown): “Though remedies like data augmentation (Li et al., 2021) and knowledge graph (Shen et al., 2022) were proposed, the magnitude of used data is still far larger than medical data.”
Improving CLIP Training with Language Rewrites (2023, background): “SLIP [41] and De-CLIP [34] proposes to improve the performance by incorporating self-supervised training techniques.”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, background): “There are an increasing number of studies to improve their generality from various modeling perspectives, including training objectives [11,13,14,38,65,68], scaling techniques [9,40,65], data efficiency [25, 28], and leveraging multilingual correlations [9, 19].”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “Contrastive learning has been widely explored in VLM pre-training, which designs contrastive objectives for learning discriminative image-text features [10], [64], [113].”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Vision-Language alignment models (VLA), such as CLIP [149], ALIGN [83], DeCLIP [107], and FLAVA [162], are designed to learn a good image and text features which can be aligned within a unified representation space.”
PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining (2022, methodology): “YFCC15M-V1 is obtained by applying the same filtering rule on YFCC100M as CLIP [6], while YFCC15M-V2 is collected by DeCLIP [10] with a different filtering strategy.”