CLIP-Adapter: Better Vision-Language Models with Feature Adapters

By Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao. 2021

1,792 citations recorded by the source corpus.

Large-scale contrastive vision-language pretraining has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in Radford et al. (International conference on machine learning, PMLR, 2021) to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions. To avoid non-trivial prompt engineering, context optimization (Zhou et al. in Int J Comput Vis 130(9):2337–2348, 2022) has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples. In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning. While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pretrained features. As a consequence, CLIP-Adapter is able to outperform context optimization while maintaining a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Recent work has explored alternative strategies for adapting these models to specific target tasks (Zhou et al., 2021; Gao et al., 2021; Zhang et al., 2021), for instance via a lightweight residual feature adapter.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, background): “[59, 66] further apply a residual feature adapter to improve the few-shot performance.”
Learning to Prompt for Vision-Language Models (2021, background): “It would also be interesting to investigate more generic adaptation methods for mega-size vision models (Jia et al., 2022; Bahng et al., 2022; Gao et al., 2021).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Adapters [22] insert lightweight adaption modules into each layer of the pre-trained transformer and have been extended across numerous domains [19, 18, 70, 71].”
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection (2023, background): “GLIP [11] formulates object detection as a grounding problem and leverages additional grounding data to help learn aligned semantics at phrase and region levels.”
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “It has inspired a few recent works for language-augmented visual models [75, 59, 70, 18].”
Grounded Language-Image Pre-training (2021, background): “Such a setting is often denoted as linear probing [21], prompt tuning [64], or efficient task adapters [11].”
Conditional Prompt Learning for Vision-Language Models (2022, background): “With the rise of such powerful vision-language models, the community has recently started to investigate potential solutions to efﬁciently adapt these models to downstream datasets [14, 53, 56, 63].”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, positive): “Furthermore, CoOp [70], CLIP-Adapter [17] and WiSE-FT [64] indicate that the performance of CLIP can be signiﬁcantly improved with either abundant or limited training data by ﬁne-tuning weight-ﬁxed CLIP with adapter or prompt optimization.”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “With the advent of CLIP [60], many CLIP-based adapters [24,69,81] were presented to transfer pre-trained knowledge to zero-shot or few-shot downstream tasks.”
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion (2022, background): “More measured approaches freeze the model and train transformation modules to adapt its output when faced with new concepts (Zhou et al., 2021; Gao et al., 2021; Skantze & Willemsen, 2022).”
GLIPv2: Unifying Localization and Vision-Language Understanding (2022, methodology): “GLIPv2-B/GLIPv2-H are based on Swin-Base/Swin-Huge and the pre-layernorm text transformer [17] as dual encoder, and are initialized from the UniCL [55] checkpoints.”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, methodology): “By following CLIP [22], we make predictions based on the cosine similarity between the final image embedding F I ( I ti ) and text embedding F T ( c tj ) .”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, methodology): “Referring to adapters [36] in natural language processing, CLIP-Adapter [22] is introduced to finetune CLIP by applying lightweight residual-style adapters.”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “For example, Clip-Adapter [33] inserts several trainable linear layers after CLIP’s language and image encoders and optimizes them while keeping CLIP architecture and parameters frozen as illustrated in Fig.”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, methodology): “On top of that, CoOp [74], CLIP-Adapter [16] and WiSE-FT [61] significantly improve the CLIP with limited training data by freezing the pre-trained weights and training additive learnable modules.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “For example, CLIP-Adapter [51] integrates residual-style adapters after CLIP’s text and visual encoders.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “After the aggregator, we insert two extra layers of residual CLIP Adapter [38].”