Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

By Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li. 2021

591 citations recorded by the source corpus.

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose \textbf{T}raining-Free CL\textbf{IP}-\textbf{Adapter} (\textbf{Tip-Adapter}), which not only inherits CLIP's training-free advantage but also performs comparably or even better than CLIP-Adapter. Tip-Adapter does not require any back propagation for training the adapter, but creates the weights by a key-value cache model constructed from the few-shot training set. In this non-parametric manner, Tip-Adapter acquires well-performed adapter weights without any training, which is both efficient and effective. Moreover, the performance of Tip-Adapter can be further boosted by fine-tuning such properly initialized adapter for only a few epochs with super-fast convergence speed. We conduct extensive experiments of few-shot classification on ImageNet and other 10 datasets to demonstrate the superiority of proposed Tip-Adapter. The code will be released at \url{https://github.com/gaopengcuhk/Tip-Adapter}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Recent work has explored alternative strategies for adapting these models to specific target tasks (Zhou et al., 2021; Gao et al., 2021; Zhang et al., 2021), for instance via a lightweight residual feature adapter.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, neutral): “Some concurrent works also extend parameter-efﬁcient techniques to CLIP models [2,54,55].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Adapters [22] insert lightweight adaption modules into each layer of the pre-trained transformer and have been extended across numerous domains [19, 18, 70, 71].”
Grounded Language-Image Pre-training (2021, neutral): “This stands in contrast to recent work that investigates prompt tuning only for shallow-fused vision-language models such as CLIP [13,71,72].”
ImageBind-LLM: Multi-modality Instruction Tuning (2023, methodology): “The cache model contains millions of image features in the training datasets extracted by ImageBind, which improves text/audio/3D/video embeddings by retrieving similar visual features, referring to Tip-Adapter [14].”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “For example, CoOp [61], CLIP-Adapter [16] and Tip-Adapter [58] use the pretrained CLIP for improving the few-shot transfer, while PointCLIP [59] and DenseCLIP [38,60] transfer the knowledge to point cloud understanding and dense prediction, respectively.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “• Vision Adapter and Prompting Modeling: Following the idea of prompt tuning in NLP, early works [265], [266] adopt learnable parameters with the frozen foundation models to better transfer the downstream datasets.”
Exploring CLIP for Assessing the Look and Feel of Images (2022, methodology): “We believe that the performance of CLIP-IQA + can be further improved by replacing CoOp with more sophisticated ﬁne-tuning schemes such as Tip-Adaptor (Zhang et al. 2021).”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “, prompt tuning [38], [39], visual adaptation [40], [41], etc.”
PointCLIP: Point Cloud Understanding by CLIP (2021, background): “CLIP-Adapter [16], Tip-Adapter [63], Action-CLIP [53] and WiSE-FT [56] further indicate that the performance of CLIP can be largely improved by infusing domain-speciﬁc supervisions.”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, methodology): “Referring to Tip-Adapter [75], we build the cache model with two kinds of keys respectively for the two pre-trained models.”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, background): “Recently, Parameter-Efficient Fine-Tune (PEFT) meth-ods [22, 28, 65, 73, 76] have demonstrated that large-scale models can quickly adapt to downstream tasks via only fine-tuning less-parameterized adapters.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Tip-Adapter [218] adopts the same design with CLIP-Adapter.”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, unknown): “Feature adaptation methods [5, 16, 33, 62, 64, 65] can further pivot VFMs towards different tasks.”