Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

By Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li. European Conference on Computer Vision, 2022

616 citations recorded by the source corpus.

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations using large-scale image-text pairs. It shows impressive performance on downstream tasks by zero-shot knowledge transfer. To further enhance CLIP's adaption capability, existing methods proposed to fine-tune additional learnable modules, which significantly improves the few-shot performance but introduces extra training time and computational resources. In this paper, we propose a training-free adaption method for CLIP to conduct few-shot classification, termed as Tip-Adapter, which not only inherits the training-free advantage of zero-shot CLIP but also performs comparably to those training-required approaches. Tip-Adapter constructs the adapter via a key-value cache model from the few-shot training set, and updates the prior knowledge encoded in CLIP by feature retrieval. On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient. We conduct extensive experiments of few-shot classification on 11 datasets to demonstrate the superiority of our proposed methods. Code is released at https://github.com/gaopengcuhk/Tip-Adapter.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a32887af7eb1fcfd3b3d892ad41f3516a37f11c1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, background): “In general, these PEFT approaches can be divided into three categories, namely preﬁx tuning ( e.g. [35,76]), reparameterization ( e.g. [10,26,43]), and adapters ( e.g. [15,24,74]).”
Is synthetic data from generative models ready for image recognition? (2022, methodology): “For datasets, we carefully select 8 image classiﬁcation datasets from recent works (Zhou et al., 2022b;a; Zhang et al., 2022) that conduct few-shot learning upon CLIP: Ima-geNet (Deng et al., 2009), Caltech101 (Fei-Fei et al., 2006), Pets (Parkhi et al., 2012), Cars (Krause et al., 2013), Aircraft…”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, methodology): “The cache model is only adopted during inference, and constructed in a training-free manner [94].”