Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners

By Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, Peng Gao. 2023

245 citations recorded by the source corpus.

Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. We then question, if the more diverse pre-training knowledge can be cascaded to further assist few-shot representation learning. In this paper, we propose CaFo, a Cascade of Foundation models that incorporates diverse prior knowledge of various pretraining paradigms for better few-shot learning. Our CaFo incorporates CLIP's language-contrastive knowledge, DINO's vision-contrastive knowledge, DALL-E's vision-generative knowledge, and GPT-3's language-generative knowledge. Specifically, CaFo works by ‘Prompt, Generate, then Cache’. Firstly, we leverage GPT-3 to produce textual inputs for prompting CLIP with rich downstream linguistic semantics. Then, we generate synthetic images via DALL-E to expand the few-shot training data without any manpower. At last, we introduce a learnable cache model to adaptively blend the predictions from CLIP and DINO. By such collaboration, CaFo can fully unleash the potential of different pre-training methods and unify them to perform state-of-the-art for few-shot classification. Code is available at https://github.com/ZrrSkywalker/CaFo.

Subfields: Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

1. Yu Qiao
2. Peng Gao
3. Ziyi Lin
4. Renrui Zhang
5. Shilin Yan
6. Pan Lu
7. Hongsheng Li
8. Junting Pan

Citations in the corpus

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “CLIP-Adapter [8], Tip-Adapter [55, 57] and PointCLIP [56,60] introduce customized adapters upon CLIP for 2D and 3D few-shot learning.”
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “We will also combine CLIP-Adapter with cache models [72, 77] and enhanced prompts [73] to further unleash the power of CLIP backbone.”
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, unknown): “Prompt tuning methods [2, 55, 63] can efficiently adapt VFMs towards dedicated scenarios by designing suitable prompts.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “semantics refiner [226], [234], LLMs significantly facilitates the progress of visual reasoning research.”
• Personalize Segment Anything Model with One Shot (2023, background): “CaFo [59] cascades different foundation models and collaborates their pre-trained knowledge for zero-shot image classification.”
• A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Pre-training allows the models to align different modalities and enhance the representation learning ability of the model encoder.”
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “Multi-modal Large Language Models (MLLMs), building upon the prevalence of Large Language Models (LLMs) [55, 56, 45, 28, 4] and large vision models [49, 29, 68, 67, 69], have become increasingly prominent in the field.”
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “Many successful PEFT approaches [10, 17, 22, 28, 31, 59, 60, 71, 73] have been proposed for adopting popular pre-trained models such as BERT [11], GPT [4, 53, 54], ViT [13], CLIP [52], and Stable Diffusion [55] to various downstream tasks.”