A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models

By Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren. 2021

176 citations recorded by the source corpus.

Large pre-trained vision-language (VL) models can learn a new task with a handful of examples and generalize to a new task without fine-tuning.However, these VL models are hard to deploy for real-world applications due to their impractically huge sizes and slow inference speed.To solve this limitation, we study prompt-based low-resource learning of VL tasks with our proposed method, FewVLM, relatively smaller than recent few-shot learners.For FewVLM, we pre-train a sequence-to-sequence transformer model with prefix language modeling (PrefixLM) and masked language modeling (MaskedLM).Furthermore, we analyze the effect of diverse prompts for few-shot tasks.Experimental results on VQA show that FewVLM with prompt-based learning outperforms Frozen which is 31x larger than FewVLM by 18.2% point and achieves comparable results to a 246x larger model, PICa.In our analysis, we observe that (1) prompts significantly affect zero-shot performance but marginally affect few-shot performance, (2) models with noisy prompts learn as quickly as hand-crafted prompts given larger training data, and (3) MaskedLM helps VQA tasks while PrefixLM boosts captioning performance. Our code is publicly available at https://github.com/woojeongjin/FewVLM

Subfields: Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, background): “More recently, a line of work has also explored zero-shot/few-shot learning for vision-language tasks by re-using pretrained large language models (Yang et al., 2022b; Jin et al., 2021; Tsimpoukelli et al., 2021).”
Language Models are General-Purpose Interfaces (2022, methodology): “For NoCaps and Flickr30k, following (Jin et al., 2022), we evaluate on their validation set and test set, respectively.”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, methodology): “FewVLM (Jin et al., 2022) ﬁnetunes the PLM using the preﬁx language modeling and masked language modeling objectives.”
Kosmos-2: Grounding Multimodal Large Language Models to the World (2023, unknown): “FewVLM (Jin et al., 2022) 31.0 -”
Modular Visual Question Answering via Code Generation (2023, methodology): “FewVLM randomly samples 16 few-shot examples for GQA.”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, unknown): “Hence, when hundreds of training examples are available, the input format proved less relevant, and the LLM was able to adapt (Jin et al., 2022).”
Making Language Models Better Reasoners with Step-Aware Verifier (2022, background): “…pretrained language models (PLMs) have shown remarkable performance on various natural language processing tasks, either by few-shot learning with prompts (Radford et al., 2019; Le Scao and Rush, 2021; Jin et al., 2022) or by fine-tuning (Houlsby et al., 2019; Hu et al., 2021; He et al., 2022).”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “[1] and [32]), it isn’t entirely obvious that this is still the case when compared with other in-context learning methods.”
On the Advance of Making Language Models Better Reasoners (2022, background): “Large-scale pretrained language models have recently exhibit exceptional performance in either few-shot learning with prompt (Radford et al., 2019; Le Scao and Rush, 2021; Jin et al., 2022) or ﬁne-tuning (Houlsby et al., 2019; Hu et al., 2021; He et al., 2022).”
MaPLe: Multi-modal Prompt Learning (2022, background): “Designing high quality contextual prompts have been proven to enhance the performance of CLIP and other V-L models [17, 42].”
Prompt-aligned Gradient for Prompt Tuning (2022, background): “Thanks to the powerful language modeling of VLMs, we can establish a communication channel in human-readable natural language, i.e ., prompt [25, 51, 19], to query the general knowledge.”
MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting (2022, methodology): “, 2021) or encoder-decoder methods (Cho et al., 2021; Wang et al., 2021; Jin et al., 2022; Li et al., 2021) and use transformer-based architectures.”
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022, background): “Although it lacks a rigorously theoretical proof, prompt learning inherits the few-shot property [Li and Liang, 2021; Schick and Schütze, 2020; Jin et al., 2021; Liu et al., 2021].”