Multimodal Few-Shot Learning with Frozen Language Models

By M. Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill, Zacharias Janssen. 2021

990 citations recorded by the source corpus.

When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of multiple interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, background): “Prompt-tuning [44], [45] may work particularly well with representations of this kind, potentially enabling more rapid adaptation to new tasks [46].”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “In models that condition on both text and image data, several previous approaches have found that image representations can be injected directly into language models’ embedding layers (Tsimpoukelli et al., 2021).”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “, 2021), and even as valuable priors when applied to diverse tasks from different modalities (Lu et al., 2021; Tsimpoukelli et al., 2021).”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, weights associated with one but not other modalities, frozen for downstream tasks (Zhai et al., 2021; Florence et al., 2019; Tsimpoukelli et al., 2021; Zakka et al., 2022).”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “Frozen [49] achieves this by jointly training an NF-ResNet-50 [3] and frozen GPT-3 with the Conceptual Captioning dataset [44].”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “We train on the dataset detailed in Section 3.3, and observe that evaluation loss does not plateau after ∼ 3M samples as reported in Frozen [52], and so continue training, resulting in two model variants – MAGMA base trained for 15000 steps in order to remain comparable to Frozen , and MAGMA long ,…”
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “Recently, Frozen (Tsimpoukelli et al. 2021) is proposed to extend such few-shot abilities to vision-and-language tasks by reusing a pre-trained language model.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “Motivated by recent works (Radford et al., 2021; Ramesh et al., 2021; Jia et al., 2021; Tsimpoukelli et al., 2021) that illustrate zero-shot learning in certain image-text tasks, we train our model using large-scale weakly labeled data only.”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “Frozen [43] fine-tunes an image encoder to transform visual tokens into LLM’s soft prompts.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “With the increased number of model parameters, it is hard to train those pretrained LLMs, thus more efforts have been paid to directly leverage the off-the-shelf frozen pre-trained LLMs for VL tasks [12, 38, 8, 46, 50].”
PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “Training such encodings can be understood as a form of input-conditioned soft-prompting (Tsimpoukelli et al., 2021), in relation to normal soft prompts (Lester et al.”
Language Is Not All You Need: Aligning Perception with Language Models (2023, background): “In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, positive): “Frozen (Tsimpoukelli et al., 2021), Flamingo (Alayrac et al., 2022)) resort to an image-to-text generation loss, which we show is insufficient to bridge the modality gap.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “[16] used a LAION subset for MAGMA, a model generating text “answers” for image-question pairs; MAGMA achieves state of the art results on OKVQA metrics and outperforming Frozen [80].”
ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering (2024, background): “…proven to possess the capability to solve a wide array of tasks, shining across various domains such as text (Brown et al., 2020; Chowdhery et al., 2023; Rae et al., 2021; Yu et al., 2022), images (Alayrac et al., 2022; Tsimpoukelli et al., 2021), and videos (Yang et al., 2022; Wang et al., 2022).”
CoCa: Contrastive Captioners are Image-Text Foundation Models (2022, background): “More recently, a line of work has also explored zero-shot/few-shot learning for vision-language tasks by re-using pretrained large language models (Yang et al., 2022b; Jin et al., 2021; Tsimpoukelli et al., 2021).”