Flamingo: a Visual Language Model for Few-Shot Learning
By Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds and 17 more. Neural Information Processing Systems, 2022
6,220 citations recorded by the source corpus.
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
View Paper: https://arxiv.org/abs/2204.14198
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Sergey Levine
- Luke S. Zettlemoyer
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
In Multimodal Learning
Citations in the corpus
- Emergent Abilities of Large Language Models (2022, background): “…PiQA question-answering benchmarks (Brown et al., 2020); PaLM 540B achieved new state of the art on three arithmetic reasoning benchmarks (Chowdhery et al., 2022); and the multimodal Flamingo 80B model achieved new state of the art on six visual question answering benchmarks (Alayrac et al., 2022).”
- Galactica: A Large Language Model for Science (2022, background): “Existing work such as Alayrac et al. (2022) has shown how to extend LLMs with this modality.”
- Faithful Reasoning Using Large Language Models (2022, background): “Language models are being applied, with great success, to many different problems Alayrac et 2022).”
- Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “In this case, the LLM can directly ask a question about the scene, and this question can be answered either by a person, or by another pretrained model, such as a Visual Question Answering (VQA) model [87, 88, 89, 90].”
- Survey of Hallucination in Natural Language Generation (2022, background): “Examples of hallucination in visual question answering (taken from [1]).”
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, 2022a), or Flamingo (Alayrac et al., 2022)), SMs may likewise benefit in the capacity to express details.”
- Visual Instruction Tuning (2023, background): “More sophisticated schemes to connect the image and language representations can also be considered, such as gated cross-attention in Flamingo [2] and Q-former in BLIP-2 [28].”
- Improved Baselines with Visual Instruction Tuning (2023, background): “Large multimodal models (LMMs) have become increasingly popular in the research community, as they are the key building blocks towards general-purpose assistants [2, 29, 42].”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Then, we present the details of zero-initialized attention mechanisms with zero gating in Section 3.2, and generalize LLaMA-Adapter for multi-modal reasoning in Section 3.3.”
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, background): “2 [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.”
- ViperGPT: Visual Inference via Python Execution for Reasoning (2023, methodology): “find("skyscraper") skyscraper_patch_1 = skyscraper_patches[0] skyscraper_patch_2 = skyscraper_patches[1] skyscraper_1_name = skyscraper_patch_1.”
- Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “Although several works have explored leveraging the pre-trained LLMs to improve the performance on the vision-language (VL) tasks, those methods supported several specific VL tasks (from language to version or from version to language) and required labeled data for training [38, 1, 22].”
- PaLM-E: An Embodied Multimodal Language Model (2023, neutral): “Task-specific finetuned models Flamingo (Alayrac et al., 2022) 82.”
- GLIGEN: Open-Set Grounded Text-to-Image Generation (2023, methodology): “During training, we gradually fuse the new grounding information into the pretrained model using a gated mechanism [1].”