Flamingo: a Visual Language Model for Few-Shot Learning

By Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds and 17 more. Neural Information Processing Systems, 2022

6,220 citations recorded by the source corpus.

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

View Paper: https://arxiv.org/abs/2204.14198

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Percy Liang
  4. Yoshua Bengio
  5. Quoc V. Le
  6. Sergey Levine
  7. Luke S. Zettlemoyer
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Ming-Hsuan Yang
  5. Xiaogang Wang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Cordelia Schmid

In Large Language Models

  1. Furu Wei
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Mohit Bansal
  5. Li Dong
  6. Noah A. Smith
  7. Hinrich Schütze
  8. Yi Tay

In Multimodal Learning

  1. Zhe Gan
  2. Jianfeng Gao
  3. Lijuan Wang
  4. Linjie Li
  5. Zicheng Liu
  6. Mohit Bansal
  7. Cordelia Schmid
  8. Jianfeng Wang

Citations in the corpus