OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

By Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, K. Marathe, Yonatan Bitton, S. Gadre, Shiori Sagawa and 6 more. 2023

642 citations recorded by the source corpus.

We introduce OpenFlamingo, a family of autoregressive vision-language models ranging from 3B to 9B parameters. OpenFlamingo is an ongoing effort to produce an open-source replication of DeepMind's Flamingo models. On seven vision-language datasets, OpenFlamingo models average between 80 - 89% of corresponding Flamingo performance. This technical report describes our models, training data, hyperparameters, and evaluation suite. We share our models and code at https://github.com/mlfoundations/open_flamingo.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

What matters when building vision-language models? (2024, positive): “We evaluate Idefics2 on commonly adopted benchmarks: MMMU (Yue et al., 2024) for multidis-cipline college-level problems, MathVista (Lu et al., 2024) for mathematical reasoning, TextVQA (Singh et al., 2019)”
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (2023, methodology): “(vi) As an open-source alternative to Flamingo [2], OpenFlamingo [4] has close performance on most vision-language tasks.”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “Open-Flamingo [3], derived from Flamingo [1], and Otter [30], an in-context instruction tuning variant of OpenFlamingo, are also included in our comparison.”
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “Models are pretrained on large-scale image-text pairs [4, 11, 19, 42, 47] or interleaved data [2, 3, 16] to learn the semantic mapping between visual and text signals.”
PointLLM: Empowering Large Language Models to Understand Point Clouds (2023, methodology): “To rigorously evaluate our model’s perceptual abilities and its generalization capabilities, we establish two benchmarks: Generative 3D Object Classification and 3D Object Caption-ing, assessed through three different methods, including”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “…methods are primarily based on interleaved image-text web data for pre-training (Laurençon et al., 2023; Laurençon et al., 2024) to extend multi-image capabilities or focused on the in-context abilities (Alayrac et al., 2022; Awadalla et al., 2023; Zhao et al., 2023) within multi-image scenarios.”
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition (2023, background): “InstructBLIP [11], based on various image-text datasets, fine-tunes MLLMs by constructing prompt templates and introduces Q-Former [32] to associate more relevant image features in advance.”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…such as LLaVA, miniGPT4, InstructBLIP, Flamingo, Multimodal Bard (Liu et al., 2023a; Zhu et al., 2023a; Dai et al., 2023; Alayrac et al., 2022; Awadalla et al., 2023; Google, 2023) that are trained on vast amount of paired (Schuhmann et al., 2022; Sharma et al., 2018; Lin et al., 2014) and…”
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (2023, positive): “Recent open-sourced LLMs [85, 69, 60] also facilitate various research studies including OpenFlamingo [8], LLaVA [49], MiniGPT-4 [87], Otter [43], InstructBLIP [22], and so on [32, 50, 81].”
OCRBench: on the hidden mystery of OCR in large multimodal models (2023, positive): “Flamingo [13] and OpenFlamingo [19] enhance a frozen pretrained LLM by incorporating novel gated cross-attention-dense layers, enabling conditioning on visual inputs.”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, positive): “We evaluate LLaVA-13B [13], InstructBLIP-13B [14], MiniGPT4-7B [41], mPLUG-Owl-7B [17], LlamaAdapter-v2-7B [16], PandaGPT-13B [18], VisualChatGPT [42], Multimodal GPT [43], Open-Flamingo v1 [11, 44] and Otter v1 [45].”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “The alignment module varies from simple linear projections [35] to more complex architectures such as Q-Former [13, 25], and gated cross-attention layers substantiated by Flamingo [1, 4].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, methodology): “Multimodal Dialogue In Tables 11 and 12, we present a qualitative comparative analysis of VQA results between our model, D REAM LLM, and other state-of-the-art models: GPT-4 (OpenAI, 2023a;b), LLaVA (Liu et al., 2023c), BLIP-2 (Li et al., 2022), and OpenFlamingo (Awadalla et al., 2023).”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, background): “…improved instruction-following capabilities (Liu et al., 2023c,b, 2024a; Li et al., 2024b; Dai et al., 2023; Zhu et al., 2023; Zhang et al., 2023; Gao et al., 2023; Ye et al., 2023a,b; Zhao et al., 2023; Li et al., 2023; Monajatipoor et al., 2023; Zhao et al., 2024; Li et al., 2024c; Lin et al.,…”
Foundation Models Defining a New Era in Vision: A Survey and Outlook (2025, neutral): “OpenFlamingo [106] is an open-source version of the Flamingo model based on the same architecture but with a new Multimodal C4 dataset and 10M samples from LAION-2B.”
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation (2024, neutral): “Compared with previous MLLMs, our instruction tuned model can generate images that are more aligned with the elements in the descriptive caption and possess artistic qualities.”