What matters when building vision-language models?

By Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh. Neural Information Processing Systems, 2024

379 citations recorded by the source corpus.

The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture choice, data, and training methods. Our consolidation of findings includes the development of Idefics2, an efficient foundational VLM of 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks, and is often on par with models four times its size. We release the model (base, instructed, and chat) along with the datasets created for its training.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ce68430823b79dd3d478c505cc2761f03cf72b30

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yoshua Bengio
2. Bo Li
3. Saining Xie
4. Quoc V. Le
5. Yann LeCun
6. Trevor Darrell
7. Alan L. Yuille
8. Mohit Bansal

Citations in the corpus

• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, background): “The ICL performance is typically considered to evaluate multimodal pre-training, which has been adopted in several known LMMs, such as OpenFlamingo [2], IDEFICS series [22, 23], VILA [33] and MM1 [41], Emu2 [51].”
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, background): “Flamingo (Alayrac et al., 2022) demonstrates limited in-context learning capabilities, while Idefics2 (Laurençon et al., 2024) has acquired a broader multi-image understanding ability through multi-image training data.”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “For comprehensive investigations on design choices of LMMs, we refer to several recent studies [51, 63, 64, 104, 133, 10].”
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “We compare our method with two strong baselines: 1) concatenation-based [117] and 2) Re-sampler [10, 65], which utilizes a similar cross-attention form but lacks both spatial inductive biases and multi-layer vision aggregation.”
• ModernVBERT: Towards Smaller Visual Document Retrievers (2025, unknown): “Models are modality aligned on a large corpus in large parts derived from The Cauldron 2 (Laurenc¸on et al., 2024c) and Docmatix (Laurenc¸on et al., 2024a).”