Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

By Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan and 3 more. Neural Information Processing Systems, 2024

900 citations recorded by the source corpus.

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/94773f22b5befd0e167a7de525d29bec2b09937a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…V3Det [243], TolokaVQA [236] Document DocReason25K [93], DocVQA [184], Docmatix [121], Synthetic Arxiv QA ALLaVA [25], SVIT [309], Cambrain-GPT4o [234], TextOCR-GPT4V [102], MMDU [159], Conversation Synthetic Real-World Conversations PMC-VQA [303], VQA-RAD [120], ImageCLEF [72], SLAKE [145],…”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, neutral): “Finally we include a diverse set of web data involving image captioning (CapsFusion [87], COCO [12]), question answering (Cambrian-7M [77], PixMo [19], VQAv2 [32]), and object localization in pre-training.”
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (2025, positive): “Prior efforts to enhance VLMs’ visual perception include combining multiple visual encoders (Tong et al., 2024d; Kar et al., 2025; Tong et al., 2024a), curating high-quality SFT data (Chen et al., 2023; Liu et al., 2024; Tong et al., 2024a), and improving the SFT training recipe by unfreezing the…”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, neutral): “…recent benchmarks such as LAMM (Yin et al., 2023b), LVLM-eHub (Xu et al., 2023), SEED (Li et al., 2024a), MMBench (Liu et al., 2023d),CV-Bench (Tong et al., 2024a), MM-Vet (Yu et al., 2024), Mantis (Jiang et al., 2024), and BLINK (Fu et al., 2024) have emerged, covering aspects from basic…”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, positive): “…since we only change the vision encoder, we provide evidence that a video encoder trained without language supervision can outperform encoders trained with language supervision, in contrast to conventional wisdom (Tong et al., 2024; Li et al., 2024b; Liu et al., 2024d; Yuan et al., 2025).”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “By exploring different vision encoders, Cambrian (Tong et al., 2024) finds that SigLIP (Zhai et al., 2023) is a strong language-supervised model and DINOv2 (Oquab et al., 2023) performs well on vision-centric tasks.”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “For benchmarks like VideoMME (Fu et al., 2024a) and LongVideoBench (Wu et al., 2024), with longer video durations up to one hour, mPLUG-Owl3 significantly outperforms existing models.”
Kimi-VL Technical Report (2025, neutral): “2.”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “For comprehensive investigations on design choices of LMMs, we refer to several recent studies [51, 63, 64, 104, 133, 10].”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “Multimodal large language models (MLLMs) have achieved significant progress in vision-language understanding [1, 7, 13, 16, 19, 34, 40, 46].”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “General LLaVA-SFT-665K [38], LLaVA-OV-SI [28], Cambrian-cleaned [39], Pixmo (docs, cap, points, cap-qa, ask-model-anything) [35] 9.87M Document DocVQA [40], Docmatix [41] 1.31M Chart/Figure ChartQA [42], MMC_Instruction [83], DVQA [84], LRV_Instruction [85], ChartGemma [86], InfoVQA [87], PlotQA…”
Mieb: Massive Image Embedding Benchmark (2025, unknown): “Vision-centric question answering Inspired by insights from MLLMs [54], we include vision centric question answering tasks, including object counting, spatial relationships, etc.”