VILA: On Pre-training for Visual Language Models

By Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, Song Han. Computer Vision and Pattern Recognition, 2023

896 citations recorded by the source corpus.

Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training data is beneficial whereas image-text pairs alone are not optimal; (3) re-blending text-only instruction data to image-text data during instruction fine-tuning not only remedies the degradation of text-only tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe we build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles. Multi-modal pre-training also helps unveil appealing properties of VILA, including multi-image reasoning, enhanced in-context learning, and better world knowledge. VILA is also deployable on Jetson Orin for on-device VLM.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2141ed804636a1cf339d606cd03fd3b3e9582133

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “This pipeline has inspired subsequent researches (Li et al., 2024c;b;a; Lin et al., 2024) aimed at generating diverse image-language instruction data across various visual domains, accelerating the development of visual instruction tuning techniques.”
Video Instruction Tuning With Synthetic Data (2024, positive): “This pipeline has inspired subsequent researches (Li et al., 2024c;b;a; Lin et al., 2024) aimed at generating diverse image-language instruction data across various visual domains, accelerating the development of visual instruction tuning techniques.”
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, neutral): “VILA [23] proposes more advanced training recipes.”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, neutral): “5 (Lin et al., 2023b mPLUG-Owl3 outperforms 8B-level language models in VQAv2, OK-VQA, GQA, and VizWizQA.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “In recent years, multimodal large language models (MLLMs) [60, 137, 246, 36, 35, 248, 140, 228, 192, 275, 143, 54, 170] have emerged as a pivotal technology in artificial intelligence, capable of processing and understanding information from multiple modalities such as text, images, and videos.”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “Multimodal large language models (MLLMs) [32, 66, 121, 21, 19, 123, 68, 114, 97, 136, 71, 31, 85, 117, 18, 89, 105, 69] have recently achieved or even surpassed human-level performance in a broad spectrum of tasks, underscoring their potential as a significant stride toward artificial general…”
MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, neutral): “…capabilities (Liu et al., 2023c,b, 2024a; Li et al., 2024b; Dai et al., 2023; Zhu et al., 2023; Zhang et al., 2023; Gao et al., 2023; Ye et al., 2023a,b; Zhao et al., 2023; Li et al., 2023; Monajatipoor et al., 2023; Zhao et al., 2024; Li et al., 2024c; Lin et al., 2024; Zhang et al., 2024).”
DeepSeek-VL: Towards Real-World Vision-Language Understanding (2024, neutral): “• While some models (01-ai, 2024; Lin et al., 2023a; Sun et al., 2023; Wang et al., 2023b) have begun to exploit pretraining, they often overlook the preservation of language skills.”
LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “5-40B [32].”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “5 [34], attains an overall accuracy of 59%.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “Note that on many VLM benchmarks, increased resolution does improve performance [44, 86, 87], but we did not see this trend (yet) for VLAs.”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “Other versatial open LMMs with potentials to excel include VILA [77], InternLM-XComposer-2.”
What matters when building vision-language models? (2024, neutral): “By releasing our findings, as well as our models and our training dataset, we aim to contribute to the ongoing evolution of VLMs and their applications in solving complex real-world problems.”