LLaVA-OneVision: Easy Visual Task Transfer
By Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li. Trans. Mach. Learn. Res., 2024
2,742 citations recorded by the source corpus.
We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series. Our experimental results demonstrate that LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.
Subfields: Mechanical Engineering, Human-Computer Interaction
View Paper: https://www.semanticscholar.org/paper/1a71f7b216b710b936da666027014adb83af8e7a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Mechanical Engineering
In Human-Computer Interaction
Citations in the corpus
- Emu3: Next-Token Prediction is All You Need (2024, methodology): “In the second stage, we sample a subset of question-answer pairs from [44] to enhance the vision instruction following ability.”
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “To integrate vision as an input modality, numerous vision-language models have been developed, including the LLava series [LLWL24, LLL + 24, LZG + 24], QWenVL series [BBY + 23, WBT + 24], InternVL series [CWT + 24, CWW + 24, CWC + 24], InternLM-XComposer series [ZDW + 23, DZZ + 24b], Molmo [DCL +…”
- LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “We evaluated the performance of 13 models that support multi images or short video input: TimeChat [33], Video-ChatGPT [23], PLLaVA [43], LLaVA-OneVision [16], CogVLM2-Video [12], LLaVA-NeXT [50], InternVL2-40B [6], mPLUG-Owl3 [44], GLM-4.”
- R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization (2025, positive): “We use DeepSeek-R1 [8] to generate reasoning processes on LLaVA-OneVision [16] and collect these data into the R1-Onevision dataset.”
- MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “…removing the redundant texts and input diagrams on 120 randomly selected problems, for closed-sourced [47, 22, 3] and open-sourced [21, 38, 16] MLLMs. a fast-evolving track, exemplified by the latest GPT-4V [47], Gemini [22], and the open-source LLaVA [39, 34, 32, 30] and SPHINX [36, 21].”
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, methodology): “MMMU has quickly become a standard evaluation tool in the field, used to assess the capabilities of many prominent multimodal models upon their release (OpenAI, 2024a,b; Anthropic, 2024; Reid et al., 2024; Li et al., 2024b).”
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (2024, positive): “For text understanding, we use data from [41].”
- VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “Following the principle outlined in LLaVA-OneVision [28], i.e., “quality over quantity”, we conduct rigorous cleaning procedures to guarantee data quality.”
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, positive): “To enable language capabilities, we train a Multimodal Large Language Model (MLLM) using V-JEPA 2 as the visual encoder in the non-tokenized early fusion (Wadekar et al., 2024) setup popularized by the LLaVA family of models (Li et al., 2024b).”
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “The scene shifts to a showroom setting, showcasing two blue cars, followed by a man in a black coat discussing features in front of a laptop.”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “The results suggest that our model achieves encouraging improvements over existing open-source models, such as LLaVA-OneVision [124], NVLM [50], VILA 1.5 [143], and Qwen2-VL [246], as well as notable progress compared to earlier versions of the InternVL2 series.”
- LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, positive): “We fine-tune LLaVA-OneVision (SI) on the joint dataset of video and image data.”
- LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “For simplicity, we combine these two steps in one stage using Single-Image data from LLaVA-OneVision (Li et al., 2024a).”