LLaVA-OneVision: Easy Visual Task Transfer

By Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li. Trans. Mach. Learn. Res., 2024

2,742 citations recorded by the source corpus.

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series. Our experimental results demonstrate that LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.

Subfields: Mechanical Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/1a71f7b216b710b936da666027014adb83af8e7a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mechanical Engineering

  1. Yann LeCun
  2. Nicolas Ballas

In Human-Computer Interaction

  1. Bo Li
  2. Yann LeCun
  3. Ziwei Liu
  4. J. Urgen Schmidhuber
  5. Juergen Gall
  6. Andrew Zisserman

Citations in the corpus