LLaVA-Video: Video Instruction Tuning With Synthetic Data

By Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, Chunyuan Li. Trans. Mach. Learn. Res., 2024

439 citations recorded by the source corpus.

The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Signal Processing

View Paper: https://www.semanticscholar.org/paper/ed44571dd9de3654b84ee1fea63015ecb67e6e63

Citations in the corpus

• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “For open-source models, we evaluate models from InternVL2 [14], ViLA [45], LongViLA [91], LongVA [101], LLaVA-OneVision [40], and LLaVA-Video [104].”
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “In particular, we share reported results of VideoMAEv2 (Wang et al., 2023), InternVideo-1B and 6B (Wang et al., 2024b), and VideoPrism (Zhang et al., 2024c) on the classification tasks we consider, when available.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “5-8B [32], and NVILA [36], as well as proprietary models like LLaVA-Video [24], Apollo-7B [14], and our previous generation model, VideoLLaMA2 [46].”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…zh) [30], VideoGPT+ (en) [174] VideoChat2-IT (en & zh) [130, 131], EgoTaskQA (en) [99], NTU RGB+D (en) [152], CLEVRER (en) [276], LLaVA-Video (en) [307], FineVideo (en) [67], PerceptionTest (en) [193], HiREST (en) [291], STAR (en) [259], General QA EgoSchema (en) [175], ScanQA (en) [10], LSMDC…”