Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

By Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, Li Yuan. Conference on Empirical Methods in Natural Language Processing, 2023

1,663 citations recorded by the source corpus.

Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding. Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models. However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers.In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other.As a result, Video-LLaVA outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Additionally, our Video-LLaVA also achieves superior performances on a broad range of 9 image benchmarks.Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos. We aim for this work to provide modest insights into the multi-modal inputs for the LLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/107fb6eec2febbae12db29bf3e311aaf5680027c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, positive): “By integrating PruMerge with Video-LLaVA [26] during the inference phase alone ( i.e ., no need for additional training) we not only expedite processing within video-LLMs but also enhance their performance across multiple benchmarks.”
A Survey on Hallucination in Large Vision-Language Models (2024, methodology): “Recent LVLMs research made strides in incorporating more modalities (e.g., videos) through methods including through cache retrieval [Han et al. , 2023] and pre-alignment [Lin et al. , 2023].”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, methodology): “…image-text interleaved inputs such as Qwen-VL-Chat (Bai et al., 2023), InstructBLIP (Dai et al., 2023), CogVLM (Wang et al., 2023), VideoLLaVA (Lin et al., 2023a), VILA (Lin et al., 2023b), Idefics2 (Laurenccon et al., 2024), Mantis-SigLIP (Jiang et al., 2024) and LLAVA-Interleave (Li et al.,…”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, methodology): “Then we train the VideoLLaVA [34] and LLaMA-VID [32] with their default training settings and hyperparameters.”
VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, neutral): “Conversely, visual language models (VLMs) struggle to model lengthy visual inputs [9,17,18,21,23].”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “The video-text sources include Panda-70M (Chen et al., 2024b), VIDAL-10M (Zhu et al., 2023a) WebVid-10M (Bain et al., 2021), and InternVid-10M (Wang et al., 2023), while the image-text sources include CC-3M (Changpinyo et al., 2021) and DCI (Urbanek et al., 2023).”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “This category includes: Otter-V [18], mPlug-Owl-V [48], Video LLaMA-2 [52], Video ChatGPT [28], VideoChat [20], VideoChat2 [21], and Video-LLaVA [23].”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “MiniGPT-4 (Zhu et al., 2023) and LLaVA (Liu et al., 2024c) simplify the integration by projecting visual features directly into the LLM embedding space using a MLP.”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “While video LMMs excel in video understanding, they often do so at the expense of image performance [72, 76].”
TempCompass: Do Video LLMs Really Understand Videos? (2024, methodology): “Recent Video LLMs (Su et al., 2023; Li et al., 2023c; Zhang et al., 2023; Lin et al., 2023a; Li et al., 2023d; Maaz et al., 2023; Luo et al., 2023; Li et al., 2023g) primarily follow the End-to-End paradigm, with optional temporal modules to model the temporal information across frames.”
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, methodology): “Commonly, they incorporate a projection network [30, 22, 19], inter-modality attention [17, 18] or a modality perceiver [45, 32, 12] as learnable interfaces.”
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “Many early efforts [18, 33, 55] have been made to inspire the video understanding potentials of MLLMs with promising results.”
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models (2024, positive): “We take video understanding tasks with Vide-oLLaVA Lin et al. (2023) as example as shown in Figure 5.”
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (2024, positive): “We directly apply our SparseVLM for Video-LLaVA (Lin et al., 2023), which is a commonly used VLM framework for video question answering.”
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models (2024, methodology): “Large Vision-Language Models (LVLMs), such as LLaVA (Liu et al., 2023c) and MiniGPT-4 (Zhu et al., 2023), have shown promising results by leveraging an image en-1 Peking University 2 Sun Yat-sen University 3 FarReel AI Lab 4 University of Rochester 5 Peng Cheng Laboratory.”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, background): “The second class of methods emphasizes the integration of models from different modalities into end-to-end trainable models.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, result): “In Figure 11, we present the results of the ”Needle in a Haystack” experiment for long videos.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “Existing MLLMs [ 11– 34 ] have Inherit from VideoLLaMA [45] and VideoLLaMA2 [46], VideoLLaMA3, a more advanced multimodal foundation model, is proposed for both image and video understanding.”