VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

By Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao and 1 more. arXiv.org, 2024

817 citations recorded by the source corpus.

In this paper, we present the VideoLLaMA 2, a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks. Building upon its predecessor, VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC) connector, which effectively captures the intricate spatial and temporal dynamics of video data. Additionally, we integrate an Audio Branch into the model through joint training, thereby enriching the multimodal understanding capabilities of the model by seamlessly incorporating audio cues. Comprehensive evaluations on multiple-choice video question answering (MC-VQA), open-ended video question answering (OE-VQA), and video captioning (VC) tasks demonstrate that VideoLLaMA 2 consistently achieves competitive results among open-source models and even gets close to some proprietary models on several benchmarks. Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and audio-video question-answering (AQA&OE-AVQA) benchmarks over existing models. These advancements underline VideoLLaMA 2's superior performance in multimodal comprehension, setting a new standard for intelligent video analysis systems. All models are public to facilitate further research.

View Paper: https://www.semanticscholar.org/paper/c6dd31c40836d616b50feab17aff569591c4262e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Video-R1: Reinforcing Video Reasoning in MLLMs (2025, unknown): “Video understanding is an essential capability for Multimodal Large Language Models (MLLMs), enabling them to interpret and reason over dynamic visual content [20, 2, 23, 29, 30].”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, unknown): “Early research on MLLMs primally focuses on text generation based on text prompts and input multiple modalities such as images [18, 19], videos [8, 31].”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, methodology): “Following the methodology established in the existing literature (Zhang et al.)”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, unknown): “Previous works [14, 19, 22, 24, 142, 149] typically follow a training recipe that involves an alignment phase, followed by supervised fine-tuning, with instruction-tuning datasets [18, 24–27] often being video dominant.”
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (2024, background): “…benchmarks, as it is capable of processing multiple images with an interleaved format. we include VideoChat2 (Li et al., 2023c), Video-LLaMA2 (Cheng et al., 2024), Video-ChatGPT (Maaz et al., 2023), ShareGPT4Video (Chen et al., 2024c), PLLaVA (Xu et al., 2024), Idefics2 (Laurenccon et al.,…”
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “Most existing works (Li et al., 2023c; Ataallah et al., 2024a; Cheng et al., 2024; Zhang et al., 2024b; Li et al., 2024a) address this challenge by uniformly sampling frames from the video, which, however, results in a significant loss of visual details within the video.”
AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning (2025, unknown): “Prominent open-source examples include ImageBind-LLM [Han et al. 2023], ChatBridge [Zhao et al. 2023], Macaw-LLM [Lyu et al. 2023a], VAST [Chen et al. 2023b], OneLLM [Han et al. 2024], VideoLLaMA2 [Cheng et al. 2024], and video-SALMONN [Sun et al. 2025b].”