TempCompass: Do Video LLMs Really Understand Videos?

By Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, Lu Hou. Annual Meeting of the Association for Computational Linguistics, 2024

342 citations recorded by the source corpus.

Recently, there is a surge in interest surrounding video large language models (Video LLMs). However, existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs. On the one hand, most of them are unable to distinguish between different temporal aspects (e.g., speed, direction) and thus cannot reflect the nuanced performance on these specific aspects. On the other hand, they are limited in the diversity of task formats (e.g., only multi-choice QA), which hinders the understanding of how temporal perception performance may vary across different types of tasks. Motivated by these two problems, we propose the \textbf{TempCompass} benchmark, which introduces a diversity of temporal aspects and task formats. To collect high-quality test data, we devise two novel strategies: (1) In video collection, we construct conflicting videos that share the same static content but differ in a specific temporal aspect, which prevents Video LLMs from leveraging single-frame bias or language priors. (2) To collect the task instructions, we propose a paradigm where humans first annotate meta-information for a video and then an LLM generates the instruction. We also design an LLM-based approach to automatically and accurately evaluate the responses from Video LLMs. Based on TempCompass, we comprehensively evaluate 8 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs, and reveal the discerning fact that these models exhibit notably poor temporal perception ability. Our data will be available at https://github.com/llyx97/TempCompass.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ac8089bb7944090cf1de5df25aadf5e6356f3040

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Haodong Duan
2. Yuhang Zang
3. Xiao-wen Dong
4. Jiaqi Wang
5. Pan Zhang
6. Xin Li
7. Dahua Lin
8. Sicong Leng

Citations in the corpus

• Video Instruction Tuning With Synthetic Data (2024, neutral): “We refer to public video question-answering benchmarks (Xiao et al., 2021; Yu et al., 2019; khattak et al., 2024; Liu et al., 2024b) to organize these questions into 16 specific categories, as shown in Fig.”
• ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, background): “Our high-quality video-caption data is particularly effective in helping LVLMs achieve significant performance improvements on benchmarks that require complex temporal understanding, such as TempCompass [38].”
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “With MLLMs displaying impressive performance on still-images across perception, reasoning, and multi-disciplinary tasks [39, 52, 98, 99], there is increasing interest in evaluating MLLMs’ video understanding capabilities [23, 24, 43, 44, 47, 51, 54, 55, 63, 85, 96].”
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “Compared to the current state-of-the-art PerceptionLM 8B (Cho et al., 2025), we observe an increase of 1.3 points on accuracy for PerceptionTest test set, 4.8 points on paired accuracy for MVP, 4.2 points on accuracy for TempCompass, 8.4 points on Multi-binary accuracy for short-QA segment for Tempo”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “However, existing video-based benchmarks [27, 32, 41, 45] are still limited to thoroughly reveal their performance, such as a lack of diversity in video types, insufficient coverage of temporal dynamics, and the narrow focus on a single modality.”
• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “There are specialized benchmarks for temporal perception [49; 42], action understanding [42; 41], video classification [14], video reasoning [45; 44], and video captioning [47; 30].”
• Video-R1: Reinforcing Video Reasoning in MLLMs (2025, neutral): “Our experiments show that Video-R1 achieves consistent and significant improvements across a suite of challenging video reasoning benchmarks, including VSI-Bench [24], VideoMMMU [8], MMVU [32], MVBench [13], TempCompass [18], and VideoMME [4].”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “…reasoning capabilities of VideoLLaMA3, we conduct evaluations on the following tasks: (1) Temporal Perception and Reasoning tasks, including TempCompass [134] and NextQA [135]; and (2) Temporal Sentence Grounding task on Charades-STA [108] benchmark, with mean Intersection over Union (mIoU) metric.”